Benchmark & Performance

What is a benchmark?

A benchmark is a standardized test used to measure and compare the performance of systems or components. In the context of neural networks, benchmarks help us evaluate:

• Speed: How quickly the model processes data
• Accuracy: How well the model performs its intended task
• Efficiency: How the model utilizes available resources

MNIST Dataset Benchmarks

The MNIST dataset is a collection of 60,000 grayscale images of handwritten digits (0-9), commonly used as a standard benchmark in machine learning.

Disclaimer: Benchmarks involving the SIMD variant of Brain4J refer to version 2.8.

Model Configuration

• Batch size: 128
• Epochs: 50
• Parameters: 110,170
• Optimizer: AdamW (0.01 learning rate, default decay)
• Architecture: MLP with layers
  • 784 (Input) → LINEAR
  • 128 → RELU
  • 64 → RELU
  • 10 (Output) → SOFTMAX

Results by Hardware Configuration

Intel i5-8400 with 32GB DDR4 @ 2667MHz RAM (performed by xEcho1337)

Framework	Seconds per epoch	Accuracy
Brain4J (SIMD)	~2.06	97.41%
Brain4J	~3.48	97.44%
DL4J	~3.71	97.39%
TensorFlow	~0.98	97.64%
PyTorch	~1.83	97.48%

AMD Ryzen 9 5900X with 32GB DDR4 @ 3600MHz RAM (performed by Adversing)

Framework	Seconds per epoch	Accuracy
Brain4J	Pending	N/A
TensorFlow	Pending	N/A
PyTorch	Pending	N/A

MacBook Air M3 2024 with 16GB LPDDR5 @ 3200MHz RAM (performed by xEcho1337)

Framework	Seconds per epoch	Accuracy
Brain4J (SIMD)	~0.78	97.66%
Brain4J	~1.29	97.62%
DeepLearning4J	~2.15	97.25%
TensorFlow	~0.44	97.37%
PyTorch	~0.37	97.62%

Intel i7-13700k with 32GB DDR5 @ 7200Mhz RAM (performed by RaynLegends)

Framework	Seconds per epoch	Accuracy
Brain4J (SIMD)	~0.92	97.52%
Brain4J	~1.20	97.55%
DeepLearning4J	~2.92	97.37%
TensorFlow	~0.62	97.44%
PyTorch	~1.11	96.83%

Note: This benchmark page is a work in progress. Additional frameworks, hardware configurations, and datasets will be added in future updates.

Next Steps

Check out Architecture