User:David.werecat/BFBench
BFBench is a benchmark suite for Brainfuck implementations by user:David.werecat. It is written in Python and is thought to be cross-platform. The most recent version of the suite (version 1.4) can be downloaded here. This page gives a brief summary of the interpreters that were tested and the benchmarks which were run on them.
Benchmark Descriptions
BFBench currently runs a variety of benchmarks of different types against each interpreter. The benchmarks included in the most recent version are as follows:
| Name | Timeout | Description | Issues |
|---|---|---|---|
| Mandelbrot | 200s | Computes and displays an ASCII art version of the mandelbrot fractal to the console. | |
| Factoring | 50s | Finds all factors of the number 123456789123456789 and displays them to the console. | Fails on 32bit Cell |
| Long Run | 200s | Runs a series of nested loops meant to take a long time to execute. | |
| Self Interpret | 200s | Runs two self interpreters stacked and attempts to execute a simple "Hello, World!" program. | |
| Golden Ratio | 5s | Calculates the Golden Ratio to 36 digits and displays the result to the console. | |
| Hanoi | 80s | Runs a simulation of the towers of Hanoi problem and displays each step as ASCII art to the console. | |
| 99 Bottles of Beer | 5s | Displays the lyrics of the song "99 Bottles of Beer" to the console. | Fails on 32bit Cell |
| Simple Benchmark | 5s | Runs a series of nested loops meant to take a small amount of time to execute. | Fails on 32bit Cell |
| Hello World | 1s | Displays "Hello world" | Displays "Hello Universe" on 32bit |
Interpreter Descriptions
The benchmark was run on several optimizing and non-optimizing interpreters. The interpreters were compiled with TCC if their source code was provided. The interpreters benchmarked in the most recent version are as follows:
| Name | Optimizing | Description | Link |
|---|---|---|---|
| bff | Yes | A moderately optimizing interpreter written in C by Alex Pankratov. | [1] |
| bff4 | Yes | An optimizing interpreter written in C by Oleg Mazonka, said to be "fastest in class". | [2] |
| bff4n | Yes | bff4 compiled without linear optimization. | [3] |
| bffsree | Yes | A very fast interpreter written by Sree Kotay (user:Sreekotay). | [4] |
| bffsree_ref | No | A reference version of bffsree written in C by Sree Kotay (user:Sreekotay). | [5] |
| bf-li | Yes | A very fast optimizing JIT interpreter written in C by Laurent Vogel (user:lvogel). | [6] |
| bf-li_slow | Yes | bf-li without JIT enabled. | [7] |
| bfref | No | The reference (slow) interpreter from the BFDB sources. | [8] |
| FBI | Yes | An optimizing interpreter written in C by user:David.werecat. | [9] |
| stackbfi | No | An interpreter written in C by Oleg Manzonka which only uses stack memory. | [10] |
| yabi | No | A very simple interpreter written in C by Mickael Riga. | [11] |
Benchmark Results
Benchmark Setup
These results have been obtained by user:David.werecat on a laptop with the following specs:
- OS: Windows 7 (64-bit)
- CPU: Intel Core i7-2670QM (2.20 GHz X 8 cores)
- Memory: 8.00GB DDR3 (665.3 MHz dual channel)
- Storage: 750GB HDD
Results Format
When the suite is run, it displays the results for all tests run against each interpreter on the console. The output format from the suite may be difficult to understand for the average person, they have been cleaned up and simplified. Each result is highlighted in a color which describes the outcome of the benchmark. The possible colors are as follows:
| Color | Description |
|---|---|
| Green | Program ran within the time limit and produced the correct output. |
| Yellow | Program ran within the time limit but produced an incorrect output. |
| Red | Program failed to run within the time limit. |
| Gray | The interpreter fatally crashed. |
Note that any test in which the interpreter exited with an exit code other than zero is displayed in italics.
Results
The results of the benchmark are as follows (all times in seconds):
| Interpreter | Mandelbrot | Factoring | Long Run | Self Interpret | Golden Ratio | Hanoi | 99 Bottles of Beer | Simple Benchmark |
|---|---|---|---|---|---|---|---|---|
| bff | 12.515s | 50.001s | 30.998s | 200.002s | 0.305s | 27.066s | 5.003s | 5.002s |
| bff4 | 15.366s | 3.266s | 9.214s | 36.209s | 0.165s | 1.165s | 0.017s | 5.001s |
| bff4n | 15.015s | 50.001s | 31.168s | 35.494s | 0.266s | 28.067s | 5.002s | 5.001s |
| bffsree | 4.516s | 1.164s | 0.214s | 4.115s | 0.115s | 0.065s | 0.017s | 0.016s |
| bffsree_ref | 33.524s | 10.139s | 64.571s | 106.597s | 0.415s | 56.988s | 0.074s | 3.566s |
| bf-li | 1.065s | 0.164s | 0.264s | 2.214s | 0.033s | 0.064s | 0.016s | 0.016s |
| bf-li_slow | 18.666s | 4.065s | 10.816s | 16.416s | 0.115s | 2.065s | 0.016s | 0.064s |
| bfref | 80.619s | 21.265s | 99.238s | 91.957s | 0.714s | 46.517s | 0.032s | 2.262s |
| FBI | 17.463s | 6.564s | 18.770s | 28.799s | 0.414s | 1.664s | 0.017s | 0.015s |
| stackbfi | 187.808s | 50.001s | 128.514s | 117.781s | 1.015s | 4.866s | 5.002s | 5.003s |
| yabi | 200.002s | 50.002s | 81.651s | 20.117s | 5.002s | 80.001s | 5.001s | 5.001s |
note: The interpreter "bffsree" produces extra output containing information about its code optimizations and the time taken to interpret the given input. This causes it to fail the output verification on all tests.
note: The interpreter "bffsree_ref" appends an extra newline to the end of the program output. This causes it to fail the output verification on all tests.