User:David.werecat/BFBench

From Esolang
Jump to: navigation, search

BFBench is a benchmark suite for Brainfuck implementations by David Catt (user:David.werecat). It is written in Python and is thought to be cross-platform. The most recent version of the suite (version 1.4) can be downloaded here. This page gives a brief summary of the interpreters that were tested and the benchmarks which were run on them.

Benchmark Descriptions

BFBench currently runs a variety of benchmarks of different types against each interpreter. The benchmarks included in the most recent version are as follows:

Name Timeout Description Issues
Mandelbrot 200s Computes and displays an ASCII art version of the mandelbrot fractal to the console.  
Factoring 50s Finds all factors of the number 123456789123456789 and displays them to the console. Fails on 32bit Cell
Long Run 200s Runs a series of nested loops meant to take a long time to execute.  
Self Interpret 200s Runs two self interpreters stacked and attempts to execute a simple "Hello, World!" program.
Golden Ratio 5s Calculates the Golden Ratio to 36 digits and displays the result to the console.
Hanoi 80s Runs a simulation of the towers of Hanoi problem and displays each step as ASCII art to the console.
99 Bottles of Beer 5s Displays the lyrics of the song "99 Bottles of Beer" to the console. Fails on 32bit Cell
Simple Benchmark 5s Runs a series of nested loops meant to take a small amount of time to execute. Fails on 32bit Cell
Hello World 1s Displays "Hello world" Displays "Hello Universe" on 32bit

Interpreter Descriptions

The benchmark was run on several optimizing and non-optimizing interpreters. The interpreters were compiled with TCC if their source code was provided. The interpreters benchmarked in the most recent version are as follows:

Name Optimizing Description Link
bff Yes A moderately optimizing interpreter written in C by Alex Pankratov. [1]
bff4 Yes An optimizing interpreter written in C by Oleg Mazonka, said to be "fastest in class". [2]
bff4n Yes bff4 compiled without linear optimization. [3]
bffsree Yes A very fast interpreter written by Sree Kotay (user:Sreekotay). [4]
bffsree_ref No A reference version of bffsree written in C by Sree Kotay (user:Sreekotay). [5]
bf-li Yes A very fast optimizing JIT interpreter written in C by Laurent Vogel (user:lvogel). [6]
bf-li_slow Yes bf-li without JIT enabled. [7]
bfref No The reference (slow) interpreter from the BFDB sources. [8]
FBI Yes An optimizing interpreter written in C by by David Catt (user:David.werecat). [9]
stackbfi No An interpreter written in C by Oleg Manzonka which only uses stack memory. [10]
yabi No A very simple interpreter written in C by Mickael Riga. [11]


Benchmark Results

Benchmark Setup

These results have been obtained by David Catt (user:David.werecat) on a laptop with the following specs:

  • OS: Windows 7 (64-bit)
  • CPU: Intel Core i7-2670QM (2.20 GHz X 8 cores)
  • Memory: 8.00GB DDR3 (665.3 MHz dual channel)
  • Storage: 750GB HDD

Results Format

When the suite is run, it displays the results for all tests run against each interpreter on the console. The output format from the suite may be difficult to understand for the average person, they have been cleaned up and simplified. Each result is highlighted in a color which describes the outcome of the benchmark. The possible colors are as follows:

Color Description
Green Program ran within the time limit and produced the correct output.
Yellow Program ran within the time limit but produced an incorrect output.
Red Program failed to run within the time limit.
Gray The interpreter fatally crashed.

Note that any test in which the interpreter exited with an exit code other than zero is displayed in italics.

Results

The results of the benchmark are as follows (all times in seconds):

Interpreter Mandelbrot Factoring Long Run Self Interpret Golden Ratio Hanoi 99 Bottles of Beer Simple Benchmark
bff 12.515s 50.001s 30.998s 200.002s 0.305s 27.066s 5.003s 5.002s
bff4 15.366s 3.266s 9.214s 36.209s 0.165s 1.165s 0.017s 5.001s
bff4n 15.015s 50.001s 31.168s 35.494s 0.266s 28.067s 5.002s 5.001s
bffsree 4.516s 1.164s 0.214s 4.115s 0.115s 0.065s 0.017s 0.016s
bffsree_ref 33.524s 10.139s 64.571s 106.597s 0.415s 56.988s 0.074s 3.566s
bf-li 1.065s 0.164s 0.264s 2.214s 0.033s 0.064s 0.016s 0.016s
bf-li_slow 18.666s 4.065s 10.816s 16.416s 0.115s 2.065s 0.016s 0.064s
bfref 80.619s 21.265s 99.238s 91.957s 0.714s 46.517s 0.032s 2.262s
FBI 17.463s 6.564s 18.770s 28.799s 0.414s 1.664s 0.017s 0.015s
stackbfi 187.808s 50.001s 128.514s 117.781s 1.015s 4.866s 5.002s 5.003s
yabi 200.002s 50.002s 81.651s 20.117s 5.002s 80.001s 5.001s 5.001s

note: The interpreter "bffsree" produces extra output containing information about its code optimizations and the time taken to interpret the given input. This causes it to fail the output verification on all tests.

note: The interpreter "bffsree_ref" appends an extra newline to the end of the program output. This causes it to fail the output verification on all tests.

External resources