There is a well known benchmark on the shootout.alioth.debian.org website comparing the performance of many different programming languages. While I am far from discussing the superiority of one language over another, I realized that it would be interesting to perform a little speed test between a … PHP and PHP (compiled with the help of HipHop for PHP).
I hope that test results presented below will be helpful in answering, how much faster is PHP after compilation, and whether the HipHop for PHP is actually as efficient as claimed by Facebook developers.
See some shocking results in second part of the benchmark: HipHop for PHP: Benchmark – Revenge of PHP [english version]
Introductory remarks
As always, I do not intend to describe the exact configuration of the system, collect detailed statistics and their distributions, or make one thousand repetitions for each test.
I do not paste entire output if these results take more than a few lines of text.
In that case, I send output to the out.txt file and present you only the execution times received by the tail out.txt command.
It’s up to the reader to draw the final conclusions or perform more accurate tests.
Testing platform
Processor: Intel(R) Core(TM)2 Duo CPU E7600 @ 3.06GHz
Memory: 1GB RAM (upgraded to 2.5 GB of RAM for test no. 6)
System: Fedora 12 (64bit)
Kernel: 2.6.32.26-175.fc12.x86_64 #1 SMP
The server was used exclusively for testing purposes – it was running only the services associated with the benchmark.
Test results
Here I present comparative results between the standard PHP and the scripts compiled with HipHop for PHP. All test results are shown in seconds (of course lower times are better)
Test #1: n-body (Perform an N-body simulation of the Jovian planets)
PHP Sourcecode:
http://shootout.alioth.debian.org/u32/program.php?test=nbody&lang=php&id=3
PHP result:
[root@localhost src]# php -n bench.php 50000000 -0.169075164 -0.169059907 DONE IN 754.474136 |
HipHop for PHP result:
[root@localhost src]# /tmp/hphp_z8ZAiC/program 50000000 -0.169075164 -0.169059907 DONE IN 360.381707 |
In this test HipHop for PHP is more than two times faster than standard PHP.
Test #2: fannkuch-redux (Repeatedly access a tiny integer-sequence)
PHP Sourcecode:
http://shootout.alioth.debian.org/u32/program.php?test=fannkuchredux&lang=php&id=1
PHP result:
[root@localhost src]# php -n bench2.php 12 3968050 Pfannkuchen(12) = 65 DONE IN 3545.183693 |
HipHop for PHP result:
[root@localhost src]# /tmp/hphp_4gdbXK/program 12 3968050 Pfannkuchen(12) = 65 DONE IN 1360.819455 |
Still no surprises, compiled PHP script once again proved to be more than two times faster.
Test #3: fasta (Generate and write random DNA sequences)
PHP Sourcecode:
http://shootout.alioth.debian.org/u32/program.php?test=fasta&lang=php&id=3
PHP result:
[root@localhost src]# php -n bench3.php 25000000 > out.txt DONE IN 138.252463 |
HipHop for PHP result:
[root@localhost src]# /tmp/hphp_dKRE8V/program 25000000 > out2.txt DONE IN 55.924453 |
HipHop for PHP is still maintaining a double advantage.
Test #4: spectral-norm (Calculate an eigenvalue using the power method)
PHP Sourcecode:
http://shootout.alioth.debian.org/u32/program.php?test=spectralnorm&lang=php&id=3
PHP result:
[root@localhost src]# php -n bench4.php 5500 1.274224153 DONE IN 238.810613 |
HipHop for PHP result:
[root@localhost src]# /tmp/hphp_N7arkg/program 5500 1.274224153 DONE IN 93.717688 |
Test #5: mandelbrot (Generate a Mandelbrot set and write a portable bitmap)
PHP Sourcecode:
http://shootout.alioth.debian.org/u32/program.php?test=mandelbrot&lang=php&id=1
PHP result:
[root@localhost src]# php -n bench5.php 16000 > out.txt DONE IN 1532.191153 |
HipHop for PHP result:
[root@localhost src]# /tmp/hphp_cMY58k/program 16000 > out2.txt DONE IN 569.224470 |
Here the difference between an ordinary PHP and compiled by HipHop gets even bigger, script execution time was reduced almost threefold.
Test #6: binary-trees (Allocate and deallocate many many binary trees)
PHP Sourcecode:
http://shootout.alioth.debian.org/u32/program.php?test=binarytrees&lang=php&id=3
PHP result:
[root@localhost src]# php -n -d memory_limit=2048M bench6.php 16 stretch tree of depth 17 check: -1 131072 trees of depth 4 check: -131072 32768 trees of depth 6 check: -32768 8192 trees of depth 8 check: -8192 2048 trees of depth 10 check: -2048 512 trees of depth 12 check: -512 128 trees of depth 14 check: -128 32 trees of depth 16 check: -32 long lived tree of depth 16 check: -1 DONE IN 37.286621 |
HipHop for PHP result:
[root@localhost src]# /tmp/hphp_s93MOe/program 16 stretch tree of depth 17 check: -1 131072 trees of depth 4 check: -131072 32768 trees of depth 6 check: -32768 8192 trees of depth 8 check: -8192 2048 trees of depth 10 check: -2048 512 trees of depth 12 check: -512 128 trees of depth 14 check: -128 32 trees of depth 16 check: -32 long lived tree of depth 16 check: -1 DONE IN 11.450954 |
After performing many different types of tests we were finally able to cross the barrier of 300%. I suspect that in this case a better result corresponds to the faster implementation of functions in native C++, commonly used in the test loop.
Update report
Previous tests were performed on a smaller data sample (n = 16) than in the reference test, since the potential memory consumption exceeded the physical abilities of test server.
Intrigued by the above results and claims from Facebook developers concerning smaller memory usage of scripts compiled by HipHop for PHP, I decided to repeat the test with a reference sample of n = 20 (such as in test at shootout webpage). To make it possible, I upgraded a test machine with an additional 1.5 Gb of RAM. Here are the results.
PHP result:
[root@localhost src]# php -n -d memory_limit=2548M bench6.php 20 stretch tree of depth 21 check: -1 2097152 trees of depth 4 check: -2097152 524288 trees of depth 6 check: -524288 131072 trees of depth 8 check: -131072 32768 trees of depth 10 check: -32768 8192 trees of depth 12 check: -8192 2048 trees of depth 14 check: -2048 512 trees of depth 16 check: -512 128 trees of depth 18 check: -128 32 trees of depth 20 check: -32 long lived tree of depth 20 check: -1 DONE IN 1020.273547 |
HipHop for PHP result:
[root@localhost src]# /tmp/hphp_s93MOe/program 20 stretch tree of depth 21 check: -1 2097152 trees of depth 4 check: -2097152 524288 trees of depth 6 check: -524288 131072 trees of depth 8 check: -131072 32768 trees of depth 10 check: -32768 8192 trees of depth 12 check: -8192 2048 trees of depth 14 check: -2048 512 trees of depth 16 check: -512 128 trees of depth 18 check: -128 32 trees of depth 20 check: -32 long lived tree of depth 20 check: -1 DONE IN 249.907754 |
As you can see, this time HipHop for PHP was faster by over 400%. Also the memory usage was much lower: 2465MB used by standard PHP vs 1802MB used by HipHop for PHP (more than 600MB of difference!).
Summary
Here is the summary of test results:
Test name | PHP | HipHop | Ratio |
n-body | 754.474136 | 360.381707 | 209,35% |
fannkuch-redux | 3545.183693 | 1360.819455 | 260,52% |
fasta | 138.252463 | 55.924453 | 247,21% |
spectral-norm | 238.810613 | 93.717688 | 254,81% |
mandelbrot | 1532.191153 | 569.224470 | 269,17% |
binary-trees (16) | 37.286621 | 11.450954 | 325,62% |
binary-trees (20) | 1020.273547 | 249.907754 | 408,26% |
As you can see HipHop for PHP dramatically accelerates the PHP scripts. Differences in the time range from 200% to over 400% and have been achieved without any changes applied to the PHP source code files.
Also, memory consumption is much lower in the case of PHP compiled by HipHop for PHP (up to 25%).
Unfortunately, there were some incompatibilities with standard PHP language: for example, lack of support for gmp library (failure to perform the pidigits test).
I describe HipHop for PHP incompatibility problems in a separate article.
Other articles about breaking the limits of PHP
- HipHop for PHP: Benchmark – Revenge of PHP
- Strong data typing in PHP, part II: autoboxing and indestructable objects
- Mod_rewrite and HipHop for PHP on Apache Web Server
- How to install HipHop for PHP on Fedora 12 Linux (google translation)
- Beyond limits: hacking SAFE_MODE and max_execution_time (google translation)
- What really slows PHP down? (google translation)
- Howto: Strong data types in PHP (google translation)
Tags: alioth shootout, benchmark, Facebook, Fedora, Linux
reverse-complement?
k-nucleotide?
regex-dna?
Ok, I’ll try to add those measurements today.
@Isaac Gouy, I’ve added two more tests for you: http://php.webtutor.pl/index.php/2011/04/04/hiphop-for-php-benchmark-revenge-of-php
b.t.w. Results are shocking!
It’s worth considering also running these same tests with APC enabled since a lot of the people who may be candidates for HipHop are already running opcode-cached software.
That’s a good idea. I’ll repeat those test with APC and try to compare the results.
Don you think this should be done in another article rather than this one?
I’ve tried n-body workload, but the result is really bad on HHVM:
do you have experience on HHVM? Is there anything I should pay more attention? like system configuration….etc.
root@ubuntu:/home/hhvmtest/n-body# time hhvm nbody.php 50000
-0.169075164
-0.169078071
real 0m1.130s
user 0m1.100s
sys 0m0.024s
root@ubuntu:/home/hhvmtest/n-body# time php nbody.php 50000
-0.169075164
-0.169078071
real 0m0.667s
user 0m0.656s
sys 0m0.004s
==========================================
do you have any data on hhvm vs. PHP?