Advertisement

Benchmarking HipHopVM against PHP 5.3 on Ubuntu 12.04

HipHopVM is Facebook’s high-performance PHP runtime, that is built to be a production-ready virtual machine that delivers superior performance. Facebook was using a PHP-to-C++ compiler (HPHPc), but is now moving to HipHopVM (HHVM) to unify their production and development environments.

HipHopVM Performance

PHP 5.3.10 is the current version of PHP on Ubuntu 12.04 (the latest LTS release). On Ubuntu 12.04 HipHopVM has binary packages available (v2.1.0).

hhvm_speed

This graph shows the average execution time (of 10 runs) on both PHP 5.3.10 (with APC) and HipHopVM v2.1.0. The Y-axis shows the amount of seconds the execution took (on average) and the numbers on the X-axis represent the following benchmark scripts:

  1. bench1.php: http://www.lornajane.net/posts/2012/proof-that-php-5-4-is-twice-as-fast-as-php-5-3
  2. bench2.php: https://coderwall.com/p/il1tog
  3. bench3.php: http://php.net/manual/en/features.gc.performance-considerations.php
  4. bench4.php: http://akrabat.com/php/php-5-3-is-quicker-than-php-5-2-2/
  5. bench5.php: http://onlinephpfunctions.com/benchmarks
  6. bench6.php: http://www.synet.sk/php/en/220-php-performance-benchmark

This is the raw data:

hhvm_speed_data

Note that the memory usage (in bench3.php) is also different:

hhvm_gc

Conclusion

The differences were smaller and less consistent than I expected. Also I was disappointed that HipHopVM did not run my code.

NB: HHVM v2.2.0 is out and it runs Symfony!

What do you think about these numbers? Please share your thoughts in the comments below.

12 Responses to “Benchmarking HipHopVM against PHP 5.3 on Ubuntu 12.04”

  • Isaac Gouy:

    Those are such tiny tiny scripts that they make the benchmarks game programs seem like major applications!

    http://benchmarksgame.alioth.debian.org/u32/measurements.php?lang=php

    Isn’t there some real PHP code that you could use for your comparison?

  • Maurits van der Schee (Innovation Engineer):

    @Isaac: Yes, valid point, but the problem is that not all code runs in HipHopVM. If it did I would have benchmarked on the Symfony or CakePHP framework.

  • Yuri:

    HHVM is JIT, PHP 5.3 – isn’t. Writing microbenchmarks for JIT is hard. Please look at this article http://www.ibm.com/developerworks/java/library/j-jtp12214/ , it is for JAVA, but the priniciples of JAVA VM have many parts similar to HHVM.

  • Maurits van der Schee (Innovation Engineer):

    @Yuri: Interesting link, thank you. I am hopefully running the code often to have enough warmup for optimal performance. If you have suggestions that improve the benchmark, please share.

  • Isaac Gouy:

    1) These are the tasks from bench6.php that were rejected from the benchmarks game back in 2005 because it was so obvious they could easily be replaced by tasks that made some kind-of sense —

    ackermann, ary, ary2, fibo, hash1, hash2, heapsort, matrix, nestedloop, sieve, strcat

    2) If Symfony or Cake won’t run on HipHop then your challenge is to find something real that will.

    The benchmarks game website is PHP —

    http://anonscm.debian.org/viewvc/benchmarksgame/benchmarksgame/website/lib/

    The only framework used is 200 lines of Brian Lozier’s templating code —

    http://anonscm.debian.org/viewvc/benchmarksgame/benchmarksgame/website/lib/lib.php?revision=1.1&view=markup

    No one will ever think the code is great PHP but at least it’s generating XHTML from data.

  • The benchmark scripts used all have one major flaw: They disable the JIT.

    Take a look at http://www.hhvm.com/blog/?p=713 where Owen explains why global-scope code is not JITted (and can’t be, really) to machine code and thus will only be about on-par with PHP. If you just wrap these benchmarks in `function f() { /* benchmark */ } f();` you’ll see a speed up. Also, bear in mind that the JIT isn’t enabled by default for command-line scripts (because perf isn’t the main goal there), you should explicitly turn it on with:

    hhvm -vEval.Jit=1 bench.php

  • Maurits van der Schee (Innovation Engineer):

    @Sara: Thank you for your comment, very informative. FYI, I did not run the scripts from the command-line. I will try the function scope around the benchmark scripts to see whether it makes a significant difference.

  • Hey, a new version HHVM came out, can you give it a shot?

    http://www.hhvm.com/blog/1301/hhvm-2-2-0

    Supposed to have major speedups and compatibility improvements!

  • Maurits van der Schee (Innovation Engineer):

    @AntonBe: Yes, I am already setting it up for tests. Since it promised to allow for Symfony to run as well, it takes me a little more time. It will be one of the next posts.

  • @Maurits, I’m excited about that as well! That also hopefully means Laravel compatibility, which I like a bit more than Symfony for various reasons.

  • Adri:

    What are you trying to bechmark, exactly? The first 11 runs in HHVM are to warm up the JIT, which means that they are going to be significantly slower than they would usually be. You should start benchmarking from the 12th run.

    It’s a little bit unfortunate that this post comes at the top when searching for hhvm and php speed comparisons.

  • Maurits van der Schee (Innovation Engineer):

    @Adri: Thank you for your comment and your concern. I took the obvious problem of warming up the cache into account and did a proper warmup. Did you find any significant different results that you want to share?

Leave a Reply