satine.org

by Charles Ying

SquirrelFish is faster than Tamarin

June 3rd, 2008

WebKit and Tamarin Comparison

Smaller bars are better.

Wow.

I compared WebKit’s new SquirrelFish bytecode JavaScript interpreter against Tamarin, the JIT JavaScript engine currently in Flash 9 and in development for Firefox.

On the SunSpider JavaScript benchmark, SquirrelFish is at least 1.9 times faster than Tamarin with tracing turned on; 1.8 times faster than vanilla Tamarin, the same engine used in Flash 9. When you add type annotations to Tamarin (SquirrelFish doesn’t support them), Tamarin gets a little faster, but is still slower than SquirrelFish on these benchmarks.

Keep in mind, SquirrelFish’s reported times include source compilation, while Tamarin is executing pre-compiled bytecode outside the test.

First Impressions

  1. I can imagine the “performance per watt” power consumption for SquirrelFish is also much lower. Good for my iPhone’s battery life.
  2. I think I’m wrong about the importance of JITs. At least until SquirrelFish has gone through a few rounds of optimizations.
  3. ECMAScript 4 type annotations can clearly help in performance critical code, but most of the web executes in untyped, vanilla JavaScript.

Notes on my tests: Running SunSpider, tamarin-tracing, tamarin-central, and SquirrelFish all pulled from source on 2008-06-02. I ran the test on a 2.4 Ghz iMac. Raw numbers from the comparison runs are available here.

Running your own version of the tests

To reproduce these tests, build WebKit, build tamarin-tracing and tamarin-central from Mozilla.org, and install the Flex SDK for a bytecode compiler. Then,

  1. Set up your environment as follows: DYLDFRAMEWORKPATH="~/jsbench/WebKit/WebKitBuild/Release" ASC="~/jsbench/asc.jar" GLOBALABC="~/jsbench/tamarin-tracing/core/builtin_full.abc"
  2. Create a “timer.js” file that adds getTimer to the SunSpider benchmark: function getTimer() { return new Date(); }
  3. Create a “jscore” script that will run the original SunSpider AS source files instead of the ABC bytecode files: #!/bin/sh ~/jsbench/WebKit/WebKitBuild/Release/testkjs -f ~/jsbench/timer.js -f echo $1 | sed s/abc/as/
  4. And call the runtests.py script in the SunSpider test/performance sub-directory: ./runtests.py --avm ~/research/jsbench/jscore --avm2 ~/research/jsbench/tracing-avmshell ./runtests.py --avm ~/research/jsbench/jscore --avm2 ~/research/jsbench/central-avmshell

Technorati Tags: , , , , ,

25 Responses to “SquirrelFish is faster than Tamarin”

  1. Jesper Noehr Says:

    Impressive! It’s nice putting things in perspective. Thanks for taking the time to do the benchmarks.

  2. Hasan Pulur Says:

    The chart is misleading. If it’s 1.08 times faster, chart should reflect that. It seems like it’s like 7-8 times faster. — Edit — Thanks Hasan, I’ve updated the graph (after getting some sleep).

  3. Bernie Sumption Says:

    Isn’t tracing supposed to make Tamarin faster? Since in this test tracing is slower, it would seem that either tracing was a waste of all that development time, or the tests aren’t set up in a way that show off the advantages of Tamarin very well.

    Still, it’s an incredible result.

  4. Maximilian Schulz Says:

    These are indeed some pretty impressive numbers. Unfortunately, I am not really using WebKit at all. But everyone will benefit, as the competition for the best engine continues.

  5. Hasan Pulur Says:

    thanks for correcting the chart.

  6. Tommy Reilly Says:

    Do you have the #’s broken down by test? Tamarin tracing has a known defective String implementation (fix in the works) that makes aggregate scores less interesting.

  7. Mark Rowe Says:

    Tommy, the blog post links to the raw data: http://spreadsheets.google.com/pub?key=pccFYFvfwkfYGx_-4bCnbIg. The figure used in the graphs excludes the string-validate-input test which clearly has skewed results in the tracing variant of Tamarin.

  8. Brian West Says:

    How does it compare to Spider Monkey 1.6?

  9. Werner Sharp Says:

    The “typed” test cases in the Tamarin depository are currently ECMA compliant test files so they can be run in any Javascript engine. If I take the “accessbinarytrees.as” file and re-write it using true classes, then the performance with the Flash Player (Tamarin Central codebase) jumps by around 3.25x. My PC takes 65 milliseconds for the ECMA typed version and only 20 milliseconds for an AS3 Class based typed version. I would bet that a lot of the other test can be sped up as well by converting then to class based scripts. (what will be Ecmascript 4 hopefully)

    TestNode.as for accessbinarytrees.as:

    package {

    public class TreeNode
    {
        var left:TreeNode;
        var right:TreeNode;
        var item:int;
    
        function TreeNode(left:TreeNode,right:TreeNode,item:int):void
        {
           this.left = left;
           this.right = right;
           this.item = item;
        }
    
        public function itemCheck():int
        {
            if (this.left==null) 
                return this.item;
            else 
                return this.item + this.left.itemCheck() - this.right.itemCheck();
        }   
    }
    

    }

  10. Wellu Says:

    These figures clearly speak for themselves. What I’m interested is these speedups really as big in all architectures like ARM or do these new optimizations really need some fancy x86 register tweaks and FPUs. This is anyway good news for the mainstream but I’m not sure if this is the case in “embedded” systems as well.

  11. gfan Says:

    “SquirrelFish’s reported times include source compilation” It seems to me that the SunSpider BenchMark only report the interpret time, not include the compile time. No matter you run the .abc or .as file, the time reported is only the interpret time. Right?

  12. David Temkin Says:

    Charles — long time no talk! for what it’s worth, openlaszlo is an interesting comparison point for this type of comparison. the development builds can now build browser-targeted JavaScript as well as SWF8 (AS2) and SWF9 (Tamarin/AS3/JIT) output, and what we’re seeing is:

    1. Safari/Webkit is the definitely the fastest of the browsers for running OpenLaszlo apps. Safari, in fact, is considerably faster than the non-JIT version of the Flash player running the same code. Even rendering and animation is faster. This isn’t true of other browsers.

    2. Tamarin is faster yet than Safari, even for untyped code, in application level tests. Early tests are showing significant speed differences compared even to Safari — 2-3x. Now, this is partially because the OpenLaszlo compiler emits AS3 native classes, which according to what’s above, have a real impact, and are not part of the tests you’re running.

  13. alanjstr Says:

    Would mozilla.org consider using this engine instead of Tamarin?

  14. gfan Says:

    For JavaScript, most of the scripts are compiled on-the-fly, so the compilation time is also very important. As David Temkin’s comment above, it is more convincing if considering both compile and interpret time for SunSpider Benchmark

  15. Oliver Says:

    @gfan: the SquirrelFish times do include compilation time, it’s completely unavoidable. Tamarin is the only one for which the compile time is not included because currently the tamarin jit is unable to to run javascript itself, it requires a separate program to compile the javascript to actionscript bytecodes.

    I’m not sure why you believe squirrelfish times don’t include that time.. in fact i’m not sure why you think it is actually possible.

    Finally, SunSpider was created to find performance issues, if it did not include compilation time it would not be useful, as we would be unable to make any changes to codegen because we would not be able to measure the resultant performance (there’s no point making a change that makes interpreting a script 2 seconds faster if the compilation takes 3 seconds longer).

  16. Oliver Says:

    @David Temkin: Any chance of you filing performance bugs at http://bugs.webkit.org ?

  17. Oliver Says:

    @gfan: ah, i thought i’d replied to you before (i did on the webkit blog). In the commandline version of sunspider it is possible for sunspider to remove absolutely all computation time (lexing, parsing, compilation) as sunspider gets the time before and after calling “load(somescript)”. In the browser that’s not completely possible as you can’t control network latency, so the tests have to calculate start and end times themselves. Unfortunately this means the the outermost (global code) will have been compiled, and all the code will have been parsed.

    Two important things to note though are: parsing takes basically 0% of the runtime — you can add fairly significant amounts of additional work to the parser without causing any change in sunspider (although one would hope you were making such changes to enable improvements later :D ). As for the compilation of the global code segment, there is only one test that actually performs the majority of its computation in the global scope and that has a very small amount of code relative to its runtime, so there should not be any significant impact from this.

    Finally, even though we do discount the compile time for a very small portion of the tests, profiling (at least in squirrelfish) shows that less than 0.3% of the execution time is spent generating the bytecode, so it really is not significant.

  18. Maciej Stachowiak Says:

    @David Temkin

    Got a pointer to the application level tests you are trying? OpenLaszlo sounds like an interesting tool for performance testing since we can compare directly to Flash for functionally similar things.

    Also, if you are generating native classes for AS, then isn’t that typed code?

    In general though I’d love to profile a heads-up comparison of an app-level benchmark and see if the time is in rendering or script execution or something else surprising.

  19. CityBlog | Safari 4 farà tremare Firefox? Says:

    […] campo dell’ottimizzazione delle macchine virtuali. Secondo i primi benchmark, SquirrelFish è quasi il doppio più veloce di Tamarin, il motore JavaScript alla base di Flash 9 e delle future versioni di Firefox, e il 46% […]

  20. Sandy Says:

    Looks like TraceMonkey is now faster than SquirrelFish on the whole. Yay competition!

    http://www.masonchang.com/2008/08/tracemonkey-vs-squirrelfish.html

  21. Performance web » SquirrelFish dans WebKit Says:

    […] moteur de script de Flash 9, Tamarin, donné par Adboe/Macromedia au projet Mozilla. Le travail sur les performances n’est pas encore aussi abouti, mais on peut tout de même prévoir de bonnes choses pour le […]

  22. Ajax for the Real World :: Browser JavaScript performance leapfrogged again Says:

    […] JavaScript interpreter, providing a huge boost to JavaScript performance that actually put Safari ahead of Tamarin, the JavaScript engine used in Adobe […]

  23. of Says:

    And there By, wonderful way for?Some features in, things ? unless.Article I want, important as it.Can turn the of, push ==> If causes writers block.This concept is, much cash they.,

  24. InariakaM Says:

    Тетрис (производное от «тетрамино» и «теннис»), англ. Tetris — культовая компьютерная игра, изобретённая Алексеем Пажитновым и представленная общественности 6 июня 1984 года.<br/>

    Найти просто на нашем сервисе бесплатных объявлений. Объявления со всех регионов России. Продай старые вещи – купи машину! Купи намного дешевле у частника. Продать так же просто, как и купить. Найди то, что ищешь по низкой цене! На нашем сайте вы обязательно обнаружите что-нибудь нужное для себя ! Посетите наш портал – вы не останитесь недовольны.

    Дети – Все развлечения Москвы для детей, детские театры, кино, мультфильмы, цирки и парки.<br/>

    Купить

  25. Jhon Says:

    Very interesting. I will recommend this site to everyone. Congratulations