[Superfly_FR]

In a previous thread, we discussed about BB10 Browser HTML5 compliance. It lead to various considerations and finally, turn out to "performance".
In my opinion, performance tests with a developer device is not valuable, as it is well known its hardware is specific and OS not yet optimized "in full".
Yet, some have posted some sunspider tests ... but I don't like this test alone: speed is something that will be ruined by exceptions and errors handling.

So I ran the test262 on my dev alpha and on my desktop ( W7 Intel Quad Core Q6600 2.4Ghz + 3Go RAM) browsers : FireFox 16.0.2 and Google Chrome 23.0.1271.95 m

test262 is a test suite intended to check agreement between JavaScript implementations and ECMA-262, the ECMAScript Language Specification (currently 5.1 Edition). The test suite contains thousands [as of date : 11571 ] of individual tests, each of which tests some specific requirements of the ECMAScript Language Specification.

Initially, I just wanted to compare compliance ... but ... as you can see, this is a huge set of tests and I ran them in // to observe speed (don't expect a chrono, I'll give rough numbers, cf. my introduction)

Don't want to bother reading the following ? 2 words : amazing BB10.

1/ Compliance

Firefox 16.0.2
firefox16.02.jpg

Chrome 23.0.1271.95
chrome.jpg

BlackBerry10 Browser
IMG_00000017.jpg

Basically, BB10 Browser is above with only 8 failed tests ... better than Chrome, nothing less.
Why Chrome base is important ? Because this test is based on the Sputnik test that was created by ... Google.

Now you tell me ...

2/ And the perfs ?

On the first run, I had the devAlpha and Firefox.
Hit the start all buttons together ... wouch ! in the 30 first seconds, Firefox (on my PC) seemed to be something that 3X quicker than the BB10 Browser.
But after a while (maybe 3-4 minutes), the gap was only a 2X factor.
Then Firefox hit numerous errors ... while the devAlpha was running smoothly with 0 errors ... and the gap wasn't a X factor anymore ... almost catched up Firefox until the end of the test.
When FF finished, DA was at 80%.

After that, Chrome ran smoothly the tests, and it keep the 2-3X factor all way long. Back to normality : my PC pees further than my DA (this rude/poor image is a private joke). CPUs expert may want to compare and give an theoretical performance ratio ... just sounds "fair" to me. Once again, this has no real sense ...

3/ So what ?

For connected devices, compliance IS the first key. You have to remember that each access to the web (to load specific fragments of code, to handle errors messages/pages and/or logs) is a deep hole in overall performance. Sure it's not enough, it's just mandatory before the "engines" get polished and optimized. I've never seen a beta OS/Browser being that much impressive and I'm certain we've not seen it all.

[missing_K-W]

BB 10 is going make the others look antiquated

[mikeo007]

Just like everyone said in the other thread, compliance on random standards means almost nothing. Even more so on tests that are admittedly unfinished or reporting false positives.

Few consumers have ever cared about these standards up until now, and it's likely going to stay that way.
RIM needs to fix this instead of worrying about random standards and draft specifications:

[Superfly_FR]

lol.
Sunspider : "This benchmark tests the core JavaScript language only, not the DOM or other browser APIs". Basically, in 2012 = test the car engine without the car.
Random standards : ECMA ... roflol. http://www.ecma-international.org/

[mikeo007]

lol. Sunspider : "This benchmark tests the core JavaScript language only, not the DOM or other browser APIs". Basically, in 2012 = test the car engine without the car.

Your browser is only as fast as its slowest part.

[Sith_Apprentice]

Well I did a bit of a chrono test and the results (estimated on completion since they were completing fairly quickly at parts) are here:


Time - Completed - Failed
2min - 1100 - 0
3min - 2250 - 0
4min - 3520 - 0
5min - 3694 - 0
6min - 3746 - 0
7min - 4580 - 0
8min - 5680 - 0
9min - 6700 - 0
10min - 7710 - 0
11min - 8600 - 0
12min - 9410 - 0
13min - 10401 - 0
14min - 11203 - 8
14:24min - 11571 - 8


Failed tests are:
15.5.4.9_CE - Tests that String.prototype.localeCompare returns - when comparing Strings that are considered canonically equivalent by the Unicode Standard
S15.8.2.13_A24 - Checking if Math.pow(argument1, argument2) is approximately equals to its mathematical value on the set of 64 argument1 values and 64 argument2 values, all the sample values is calculated with LibC
S15.9.3.1_A5_T1 - 2 arguments, (year, month)
S15.9.3.1_A5_T2 - 3 arguments, (year, month, date)
S15.9.3.1_A5_T3 - 4 arguments, (year, month, date, hours)
S15.9.3.1_A5_T4 - 5 arguments, (year, month, date, hours, minutes)
S15.9.3.1_A5_T5 - 6 arguments, (year, month, date, hours, minutes, seconds)
S15.9.3.1_A5_T6 - 7 arguments, (year, month, date, hours, minutes, seconds, ms)

[Sith_Apprentice]

Just like everyone said in the other thread, compliance on random standards means almost nothing. Even more so on tests that are admittedly unfinished or reporting false positives.

Few consumers have ever cared about these standards up until now, and it's likely going to stay that way.
RIM needs to fix this instead of worrying about random standards and draft specifications:

http://forums.crackberry.com/attachm...-dev-alpha.jpg

For the record, I receive a score on the Dev Alpha B that is about 1490ms, Dev Alpha A scores around 1520.

[mikeo007]

For the record, I receive a score on the Dev Alpha B that is about 1490ms, Dev Alpha A scores around 1520.

What's the HW difference between the 2 units? Is the Dev B running a 4460?

[Sith_Apprentice]

Just like everyone said in the other thread, compliance on random standards means almost nothing. Even more so on tests that are admittedly unfinished or reporting false positives.

Few consumers have ever cared about these standards up until now, and it's likely going to stay that way.
RIM needs to fix this instead of worrying about random standards and draft specifications:

http://forums.crackberry.com/attachm...-dev-alpha.jpg

Also, compliance IS important in terms of what is supported and what is not. Performance is also important, but if you perform well on something that is never or rarely used, it is equally useless. There should be some sort of mix between the two. Tests like these show where the browser *can* perform and where it cannot. It doesnt show the quality of performance, just a binary yes/no.

[Sith_Apprentice]

What's the HW difference between the 2 units? Is the Dev B running a 4460?

Spec Shootout: Dev Alpha A vs Dev Alpha B | CrackBerry.com

[notfanboy]

GS3 using Dolphin.


I'm impressed by the BB10's browser high compliance score, however I would be more impressed with how it does on real web pages.

I suppose you have tried it on some actual websites? Would be interested if you shared those findings. It's understood that it is still running a pre-release version.

I tried out those seven tests on my iPad2/iOS5 and it performed quite well when it was able to run the test.
1. blank page
2. 26fps
3. 60fps
4. 25fps
5. did not render the kaleidoscope
6. n/a (Flash)
7. did not run

[Superfly_FR]

GS3 using Dolphin.
http://i.imgur.com/EkY6G.png

I'm impressed by the BB10's browser high compliance score, however I would be more impressed with how it does on real web pages.

I suppose you have tried it on some actual websites? Would be interested if you shared those findings. It's understood that it is still running a pre-release version.

I tried out those seven tests on my iPad2/iOS5 and it performed quite well when it was able to run the test.
1. blank page
2. 26fps
3. 60fps
4. 25fps
5. did not render the kaleidoscope
6. n/a (Flash)
7. did not run

Thanks for the Dolphin/GS3 comparison.
Can you please offer a list - say 10 ? - of what you believe is "actual websites" ? I'll try.

[notfanboy]

Try the ones on this link. (I messed up the URL copy and paste in my earlier post, fixed now).

[Double_J75]

As a point of reference I'd rather see some speed tests of actual pages. Also I'd like to see the PB included in the devices as another reference point.

[Superfly_FR]

Try the ones on this link. (I messed up the URL copy and paste in my earlier post, fixed now).

IE test drive ? can't be compliance anything
Also, cache cleaning is mandatory for this king of tests.
I'll make a list on general audience (real) sites. But sry, for now, I must go back to work.

[notfanboy]

IE test drive ? can't be compliance anything
Also, cache cleaning is mandatory for this king of tests.
I'll make a list on general audience (real) sites. But sry, for now, I must go back to work.

I'm guessing you already tried those test but did not get the desired results. For if you got good results, I really think there would have been a new thread about it already. If my armchair psychology is wrong, then I will apologize.

Anyway, here are some more fun websites to try. HTML5 Gallery

[missing_K-W]

I'm pretty sure the BB 10 browser hasn't reach a point where its being heavily optimized. Eventually it'll freeze and go into high octane optimization IMHO. Anyone know the internal current state of the browser? Is it primarily focused on construction or optimization currently?

[Sith_Apprentice]

GS3 using Dolphin.
http://i.imgur.com/EkY6G.png

I'm impressed by the BB10's browser high compliance score, however I would be more impressed with how it does on real web pages.

I suppose you have tried it on some actual websites? Would be interested if you shared those findings. It's understood that it is still running a pre-release version.

I tried out those seven tests on my iPad2/iOS5 and it performed quite well when it was able to run the test.
1. blank page - loads and is slightly choppy, missing a couple frames here and there. 27/28fps
2. 26fps - on a fairly unstable connection (barely 3G) it takes a while to load and ends up failing on the layers. Fish appear but do not move, water moves behind the fish and no FPS reading. Audio track plays
3. 60fps - 60fps
4. 25fps - 44fps
5. did not render the kaleidoscope - Videos play at 30-35fps but do not rotate or have an aura. mute works
6. n/a (Flash) - Flash works on the device, no file to test Angry birds specifically
7. did not run - renders all images fine, slider works etc

Dev Alpha B test results in bold
For all tests I assumed default settings, and i full screened those that were able. Also keep in mind this is alpha hardware with beta software. The full browser should be significantly better. (the Dev browser does NOT pass the Facebook ring test level 1, it fails on 3 tests while the current London build passes level 1)

[Superfly_FR]

I'm guessing you already tried those test but did not get the desired results. For if you got good results, I really think there would have been a new thread about it already. If my armchair psychology is wrong, then I will apologize.

Anyway, here are some more fun websites to try. HTML5 Gallery

in fact, as I was browsing using firefox, I went to the Aston Martin site ... and hang half way on loading ... then I wondered why
OK, I'll see the other link

Edit : Thanks to sith_Apprentice, you have the results, but I'll jump to your second list ASAP.

[Superfly_FR]

Anyway, here are some more fun websites to try. HTML5 Gallery

- Bainden - tested several parts especially word-illusions. Fluid, no problem
- Two players games online : Single Player/ Sniper Hero. Flash game (enabled during session, no exit/restart) , keyboard required but loads fast and sound clear.
- Adek lots of videos/images/sound : normal browsing, some resize temporary issues (css positioning ?) yet good scaling. Tested peek (to the hub) : working fine. Minimize = stop sound
- Aleksi Liukkonen : portfolio. Not sure there is not a mobile theme applied (should force portrait orientation IMHO), grey svale to color OK, open image ok.

Nothing very tricky on these sites, I believe. I'll search for others later.

[mikeo007]

Spec Shootout: Dev Alpha A vs Dev Alpha B | CrackBerry.com

Oh, so it's running the same 4470 that's rumoured to be in the L-series launch devices.
Those sunspider scores are pretty rough for a CPU that beefy. Not terrible, just not great by any means. I guess we'll just hope that they optimize the engine sooner rather than later.

Does anyone know what JS engine they're building on? Is it the old torch engine, or something new?

[Sith_Apprentice]

Oh, so it's running the same 4470 that's rumoured to be in the L-series launch devices.
Those sunspider scores are pretty rough for a CPU that beefy. Not terrible, just not great by any means. I guess we'll just hope that they optimize the engine sooner rather than later.

Does anyone know what JS engine they're building on? Is it the old torch engine, or something new?

Yeah the Sunspider scores definitely need to be improved. Next time I get my hands on a London I will run that and let you know. This browser is really not optimized at all though, and just used for testing. The browser on the London was much better the last I used it.

[dentynefire]

is it reasonable to expect that when the SDK goes gold on the 11th, that the update will offer the optimized browser?

[Sith_Apprentice]

is it reasonable to expect that when the SDK goes gold on the 11th, that the update will offer the optimized browser?

Potentially, but all these performance enhancements are "necessarily" needed for development efforts as the Dev A/B/C software doesnt have anything directly to do with the London software. Since these devices are built purely for testing, that may be how they remain.

[Superfly_FR]

Potentially, but all these performance enhancements are "necessarily" needed for development efforts as the Dev A/B/C software doesnt have anything directly to do with the London software. Since these devices are built purely for testing, that may be how they remain.

Amen, thanks.