Question For Those In The Know...what Does It Take?!

This question is targeted directly to those who test the beta OS's for RIM and/or VZW or have direct knowledge:

I find it funny to sit back and watch flamers hee and haw about what will be made official, the Gold Code, the newest known builds and future builds. First of all, I find it very interesting that people put so much stock in what THEY, individually, find using a leaked OS. i.e they claim to have horrible battery life or mms issues and therefore claim that the OS is garbage and will never be made official. In my limited knowledge of the scientific method, isn't sample size extremely important?! For one person, or two or 20 to claim they are having an issue out of hundreds of thousands or millions is pretty marginal.

I have always been of the opinion that you can't create a perfect OS for millons of different people with millions of different applications and configurations on their devices. Glitches will happen, people will have some problems from time to time. So I guess my question is, does RIM share that same view?

When RIM sends out a build for beta testing, is there a success rate they are looking for or will they scrap it altogether if that rate isn't met? And for those testing for VZW, can the same be said for VZW? Do those success rates need to be within the same rough area for RIM and VZW testers? And I think the most important question is what is that success rate RIM or VZW is seeking? 90%? 95%?

I asked my buddy and the abbreviated response I got was,
"The test generally has a script to follow to create potential bugs. The main concern is being to duplicate the issue. Success rate is kinda out the window. If one person can duplicate the issue, and its serious enough to impair the use of the device, generally grounds for immediate fail. IE mms forwarding on .113. "

Posted from my CrackBerry at wapforums.crackberry.com

That makes sense. I think my hang-up is sample size still. If just two people are able to "duplicate" a bug or glitch, then that is enough reason for a build to be a "dud"? I understand there probably isn't a specific rate of success by percentage, but if such a small section of testers have an issue it can't be overlooked? When you do the math, and I can't, if you have 1 million people using a particular device, and all of those users have different configurations and third party apps, then there will be a ridiculous logarithm of possible outcomes when it comes to how a OS reacts with said device. I still wonder if RIM expects every OS to work perfectly no matter?

Originally Posted by AppStateAddict

That makes sense. I think my hang-up is sample size still. If just two people are able to "duplicate" a bug or glitch, then that is enough reason for a build to be a "dud"? I understand there probably isn't a specific rate of success by percentage, but if such a small section of testers have an issue it can't be overlooked? When you do the math, and I can't, if you have 1 million people using a particular device, and all of those users have different configurations and third party apps, then there will be a ridiculous logarithm of possible outcomes when it comes to how a OS reacts with said device. I still wonder if RIM expects every OS to work perfectly no matter?

I understand where you are going with the scientific process, but I am not sure it applies when it comes to software testing. The OS code is designed to work in a certain way with the hardware platform, there is no expectation that a piece of code will react differently on what should be an identical piece of hardware. When it comes to 3rd party apps, the OS is designed to respond to api calls made by 3rd party apps in a certain way, any app that makes the same api call, should get the same response from the OS.

I do think your point is relevant when it comes to the manufacturing of the Storm's hardware, an adequate sample size is required to ensure that the hardware itself meets RIM's expectations.

I hope I didn't misunderstand your post, I'm on my 4th glass of Cognac. :D

The big difference is the testing script. If a specific process produces a bug or glitch that can be replicated, that's considered grounds for FAIL! However its up to the carrier makes the ultimate decision. It all has to do with the severity of the bug and its impact on function and services.

And what tank said sorta covers the part I didn't say.

Posted from my CrackBerry at wapforums.crackberry.com

You've also gotta remember that even if only two out of ten testers have a problem that means at lease 200 out of a 1000 will probably have it. Do YOU want to take those 200 support phone calls?

No they need to err on the side of caution, IMHO, especially on things that have the potential to hamper core features of the device like MMS.

I understand what the OP is saying. In software development, sample size is often irrelevant. At least in the wild.

People on here can claim "Phailz0rs" because their a piece of software did XXX. A hundred people can claim the same. However unless the developer can replicate the bug it'll generally be ignored. The problem the developer would have is the lack of a controlled environment. Therefore they can't take anyone's word for it.

RIM has absolutely no idea how many times mr_phailz0rs has drop his phone. Or yanked the USB cable out of his Storm in the middle of a write process. They have no idea what average signal strength the person might have, which would directly affect battery. No debug info showing what processes (or worse, combination of processes) were running to cause his app memory to zero out.

Unless its something that is blatantly obvious or replicable in their labs (ie: a programmer reading these forums says "Oh, I never thought to try that, does and fails), we are probably pretty much ignored as far as our opinion goes.

Originally Posted by kuroshio

I understand what the OP is saying. In software development, sample size is often irrelevant. At least in the wild.

People on here can claim "Phailz0rs" because their a piece of software did XXX. A hundred people can claim the same. However unless the developer can replicate the bug it'll generally be ignored. The problem the developer would have is the lack of a controlled environment. Therefore they can't take anyone's word for it.

RIM has absolutely no idea how many times mr_phailz0rs has drop his phone. Or yanked the USB cable out of his Storm in the middle of a write process. They have no idea what average signal strength the person might have, which would directly affect battery. No debug info showing what processes (or worse, combination of processes) were running to cause his app memory to zero out.

Unless its something that is blatantly obvious or replicable in their labs (ie: a programmer reading these forums says "Oh, I never thought to try that, does and fails), we are probably pretty much ignored as far as our opinion goes.

Not just prety much, we ARE ignored. They can't get any usefull feedback from us because by the time it leaks they are a dozen or more builds ahead of that one and the bug we find has almost certainly been found and fixed.

Completely agree with everything being said so far. And I definitely would understand why RIM would drop the axe if 2 out of 10 people were able to duplicate a problem. But what if 2 out of 100 or 2 out 200? You then have a situation where 98% or 99% of users are having no issue at all, and that is where I wonder how RIM makes a judgement call because that seems like a high rate of success to me.

And for the record, I would love to see a proper procedure checklist for testers. If everyone on here used that same checklist instead of the typical "pros and cons", some real feedback might occur. I put no stock in what is said in those feedback threads for the leaked OS.

Well i can't tell for RIM what they is the margin they can allow for failure when releasing a build but i can tell that for us (i work for Telus so i will mainly talk of Telus here) carrier that we want the best result for everyone. Like the MMS issue we need to look at the percentage of user that REALLy use this function, and the browser speed how mainy complaint we got on the amount of unit we have solf until now? That's why we are testing so many build and that sometimes even the one we release is still a bit buggy. The main point is, will this build fit the normal user, what i mean by normal user is the common day user the one that use his phone for standard task and that fit in the biggest percentage of the basic use of the phone that's why we are seeing so many build. It's because we try at each time to get more and more tye of user in the same build look at .65 when the storm came out it was **** but **** dude i can tell you it was better than what we had before. telus wanted to release an os as stable as possible and as quick as posible so it can be use by the basic user.

That's why so everytime we try to improve and each new build we get each time we focus on something new so that more and more user get implicated in this process.

And now about the gold code, it will never really exist, well for some maybe .146 will be a great release maybe not! we will never be able to get a perfect OS for everyone everytime we will push something more to the user they will always ask for more... That's why there is people that are no way able to appreciate what they get and i can honestly tell that i'm one of them :P but i can understand that we need to wait and MAYBE one day i will find the build that fit my needs!

I think i can say that i'm pretty sure that any VZW employee will agree with me and surely any Rim employee too but that's the way software developpement works.

I realize my illustration was simplistic but the point stands. They have to know that what they see in the lab and in their testers is but the tip of the iceburg when it comes to what will happen when real world users are using the software.

Take a look at Civic's thread about how to fix the MMS bug in .113. She and a couple others initially state that it works for everybody and initial impressions seemed to be that way but then a flood of people came out and said it did NOT work for them or that they had to repeat the procedure two or three times to get it to work.

I mention this not to criticize Civic or anyone else but to prove a point. You can't just go by initial impressions or just assume that if if something works for a good amount of users that it is good enough to give to the masses.

Originally Posted by VZWRocka

I realize my illustration was simplistic but the point stands. They have to know that what they see in the lab and in their testers is but the tip of the iceburg when it comes to what will happen when real world users are using the software.

Take a look at Civic's thread about how to fix the MMS bug in .113. She and a couple others initially state that it works for everybody and initial impressions seemed to be that way but then a flood of people came out and said it did NOT work for them or that they had to repeat the procedure two or three times to get it to work.

I mention this not to criticize Civic or anyone else but to prove a point. You can't just go by initial impressions or just assume that if if something works for a good amount of users that it is good enough to give to the masses.

Yep that's exactly why we are testing part by part and when the carrier decide that it fit for the user need, they (carrier) will release a new os.

Originally Posted by ruffhunter

Well i can't tell for RIM what they is the margin they can allow for failure when releasing a build but i can tell that for us (i work for Telus so i will mainly talk of Telus here) carrier that we want the best result for everyone. Like the MMS issue we need to look at the percentage of user that REALLy use this function, and the browser speed how mainy complaint we got on the amount of unit we have solf until now? That's why we are testing so many build and that sometimes even the one we release is still a bit buggy. The main point is, will this build fit the normal user, what i mean by normal user is the common day user the one that use his phone for standard task and that fit in the biggest percentage of the basic use of the phone that's why we are seeing so many build. It's because we try at each time to get more and more tye of user in the same build look at .65 when the storm came out it was **** but **** dude i can tell you it was better than what we had before. telus wanted to release an os as stable as possible and as quick as posible so it can be use by the basic user.

That's why so everytime we try to improve and each new build we get each time we focus on something new so that more and more user get implicated in this process.

And now about the gold code, it will never really exist, well for some maybe .146 will be a great release maybe not! we will never be able to get a perfect OS for everyone everytime we will push something more to the user they will always ask for more... That's why there is people that are no way able to appreciate what they get and i can honestly tell that i'm one of them :P but i can understand that we need to wait and MAYBE one day i will find the build that fit my needs!

I think i can say that i'm pretty sure that any VZW employee will agree with me and surely any Rim employee too but that's the way software developpement works.

I couldn't have said it better myself; in broken English that is......

Originally Posted by k0ugar

I couldn't have said it better myself; in broken English that is......

yep dude i know my english suck but i still know that you did understand my point :P!