Mystic OS manipulation and fact based discussion

Hey all... I've been trolling around for some time putting my two cents in here and there and have come to the conclusion that, for the mass majority, the forums are populated by the unwashed masses that abound in any forum as one might expect given the widespread proliferation of the internet amongst the arguably under educated English speaking population of the world. Not that I'm a **** of a lot better...

Despite my own shortcomings, I would like to bring light to a disturbing trend in these forums that I call "Voodoo OS management". This is hardly something that's limited to the Blackberry. Driver management and a number of other areas that involve generations of software intermingling are a hotbed of insanity.

To clarify what I find both amusing and disturbing, I used to work as a tech support for an internet service provider. We had a number or "methods" to alleviate a customer's issues (more often than not it was reboot.) I noticed that a number of the situations were not directly addressed and that we would use catch-all approaches that were not specifically addressing the issue at hand. Quite often, the direction just became a rote step by step approach to dealing with any situation without attempting to ascertain what was wrong in the first place. I began to realize that the reason for this wasn't exactly laziness, it was the fact that those doing the troubleshooting had no fundamental understanding of the operating system or TCP/IP in general. These were people that have stumbled into their positions through one way or another and were essentially propagating lies.

I see the same thing happening here all the time. Unfortunately it's nothing that I can expect to change. We'll always have idiots with more time and access than sense that will opine and proselytize their ***** agendas back and forth until even the most straightforward thread is reduced to flaming and moronic attitude. I for one never get past that point. Unless I happen to be lucky enough to click on the last page and get something valid out of the drivel, it's generally a lost cause.

Okay, so now that I've done my soap box routine, I am hoping that I might be able to spawn a thread that is based upon fact. If you agree with me, DO NOT POST A REPLY. If you think I'm an ***... DO NOT RESPOND HERE. You can flame me or praise me in personal posts all you want. I don't guarantee a response, but feel free to do so.

What I would like to know is... what do we know about the operating system in general? There's a number of hybrid OS' about that appear to operate in various degrees of efficiency but there isn't a heck of a lot of specifics.

What I'm curious about is picking apart the individual pieces of the OS. You can utilize a different JVM or specific chunk of an operating system that has been released, but no one seems to address individual binaries or how they affect the system. If this isn't something possible, I'd like to know why. I know a lot is in java but we're typically dealing with byte code and each application must have individual modifications. Why should I be utilizing the radio code for one portion and the apps from another? I understand the radio code is modularized and is probably a good candidate for grouping but when it comes down to apps and basics, it's like drivers. Some code doesn't jive with others because there may be calls that aren't understood or variables passed that weren't there before.

I understand that a lot of this information may not be specifically available and that those who are in the know may not be able to divulge information but the propagation of voodoo ideas hurts us all in the long run. This is a call to informational justice. Let us quell the legend and usher in an era of informative value.

Great post, and I know where you're going but not sure how easy it'll be to get there.

My initial thoughts are for some of the more prominent application developers that browse crackberry.com to come forward and start discussing aspects of the various API interfaces that are available for use by third party applications.

Whilst this won't give us the depth of insight that I think you're hoping we collectively attain, it might be a place to start without looking into the byte code itself. The problem with this approach is that many of the components that are of most interest are likely 'closed' with no published methods for third party manipulation which is going to make it difficult to look deeper into what the components do and how they interact at the hardware level.

We're also lacking (at least from what I've read) a good technical 'roadmap' of the device's hardware. I recall an interview with a gentleman at RIM where he apparently pushed a single computer chip towards the interviewer, commenting that it was the 'most advanced smart phone ever created to date'.. to which the reviewer countered that without a functional and reliable user interface, it was nothing more than an unusable technological wonder.

So to respond to your question, I'd like to ask if you have any thoughts on where to begin? How do we even start to address the task that you're challenging us to comprehend.

Any thoughts are most welcome. :)

- Ray

Edit: Afterthought -- I would love to get to the point where the now-familiar 'battery pull' is a thing of the past.

Deep.... Very Deep

i wonder how many people who have put together hybrids and claim it to be their own will post here and try to explain what they already dont have a clue about.anyone can take a ready built OS and piece it together with other stuff. but to go where this post is going..well.. this should seperate the pros from the pretenders. i admittedly am just a average user with no real knowledge of any of this beyond what i read here but at least i can admit that. cant wait to see some of the responses and who writes them. should be interesting.

If the OP is talking about what I think he's talking about (basically writing a custom OS) wouldn't that require knowledge of the OS on a level that no one outside of RIM has access to? Or have the OS sources been leaked, too?

EDIT: I ask, because i'd think it would be an extremely tedious process to reverse engineer the byte code..... Although my programming skills are limited to some C++/VB 15 years ago....

So, while I don't know a lot about the Blackberry operating system in general as far as Java byte code goes: it's relatively simple to reverse engineer. The problem comes when dealing with blackberry's .cod files. I haven't tested this but I have a suspicion that they likely use some form of encryption to avoid this exact type of thing.

Edit: Actually, reverse engineering has been done on .cod files. Unfortunately, this is the part of the OS I'm least interested in looking at.

Unfortunately I am not personally able to live up to the aspirations of my post... one of the reasons I've put this out. I believe the largest stumbling block is the closed nature of our beloved smartphone. I've mentioned in previous posts how I would like the ability to utilize more than Java for development, however this would likely require transparency that RIM naturally shies away from. In essence, I would hope that this might spurn some reflection upon what we know as fact vs. what we have adopted as urban legend. Unfortunately the majority of content on this board is based upon conjecture and vapid responses.
Don't get me wrong... they all have their place, but you can't actually validate any of this information without an idea of how the various modules interact. While a certain amount of information will inevitably remain a corporate secret, if RIM is to truly thrive in this information age, it might behoove them to provide a bit more transparency to harness the power of the highly intelligent user base that they have gathered. True, the majority of users aren't likely to contribute in a fashion more than the beta tester that they appear to encourage, however there is a large segment of their captive audience that could bring a lot more to the table if allowed to do so. Perhaps this will never happen... They want the government to be able to trust the Blackberry as a platform that will not be hijacked but considering they won't let our Commander-In-Chief utilize the BB without considerable restrictions, it's hardly a hardened military secret.
Sooo, that being said, I would like to see where we stand. What do we actually know? I'm sure a lot of things will change when we move to OS 5, however the underpinnings are unlikely to be wildly different. I'm still a Blackberry newbie, but I've got experience that makes me question the approach that we ahve all been taking to this issue. I would imagine that the core applications, despite the tight integration, are relatively modular. If we are having difficulty with the camera application, is that issue rooted in the handling of the hardware? It is rooted in the calls that the OS utilizes to communicate with the hardware or is it in the implementation of the application? In the various implementations of the OS, it seems that typically it's two steps forward, one step back. Is this due to the fact that when the code for one portion is modified, it breaks ancillary code? I would believe that, for the sake of efficency I would think that modularity would be more appropriate. Of course as things change it may become necessary to pas more and.or different variables and data which would necessitate changes in the related applications.
This is one of my major concerns about the "mystic" approach. We can throw things together that are not able to communicate appropriately and, while it's most likely it just won't work, it's quite probable that one module will end up passing on data that another interprets as what is expected and no one is the wiser until the thing goes belly up. I would expect this to be especially likely utilizing Java wherein (at least in my limited experience) you can essentially shove a round peg into a square hole with unexpected results. That's one of the beautiful and yet equally tragic pitfalls. If someone out there is more adept at Java, I would love to hear your comments as my limited understanding could lead to yet more misapprehension that we need to attempt to avoid. I want nothing more than to examine the foundations upon which our suppositions are based prior to moving forward.
Well my friends... what say you?
~Nate

Actually, if encapsulation is being properly used these modules should be relatively interchangeable. If the JVM changes or functionality is added to their API using a newer .cod file on an older OS would have unpredictable results. This is not always the case in reality though and would have to look at the actual code to determine. I do have to mention, I'm not excellent with Java but this has been my experience with other languages.

My god...This is the best first post in any topic ever.

I wish I could break it down and explain. Sadly I'm a youngster who had yet to reach that level of talent.

All of the recent device versions use version 6 COD files, devices using the early 4.2 OSes and before use version 5 COD files, which are backwards compatible. There are a handful of OS libraries that still use version 5 CODs (like the SecurID library).

CODs aren't encrypted, or really even obfuscated. They are just an efficient and compressed packing of Java class files to remove the mass duplication of data typical in say, a JAR. Google had similar objectives with the Dalvik VM used in Android, but Google decided to ditch Java bytecode for a new register-based instruction set (the merits of this can be contested).

The RIM operating system also includes 774 native API calls to firmware (some sort of JNI). The firmware is written in C, and handles everything from controlling hardware access, poking the SIM card, to including media player functionality (evidently the JVM overhead is too large to do decoding in Java, or it is too power inefficient to do so).

Operating system modules are signed using the RIM "3" authority to prevent people from modifying those modules and hacking together their own OS. You can mix and match different signed versions of these modules in those hybrid operating systems, but unless you disable code signature checks on your device, you won't be able edit those modules. I don't know if firmware images are signed and checked by the boot ROM, but it seems likely (not that it really matters, since large scale modification of the firmware without the C source is a huge engineering effort).

Enough factual information to satisfy you?

yea that was good. havent got a clue what you said or what language you used to type it(lol) but it made sense if that makes sense.

Originally Posted by Thyth

All of the recent device versions use version 6 COD files, devices using the early 4.2 OSes and before use version 5 COD files, which are backwards compatible. There are a handful of OS libraries that still use version 5 CODs (like the SecurID library).

CODs aren't encrypted, or really even obfuscated. They are just an efficient and compressed packing of Java class files to remove the mass duplication of data typical in say, a JAR. Google had similar objectives with the Dalvik VM used in Android, but Google decided to ditch Java bytecode for a new register-based instruction set (the merits of this can be contested).

The RIM operating system also includes 774 native API calls to firmware (some sort of JNI). The firmware is written in C, and handles everything from controlling hardware access, poking the SIM card, to including media player functionality (evidently the JVM overhead is too large to do decoding in Java, or it is too power inefficient to do so).

Operating system modules are signed using the RIM "3" authority to prevent people from modifying those modules and hacking together their own OS. You can mix and match different signed versions of these modules in those hybrid operating systems, but unless you disable code signature checks on your device, you won't be able edit those modules. I don't know if firmware images are signed and checked by the boot ROM, but it seems likely (not that it really matters, since large scale modification of the firmware without the C source is a huge engineering effort).

Enough factual information to satisfy you?

I believe you are correct sir. The signature checks are most likely what prevent most forum "experts" from creating a Kitchen. There isn't a way to modify the files without the device knowing. I previously attempted to find the OTA files and edit the location they searched for new updates to no avail....I was hoping that somehow by doing this we could setup a host that the phone would search and we could get OTA leaks.

This is the best Blackberry or Blackberry OS post in the history of Mankind. LOL. I am so sick of people responding to any question with an offhad "It can't be done" or "It's impossible' that is not based on any true background knowledge. I have also learned to not go by appearances. I work in I.T. for a rather ginormous airline in the southeastern U.S. Growing up in this city I had a certain image of said airline and therefore was rather shocked to find out upon being emplyed with them that they have the same messed up corporate I.T. as everyone else. The same gos for RIM I would suppose. I am betting they are not as infallible and their OS is not as unhackable as people make it out to be. One of the big problems I would love to resolve is configurability of the TCP/IP on CDMA Blackberries. It seems that RIM somehow is hiding the TCP menu on CDMA Blackberries. So if you try and flash your Berry to another provider (Say from Sprint to MetroPCS) you have no way of obtaining internet on your new provider's network. The standard answer I have gotten by people who know far less than I (and I dont know much) is that "It cant be done" and "It's hard coded". I am one of those people who doesnt readily accept what I am told. Not without putting some sort of science to the claim. It's nice to see like minded individuals here on this thread.

Originally Posted by Thyth

All of the recent device versions use version 6 COD files, devices using the early 4.2 OSes and before use version 5 COD files, which are backwards compatible. There are a handful of OS libraries that still use version 5 CODs (like the SecurID library).

CODs aren't encrypted, or really even obfuscated. They are just an efficient and compressed packing of Java class files to remove the mass duplication of data typical in say, a JAR. Google had similar objectives with the Dalvik VM used in Android, but Google decided to ditch Java bytecode for a new register-based instruction set (the merits of this can be contested).

The RIM operating system also includes 774 native API calls to firmware (some sort of JNI). The firmware is written in C, and handles everything from controlling hardware access, poking the SIM card, to including media player functionality (evidently the JVM overhead is too large to do decoding in Java, or it is too power inefficient to do so).

Operating system modules are signed using the RIM "3" authority to prevent people from modifying those modules and hacking together their own OS. You can mix and match different signed versions of these modules in those hybrid operating systems, but unless you disable code signature checks on your device, you won't be able edit those modules. I don't know if firmware images are signed and checked by the boot ROM, but it seems likely (not that it really matters, since large scale modification of the firmware without the C source is a huge engineering effort).

Enough factual information to satisfy you?

Now, THAT's the ticket... has anyone done any work on figuring out a way to re-sign the modules or disabling the signature checks or is that a taboo subject here? I'd imagine that, at least in theory, it would be similar to re-signing Halo maps.
As to the firmware being written in C, that's an interesting tidbit... If we could figure out how to run unsigned code, there may be some hope to getting ScummVM to run on the storm! I know... probably misplaced priorities, but Monkey Island was an important part of my childhood :P
Sooo, if we could figure out how to wedge something between the boot rom & firmware... I'm not familiar with the details of how the iPhone was jailbroken but I would imagine that, while a large percentage of the code is open source due to the BSDesqe nature, is the firmware code open too? I'm kind of surprised, given the market penetration, that the BB hasn't had more kernel hacker types chipping away. Granted it is geared towards those looking to use it as a means to an endrather than as the end itself.

Originally Posted by psavach

So, while I don't know a lot about the Blackberry operating system in general as far as Java byte code goes: it's relatively simple to reverse engineer. The problem comes when dealing with blackberry's .cod files. I haven't tested this but I have a suspicion that they likely use some form of encryption to avoid this exact type of thing.

Edit: Actually, reverse engineering has been done on .cod files. Unfortunately, this is the part of the OS I'm least interested in looking at.

If I remember right weren't the main files *the actual core of the OS* md5 hashed and would be check each time the OS boots by something else? You would have to bypass the checker and most likely create a way to inject files on the file with a reboot.

I'm glad I went looking for this thread. It is truly a superb exchange :) I wish I had the smarts to contribute something worthy.