[Bratza]

Since BlackBerry quit using BIS/BES on BlackBerry 10 devices, they started using Microsoft Exchange Active Sync (MS EAS).

This has been confusing me for a while, and I was not able to find the correct answer:

Is EAS true push technology in the same way that BES is? Are there any delays in transmiting data? Is EAS technology battery efficient the same as BES is? Isn't EAS kind of idle connection delivery method, opposed to BIS/BES push?

Could someone please explain?

Thanks!


Sent from my BlackBerry 9780 using Tapatalk

[abramadhi]

I can't explain you the technical details but most of the time i received my emails in my device first and a couple of seconds later my laptop receive the email.

So yeah, i think it's still true pushed.

Posted via CB10

[AnimalPak200]

I don't think "Blackberry uses EAS" per se, although I imagine it can if configured for a particular account.

They implement push notifications via a Push Notification Server even though they no longer route all the traffic through BIS servers.

http://devblog.blackberry.com/2012/10/blackberry-push/

Posted via CB10

[so crow]

I can't explain you the technical details but most of the time i received my emails in my device first and a couple of seconds later my laptop receive the email.

So yeah, i think it's still true pushed.

Posted via CB10

Same.

Posted via CB10

[talberry]

Agreed.. there's even the option to choose if you want your emails pushed or synced at different intervals.

Posted via CB10

[Bobb1138]

But something is wrong with the push, I have turn push off, but still get mails pushed.... I have tried several settings with push on/off, different interval, but still get the mail when it hits our server...
/bob
Posted via CB10

[Omnitech]

But something is wrong with the push, I have turn push off, but still get mails pushed.... I have tried several settings with push on/off, different interval, but still get the mail when it hits our server...

Some people noted that behaviour on recent OS 10.1 builds. I think there is still some "strangeness" in the email functionality in those builds.

I also think that this may be related to some of the networking/communications changes I assume they made in those builds to address the self-reboot problem a lot of people were having.

For the lowdown on email retrieval and syncing and "push" on Blackberry 10, please see the following:

http://forums.crackberry.com/blackbe...yncing-777774/

[Bratza]

I don't think "Blackberry uses EAS" per se, although I imagine it can if configured for a particular account.

They implement push notifications via a Push Notification Server even though they no longer route all the traffic through BIS servers.

Let

Posted via CB10

Does this mean BB uses a different method, as opposed to Apple/Android does? Is it saving the battery life the same way as old legacy BIS/BES devices, by getting emails pushed by network? Or is it a simple IDLE push?

I would appreciate someone's thorough explanation. Unfortunately, I'm still unable to grasp the concept the right way.

[tickerguy]

Depends on the server.

IMAP uses what is known as the "IDLE" command. If the server supports it then the client may, after it gets done checking email (or doing something else) send the IDLE command to the server.

This tells the server to monitor the folder named (each "IDLE" command is associated with a given folder, such as the INBOX.) When the contents change the server then sends down a short status message and the client knows to request the changes. An IDLE connection is consuming a socket on the server and as such the duration for which the command is valid is at the server's discretion. The server may while "IDLE" send a close command to the client should it time out or if it finds itself short of resources and needs to free them. If it does, the client may re-establish IDLE at its discretion.

During the time the server and client are in IDLE mode TCP keep-alives are relied upon by both ends to notice if something happens to underlying socket connection. Should either end notice that the other has disappeared then the connection is torn down, IDLE mode is exited, and the client is responsible for signing back into the server.

With IDLE notification is effectively immediate as soon as the server is aware of a new email arriving; my Z10 typically has notification and a new email within a couple of seconds of an email being received on my mail server.

Note that some IMAP servers incorrectly implement IDLE. One of the more-serious offenses is for a server to SILENTLY drop an IDLE connection. Since a server MAY terminate an IDLE connection at any time it desires but is supposed to notify the client if it fails to send notification but leaves the TCP socket open the client has no way to know that this has happened. This is a rank violation of the protocol but some email servers, particularly large public email servers, will do this and it appears to be intentional.

[hillsider]

Same.

Posted via CB10

Which email service are you using?

[Omnitech]

QUOTE=tickerguy;8737363]

Note that some IMAP servers incorrectly implement IDLE. One of the more-serious offenses is for a server to SILENTLY drop an IDLE connection. Since a server MAY terminate an IDLE connection at any time it desires but is supposed to notify the client if it fails to send notification but leaves the TCP socket open the client has no way to know that this has happened. This is a rank violation of the protocol but some email servers, particularly large public email servers, will do this and it appears to be intentional.[/QUOTE]


I'm curious which particular systems are known to be guilty of this and why you think it's intentional.

Yahoo doesn't support IMAP IDLE, Outlook.com doesn't support IMAP at all and if you're referring to Google are you of the opinion they are trying to push people towards Android or one of their proprietary apps?

[tickerguy]

I think it's intentional because keeping the socket open requires resources. If the server properly notifies the client that IDLE has timed out (which under the protocol it is entitled to do) the client is very likely to immediately send another IDLE request (duh!)

If you're going to not support IDLE properly then don't support it at all (e.g. Yahoo.) Just don't expect anyone who needs email to be delivered "right now" to take you seriously without either a working IMAP IDLE or a working Exchange Push implementation!

If this is a bug its a bad bug and it's gone on long enough that it should be fixed. Oh, and I note that Android's BASE email client (that ships with Android phones) doesn't handle IDLE either (it's polling mode only), NOR does it properly handle STARTTLS for SMTP, which is really nasty. It will work with dedicated SSL on the SMTP listener but nearly all transports that also talk to other things want to run STARTTLS because there are a LOT of MTAs that don't run SSL for transmission for a whole host of reasons. Indeed there's a fairly clean argument for unencrypted email that the only real reason to run SSL for sending in general is so you can authenticate (e.g. log in as a sender to shut off anti-relaying checks) without a password being exposed. Working around STARTTLS being missing isn't that big of a deal but the lack of IDLE support is.

BlackBerry's BB10 client had bugs with linefeed handling in 10.0.9 and 10.0.10, but they're fixed in 10.1 and the client DOES properly handle both STARTTLS and IMAP IDLE.

I use this very, very heavily and am utterly reliant on it working right on my phone. On Android phones I had to load K9Mail to get it to work at all.

[Omnitech]

I think it's intentional because keeping the socket open requires resources. If the server properly notifies the client that IDLE has timed out (which under the protocol it is entitled to do) the client is very likely to immediately send another IDLE request (duh!)

If you're going to not support IDLE properly then don't support it at all (e.g. Yahoo.) Just don't expect anyone who needs email to be delivered "right now" to take you seriously without either a working IMAP IDLE or a working Exchange Push implementation!

I can understand the resource-conservation motivation but if you tout a feature, the feature should work - unless you have either some bizarre "kick me" instinct, or you are trying to push some other commercial agenda - ie your proprietary apps - or you are just plain sloppy.

Re: push on Yahoo, it exists. It's a proprietary system that uses some sort of UDP notification. Blackberry supports it with a "BIS plugin", but not on BB10. Just like they support various Gmail features using the old Gmail BIS Plugin that they don't support on BB10. (ie remote search)

Yahoo also has specific support for Apple's proprietary push notification system. You can see evidence of this if you issue an IMAP capabilities command, or just in the IMAP login banner:

* OK [CAPABILITY IMAP4rev1 ID NAMESPACE X-ID-ACLID UIDPLUS LITERAL+ CHILDREN XAPPLEPUSHSERVICE XYMHIGHESTMODSE
Q LOGINDISABLED AUTH=XYMCOOKIE AUTH=XYMECOOKIE AUTH=XYMCOOKIEB64 AUTH=XYMPKI STARTTLS] IMAP4rev1 imapgate-0.7.68_14.446672 imap405.mail.ne1.yahoo.com

(FWIW - Blackberry has its own proprietary push system for BB10, but I doubt they have enough marketshare at this point to get major providers like Yahoo to adopt it yet.)

Oh, and I note that Android's BASE email client (that ships with Android phones) doesn't handle IDLE either (it's polling mode only), NOR does it properly handle STARTTLS for SMTP, which is really nasty.

I think the way most people get around that with Android is just to run a 3rd-party email client, if they need that. Of course Google has a vested interest in making their email service appear to be more attractive than the competition.

Re: STARTTLS, I don't know where that trend started (perhaps Microsoft email clients?), but it seems to be a trend of some kind. Perhaps SSL libraries with amenable license terms are easier to find than libraries that also include STARTTLS?

BlackBerry's BB10 client had bugs with linefeed handling in 10.0.9 and 10.0.10, but they're fixed in 10.1 and the client DOES properly handle both STARTTLS and IMAP IDLE.

I have seen conflicting reports on this. Some people seem to still have issues with SMTP on even fairly recent 10.1 builds. Though I do see fewer reports of it lately.

http://forums.crackberry.com/blackbe...rsions-779341/


Then of course you have some of the various other email issues, like the "encounter transient connectivity or auth issue, delete password and silently disable account" issue. (Though it appears to be slightly less "silent" on more recent FW builds.)

[tickerguy]

Re: STARTTLS, I don't know where that trend started (perhaps Microsoft email clients?), but it seems to be a trend of some kind. Perhaps SSL libraries with amenable license terms are easier to find than libraries that also include STARTTLS?

No, STARTTLS is simply SSL that is initiated by the STARTTLS command. That's all. It's trivial to support; there is no licensing issue. My Spamblock intermediary supports it and uses OpenSSL to implement it so I am intimately familiar with how it works....

UDP notification is a problem to mobile clients (or anyone behind a NAT gateway for that matter) as unless the gateway maintains stateful caching it won't work. Most gateways that DO cache UDP mapping only do it for a very short period of time and are rather unsuitable for a mobile client, especially for sparse messages (which this sort of use is) for that reason as the maps will get dropped and then you're screwed as you won't get the notifies. It would work going to BIS because a carrier BIS server is going to be on a fixed address and directly attached, and thus not behind a NAT device. The server side of such a thing of course loves it because there's nearly zero overhead involved and no per-client structure required; you just maintain an open socket and issue UDP packets down it to the IP addresses of your connected client machines, able to serve an effectively infinite number of clients off one server socket resource. If the clients disconnect between actual mailbox accesses then the server overhead concerns become far less troublesome. This only comes into play for very large installations, of course.

I wonder what Apple's "protocol" actually looks like under the hood; it ought to be entirely possible to reverse-engineer it from Yahoo's server with a bit of effort by executing the command and then looking at what comes down the pipe. I bet it's rather simple -- leave it to Apple to do something strange rather than just use IDLE, which works perfectly well.

As for problems with 10.1, my money is on the server side being screwed up if people are having trouble with it. I write this sort of code (SMTP and MUA "talkers" and "listeners" such as IMAP) for a living and have for over a decade and I have no errata against the current 10.1 firmware at all. The previous stuff interoperated against my servers without problems but did throw warnings about the linefeed problems. Those people who were getting garbled emails were getting them because their SERVERS were running off the end of internal buffers -- it wasn't the client emitting trash.

[Omnitech]

No, STARTTLS is simply SSL that is initiated by the STARTTLS command. That's all. It's trivial to support; there is no licensing issue. My Spamblock intermediary supports it and uses OpenSSL to implement it so I am intimately familiar with how it works....

My assumption is based on the fact that some vendors may not want to put open source code in their products due to the code-sharing requirements of the license (I don't know what OpenSSL's license is, I assume it is a BSD license or some variant of that, which is definitely more friendly to commercial closed-source implementations than the GPL), and/or the fact that I assume various easily-available SSL libraries (ie bundled with MS Windows) probably are designed around web implementations, not email implementations, so may not have any reason to include STARTTLS functionality.

UDP notification is a problem to mobile clients (or anyone behind a NAT gateway for that matter) as unless the gateway maintains stateful caching it won't work. Most gateways that DO cache UDP mapping only do it for a very short period of time and are rather unsuitable for a mobile client, especially for sparse messages (which this sort of use is) for that reason as the maps will get dropped and then you're screwed as you won't get the notifies. It would work going to BIS because a carrier BIS server is going to be on a fixed address and directly attached, and thus not behind a NAT device.

Well obviously the reason this works for BIS is because all that infrastructure is essentially a proxy for the email services on the handheld devices, and run 24x7.

But that model falls apart when the handheld device is the actual endpoint of the email traffic, rather than simply a client of a server-based system acting as an email proxy server more or less.

In the case of Apple, I assume they have their own push infrastructure (not unlike Apple's version of BIS, absent the out-of-band signaling system, I would guess) logging-in to Yahoo's servers and doing the email checking, then pushing notifications to their iDevices. Which is probably why Apple is the only major smartphone platform these days (other than legacy BBOS) that gets "push" email notifications from Yahoo using their native email client. (As you mention, Android doesn't since its native client doesn't support IMAP IDLE, and neither does Windows Phone 8 support it natively)

I wonder what Apple's "protocol" actually looks like under the hood; it ought to be entirely possible to reverse-engineer it from Yahoo's server with a bit of effort by executing the command and then looking at what comes down the pipe. I bet it's rather simple -- leave it to Apple to do something strange rather than just use IDLE, which works perfectly well.

It's publicly documented.

Local and Push Notification Programming Guide: About Local Notifications and Push Notifications (Apple)

http://en.wikipedia.org/wiki/Apple_P...cation_Service

I would guess the reason they developed it is similiar to the reason RIM did: it does things that can't be done in other ways, and they have a large enough market-share to motivate vendors to get aboard.

I would also argue that IMAP IDLE is not a particularly elegant or sophisticated protocol. Take a look at the Exchange ActiveSync documentation some time if you want to see the contrast. I think it's main advantage is it's simplicity, but that goes along with significant limitations.

As for problems with 10.1, my money is on the server side being screwed up if people are having trouble with it. I write this sort of code (SMTP and MUA "talkers" and "listeners" such as IMAP) for a living and have for over a decade and I have no errata against the current 10.1 firmware at all. The previous stuff interoperated against my servers without problems but did throw warnings about the linefeed problems. Those people who were getting garbled emails were getting them because their SERVERS were running off the end of internal buffers -- it wasn't the client emitting trash.

What was happening with most of those SMTP servers, from what I gather, was that they saw what appeared to be grotesquely malformatted submissions and blocked or sidelined it as spam, essentially. This is why tons of people could not even setup email accounts on BB10 devices, because when you setup a POP or IMAP account on BB10, it attempts to initiate an outgoing session with the configured SMTP server, and if BB10 could never actually get a successful handshake or send a test message, it wouldn't ever enable the "save" button on the account setup screen because it considered the setup to be incomplete/failed.

In my case I have a POP server that I've been running for over 10 years that with BB10 gets the infamous "forgot password - silently disable account" problem all the time. I have NEVER on ANY email client had any sort of issue like this with that server.

And you realize that most of the 10.0.x.x releases up until the very recent ones did not even include a proper UTC offset string in outgoing email messages, right?

To me this is emblematic of the problems I see all over BB10 - it's almost like they fired all the people that wrote and perfected the legacy BBOS email functionality over the years, then had some people (presumably the QNX crew) write an email client that had never written one before. There are certainly some nice aspects to it, but it's also filled with some extremely amateurish flaws.

[tickerguy]

Well, yes, but the CR-LF problem is distressingly common. I'd guess that ~5-10% of the SMTP servers around the net do exactly what the Z10 did. It's in extremely bad form to fail to process that.

As for the date header that's bad too but again the old "be generous in what you accept, strict in what you emit" rule applies here too. My server had no problem eating it.

I don't know that I would deride IDLE much, frankly. I sort of like how it's implemented given what it is supposed to do. I do recognize the resource constraints it can put a server under that has to handle a scadload of clients but without some sort of standard out-of-band signalling (e.g. via SMS) there's all sorts of trade-offs to do it a different way, and all of them are bad in one form or another. Having something self-contained within the IMAP protocol has material advantages over the alternatives.

I ran some detailed diagnostic traces on the SMTP issues when people started reporting them since there was no problem interoperating with my software here. The messages were fine other than the Date header being incorrect and the missing CRs. The problem some servers were having with them largely came from the use of SSL or STARTTLS; when you do that you have to implement your own line buffering because you are then reading from a byte stream (e.g. you can't use the buffered I/O routines), and it appears some of the server folks got more than a bit sloppy in that regard in their code resulting in trashed messages as they were literally running off the end of their internal buffers. That's not the senders fault though -- that should never happen in properly-coded software.

The implications of that sort of bug are very serious on the server side, particularly if the transport is running with elevated privileges. If you can coerce a privileged executable to run off the end of an internal buffer you can potentially stuff a return stack with arbitrary code and break into the machine that way.

That's bad juju.

[Omnitech]

As for the date header that's bad too but again the old "be generous in what you accept, strict in what you emit" rule applies here too. My server had no problem eating it.

Well it may well have accepted the traffic using that grand'ole axiom (RIP Jon Postel), but that doesn't help the fact that recipients end up with email messages with nonsensical / inconsistent date-stamps on them, which are at the mercy of whatever "fixing" random MTAs in the path attempt to impose on such malformatted messages.

I don't know that I would deride IDLE much, frankly. I sort of like how it's implemented given what it is supposed to do. I do recognize the resource constraints it can put a server under that has to handle a scadload of clients but without some sort of standard out-of-band signalling (e.g. via SMS) there's all sorts of trade-offs to do it a different way, and all of them are bad in one form or another. Having something self-contained within the IMAP protocol has material advantages over the alternatives.

The issues I'm talking about include greedy use of resources (ie an open socket for every single monitored folder - EAS does not have this problem), and fragility of its IDLE "push" mechanism via network elements that close TCP sockets prior to the IDLE timeout / re-establishment period. Most IMAP clients seem to try to use a ~30 minute window, but if you are like lots of people with crappy resource-constrained home networking equipment that wants to close sockets after 5 minutes (or even 60-120 seconds, I've seen), then "push" just goes out the window and people have no idea why it's not working.

Whereas with EAS it has a probing mechanism that if the socket gets closed prior to the initial timeout period, it keeps trying shorter timeframes until it starts working. Stuff like that.

I ran some detailed diagnostic traces on the SMTP issues when people started reporting them since there was no problem interoperating with my software here. The messages were fine other than the Date header being incorrect and the missing CRs. The problem some servers were having with them largely came from the use of SSL or STARTTLS; when you do that you have to implement your own line buffering because you are then reading from a byte stream (e.g. you can't use the buffered I/O routines), and it appears some of the server folks got more than a bit sloppy in that regard in their code resulting in trashed messages as they were literally running off the end of their internal buffers. That's not the senders fault though -- that should never happen in properly-coded software.

OK - so if this is the case, why do we not see this problem in the other platforms? Network software development is always going to be filled with lots of "code around common poor implementations", right?

I'm getting lots of crap now from a client because their (older) email server rejects messages sent from dimwits who think the "TO:" field is designed to put 200 recipients of a mailing list into, and who apparently have never been taught what a BCC: field is. The email server rejects the message since such messages are technically malformatted as per RFC822, but in the intervening years of clueless email users, servers have started to relax their expectations on that mostly because people have demonstrated that they could not be trained not to do that. (I believe the language in RFC2822 is a little more forgiving as well) And yes we are in the process of replacing that server but I just wanted to use it as an illustration.

The implications of that sort of bug are very serious on the server side, particularly if the transport is running with elevated privileges. If you can coerce a privileged executable to run off the end of an internal buffer you can potentially stuff a return stack with arbitrary code and break into the machine that way.

That's bad juju.

Well if it really is a buffer overflow issue I'd agree with you, but these days lots of MTAs are downright militant about email formatting and reject all sorts of otherwise innocuous stuff. I've seen high-profile/popular email security services or server implementations that tack on a bunch of "spam points" when an email starts out with "Dear [recipient]:", or when they see certain X-Mailer strings simply because they think that some spammers have a habit of using X-Mailer strings from (ie) Outlook Express or Pegasus Mail, etc. Providence help you if you happen to legitimately use such MUAs.

FWIW some of the services that people had the most trouble with outgoing SMTP from BB10 devices (even with 10.1 releases) have been GoDaddy and Network Solutions. Not exactly unknown providers.