Be Aware, Exchange, Ridiculously Large Email Stores and BB10
- Here's a new one I ran into today, and it wasn't fun.
I have an utterly huge email store going back (literally) to the 1980s. It is all visible under my desktop Thunderbird client via Dovecot (a really amazing IMAP server, by the way, that can handle ridiculous levels of load and I have deployed at clients with huge user bases and even bigger mail stores.)
Anyway, I recently implemented an Exchange connector front end to this since I have clients that want it, but don't want to give up a working email system (don't get me started on Exchange!)
So I did, and life appeared to be good.
Until I pointed my Passport at my native email store.
All of it.
Now I wasn't ridiculous about it -- I only told it to sync the INBOX and a half-dozen folders. But -- it went and got state information for all of them.
What's worse, it kept maintaining that internally on the Passport.
Now the good news is that the phone didn't wig out and crash.
The bad news is that the PIM and Hub CPU and battery consumption went to somewhere north of Mars and stayed there with the server being repeatedly hit for state changes.
In a word, meh.
Not that I think anything else would do better; I run IMAP on my PC and Laptop for a reason, and this is the reason. I can subscribe to a handful of folders for push notification but not have the damn server hitting me up every 30 seconds with this and that, and as a result my CPU usage is nil yet I still have access to everything.
But that's not an option if you want BES management of BB10, nor if you want S/MIME even after 10.3.2. You run exchange or you don't get it, period.
I'm not saying it's good or bad, just saying it is what it is and you need to be thinking about how you're going to compartmentalize email in some sort of fashion if you have utterly huge archives that you want access to. This much I can tell you -- the BB10 phones can't handle being terminals into a mail system with a couple of thousand folders accessible at a given time on Exchange.
Access to this sort of data store works perfectly fine on IMAP without a care in the world but then you give up S/MIME and BES management.
Having to make that choice sucks; BlackBerry needs to open up S/MIME so it works on IMAP servers and having BES manage them would be nice too.chgaida likes this.04-20-15 08:25 PMLike 1 - That is a problem of your active sync implementation. With IBM traveler and exchange you can configure and sync only the folders you want. This is not a device issue.
The active sync server / service has to support active sync 14.1 to work probably with your BlackBerry device.
And as far as I know BES12 support for imap is coming with one of the next releases.
Posted via CB1004-20-15 11:27 PMLike 0 - That is a problem of your active sync implementation. With IBM traveler and exchange you can configure and sync only the folders you want. This is not a device issue.
The active sync server / service has to support active sync 14.1 to work probably with your BlackBerry device.
And as far as I know BES12 support for imap is coming with one of the next releases.
Posted via CB10
There are not a lot of folks out there with a few thousand folders. I'm one of them, and I can get around this by moving those unlikely to be accessed to a different account (thus removing them from the "view" of the mobile) but it's just one of those limitations of Exchange. The phone has no problem on the same mailstore under IMAP because IMAP doesn't pull a catalog until and unless you go to browse the folder list.
There's a secondary problem with extremely large messages in the "Sent" folder, which the phone insists on syncing and cannot be shut off. I can get around that by having a separate mapping for the "Sent" folder on the phone as opposed to desktop clients; this becomes an issue when you start doing things like sending data dumps to clients that are several megabytes in size (or crash dumps, etc.) Since those go in your "Sent" folder and Exchange insists on syncing that if you throw a few of those out from your PC you will get a surprise with regard to what it does on your BlackBerry!
The lack of S/MIME support on IMAP accounts bites, as does the lack of BES support for them on BlackBerry devices. IMAP is a nearly-perfect protocol for mobile devices when it comes to email exchange, and coupled with CalDAV and CardDAV (which BlackBerry supports native) for contacts and calendars it would be pretty simply to link them. You can do it now on the personal side and I have since the original 10.0.10.85 firmware with good effect.
I'm looking forward to a PROPER S/MIME implementation (there is a very severe problem with the current code on the phone in that it emits improperly formatted MIME messages, even though Exchange, when signing or encryption is turned on) and, hopefully, BlackBerry will support IMAP for it as well as adding IMAP + Caldav/CardDav to BES. For those who aren't married to the Microsoft paradigm and back end (think those firms with cloud-based infrastructure rather than large-company in-house deployments) this would be a godsend.04-20-15 11:30 PMLike 0 - Ok... I'll bite. What's wrong with Exchange?
For the record IMAP was not designed for lightweight communication with mobile devices with push notifications. ActiveSync was. Google and others have extended IMAP and bolted on other protocols to make up for its short comings, so obviously it can be made to work, but again that's a retrofit of an old protocol versus one designed for mobile devices from the ground up.
It sounds like your issue is you have thousands of folders, which as you note is not common, and you believe there is a performance issues with your device trying to download the "catalog" of them.
I could see any device needing to download the list of folders so you the end user could choose which ones to sync. I'm unaware of this being an issue with ActiveSync, but then again I am not aware of any customer with thousands of folders in their mailbox.
It also sounds like you front ended your IMAP server that emulates Exchange, and not Exchange itself. If so how do you know the issue isn't with the front end you placed in front of your IMAP server.
Posted via CB10 on Z30STA100-5/10.3.1.272604-21-15 07:12 PMLike 2 - And for the record Microsoft hosts 50 GB mailboxes, which I consider "large email stores", and users don't have any issues using ActiveSync against them. So the size of the mailbox isn't the issie.
Posted via CB10 on Z30STA100-5/10.3.1.272604-21-15 07:14 PMLike 2 - Ok... I'll bite. What's wrong with Exchange?
For the record IMAP was not designed for lightweight communication with mobile devices with push notifications. ActiveSync was. Google and others have extended IMAP and bolted on other protocols to make up for its short comings, so obviously it can be made to work, but again that's a retrofit of an old protocol versus one designed for mobile devices from the ground up.
Exchange has its purposes, but like most things Microsoft overpromises and underdelivers, along with gratuitously breaking things that are otherwise expected to interoperate with that beyond its borders. If all your world is Microsoft none of this matters much at all, but as soon as it's not the issues start to show up.
My point in this was simply that IMAP doesn't come up with an issue such as this because it doesn't have a presumption that knowledge of the hierarchy matters except at the point where the client wishes to ask for it (e.g. to subscribe to a particular folder for synchronization purposes), and then only to the extent the client wants (e.g. "give the client headers only", etc). This also means that it doesn't enforce a sync of anything you do not wish immediate notification on (think the "Sent" folder for how this could go bad on you in a multi-client device, one mail-store environment.)
I've always found the Exchange partisanship amusing given that it's basically a proprietary Microsoft thing, particularly in a world that is supposed to be built around open interchange and IETF standards.... I just wanted to note for the peanut gallery my bemusement at it choking on something that I've handled using standard protocols now for a couple of decades on clients with far less horsepower than today's high end phone devices without so much as a wimper of protest.chgaida likes this.04-21-15 08:37 PMLike 1 - Actually, IDLE preceded Google by quite some time, and it is far more lightweight than EAS for that purpose (immediate notification of new inbound messages.)
Exchange has its purposes, but like most things Microsoft overpromises and underdelivers, along with gratuitously breaking things that are otherwise expected to interoperate with that beyond its borders. If all your world is Microsoft none of this matters much at all, but as soon as it's not the issues start to show up.
My point in this was simply that IMAP doesn't come up with an issue such as this because it doesn't have a presumption that knowledge of the hierarchy matters except at the point where the client wishes to ask for it (e.g. to subscribe to a particular folder for synchronization purposes), and then only to the extent the client wants (e.g. "give the client headers only", etc). This also means that it doesn't enforce a sync of anything you do not wish immediate notification on (think the "Sent" folder for how this could go bad on you in a multi-client device, one mail-store environment.)
I've always found the Exchange partisanship amusing given that it's basically a proprietary Microsoft thing, particularly in a world that is supposed to be built around open interchange and IETF standards.... I just wanted to note for the peanut gallery my bemusement at it choking on something that I've handled using standard protocols now for a couple of decades on clients with far less horsepower than today's high end phone devices without so much as a wimper of protest.
Also I think you are confusing IDLE, which isn't a push notification for Lemonade which is, at least that is how to was explained to me.
IDLE is more lightweight than EAS? What are you basing this statement on? Even if it were true, EAS incorporates everything else needed to sync the contents of a mailbox including calendar and contacts, without having to bolt on other components, so it is a more complete solution:
http://blogs.technet.com/b/exchange/...0/3403386.aspx
Exchange over promises and under delivers? OK...LOL.
As for downloading headers only, go turn that feature on in your ActiveSync account. By default it is set to Full Messages, but you can change that in BlackBerry 10 to be Headers Only or a couple of different size limits:
Given this understanding I don't see how syncing the Sent Items is a problem unless you are trying to sync all items with no time limit, and then that's on you for having years and years in your Sent items and choosing that setting. It certainly doesn't have anything to do with their size since you can control that behavior.
As for Exchange being proprietary... I guess that in the sense that any corporate email solution that isn't 100% open source is. Otherwise Microsoft makes the ActiveSync protocol available to the public, and flows RFCs for other things like SMTP.
So far the only potentially legitimate complaint I see is that you don't like how ActiveSync tries to download your entire folder hierarchy of "thousands" of folders. I can't speak to the potential performance problems as I know no one who tries to manage their email that way for me to analyze and test with.
Posted via CB10 on Z30STA100-5/10.3.1.2726Last edited by HotFix; 04-21-15 at 10:09 PM.
04-21-15 09:52 PMLike 0 - IDLE is more lightweight than EAS? What are you basing this statement on?
Intimate knowledge of the protocols involved and what the back end must process in order to return the requested functionality, including developing code to run against it in both instances, beginning with a very high-volume production environment with many thousands of active users (specifically, my ISP) in the 1990s. I've only been at this for more than 20 years now in a professional capacity. You?
Even if it were true, EAS incorporates everything else needed to sync the contents of a mailbox including calendar and contacts, without having to bolt on other components, so it is a more complete solution.
Microsoft's Exchange puts everything in one box which is great so long as everything's Microsoft and nothing ever goes wrong with it (in which case it all goes wrong at once.) It's not so good when that's not the case for one reason or another (like, for instance, all the world not being a Windows machine on the user desktop end.)
Have a nice evening.04-21-15 10:06 PMLike 0 -
Intimate knowledge of the protocols involved and what the back end must process in order to return the requested functionality, including developing code to run against it in both instances, beginning with a very high-volume production environment with many thousands of active users (specifically, my ISP) in the 1990s. I've only been at this for more than 20 years now in a professional capacity. You?
Except calendars and contacts aren't in your mailbox, of course. Never mind that the protocol to do one is hardly ideal for the other and that there are standards-based ways to do both, with BB10 supporting them since 10.0.10.85 (with dead-nuts reliability too, I might add.)
Microsoft's Exchange puts everything in one box which is great so long as everything's Microsoft and nothing ever goes wrong with it (in which case it all goes wrong at once.) It's not so good when that's not the case for one reason or another (like, for instance, all the world not being a Windows machine on the user desktop end.)
Have a nice evening.
For the record I was editing my comment to provide better clarification (typing a lengthy response on a mobile device is not conducive to proof reading), adding an example of how you could have header only emails with ActiveSync, and examples (such as adding a functional comparison of EAS to IMAP features from the Exchange team Blog) when you had responded. I also believe there is a difference between IDLE and "lemonade" since it appears as if IDLE itself is not a push notification service like EAS.
I am disappointed you decided to turn this into a sword waving contest because I questioned you your statement...
Not that it should matter but I have been working with email (predominantly Exchange) for ~20 years myself, and I currently directly support an environment of over 100K users that spans the globe, and I also indirectly support many other large and diverse environments. One of the previous environments I supported was a very pro Mac and Linux enterprise, and there were no real issues with those platforms using Exchange as the corporate email platform.
Sigh... yes contacts and calendars are stored in a users mailbox in Exchange. I thought we were talking about that since you were referring EAS and Exchange (as it is in the threads Subject line), and not a generic implementation of ActiveSync.
You clearly have a bias against Exchange, and that's fine as everyone is entitled to their own opinion. However if you are going to make statements comparing two components as fact, then be prepared to back them up with data and not you pedigree.
As for using you pedigree as a justification for your statements, I have a hard time trusting the technical word of someone that chooses to use thousands of folders to organize their mail, not to mention keeping 3 decades of email in one mailbox, but then they complain that some email component doesn't work the way they think it should with that uncommon choice of a configuration. I wouldn't blame any email product or protocol for not working well with that configuration. I am glad you found one protocol that does work for you though.
You have a great evening as well.
Posted via CB10 on Z30STA100-5/10.3.1.272604-21-15 10:50 PMLike 0 - I think the mailboxes work well if you stick to 30 days to sync to the phone.
If someone is syncing all of their mailbox and expect anything to keep up I am going to say to get their head checked. Run a remote search and that can work in a pinch.
Assistant is becoming very powerful for keeping up with clients. I can usually type one of my clients into it and I see not only my remember notes, but contacts, emails, attachments, calendar appointments all related to the client. That type of stuff is unique to BB10 I think.
Posted via CB1004-21-15 11:17 PMLike 0 - Thank you Hotfix for your words. I would have written the same if English were my main language.
To add some more things:
We are using IBM Domino and Traveler with BES 12, my maildatabase has about 32 GB and 68000 documents stored in it. I use about 90 folders, and in my opinion even 90 is too much and one day when I find the time I will create a new folder structure with less folders. In my humble opinion less folder (well structured) is more efficient.
Whatever, beside that document numbers I have a well filled calender and about 2500 addresses in my addressbook.
And guess : everything is syncing fine with my blackberry. Even the attachments in my synced sent box are NOT synced to the device, if I need them, I tap on them to get it downloaded. I sync 90 days, but I think 30 days should be enough because remote search works well and if I need an older email I just search for them.
The most sync problems we had during the last years were not issued by the device but on the EAS side, named traveler in our case.
The issues we had were reported to IBM and extremly fast fixed by them.
We always got a separate Hotfix for the issues a day ore two after reporting and a couple of weeks later the official service pack arrived. Well done IBM.
On the other side s/mime does not work with IBM Traveler, but this is something IBM is working on and will be there some day. Notes native encryption is working fine so when your s/mime certificate is imported to your notes.id , you can decrypt s/mime mails. In our case it is not needed because we are using a central s/mime and PGP gateway (zertificon).
Long words short: We support 8900 users with there mailboxes and different syn techniques like EAS, IMAP, POP3, etc etc,
And EAS in most cases is running fine and working for 99% of all users.
Last word: and I already told you in another thread that IMAP support for bes 12 is comming.... :-)
Posted via CB1004-22-15 12:37 AMLike 0 - Taking things out of context (or flatly inventing what was not said) appears to be a sport with some folks.
Having a very large historical mail folder structure is only unusual if you don't have much history in a given organization. In this case the organization is and was literally mine. It isn't often that I have a reason to go back and look at such things, but when I do it's nice to have access to them in a seamless fashion.
As far as "push" notifications go I find it particularly amusing that someone would think EAS is a "true" one and IMAP/IDLE is not. Both paradigms have the same basic problem in a mobile environment, namely that datagram transport doesn't work in the general case behind NAT since it's connectionless. For this reason both abuse a held-open TCP connection for this purpose. It would be nice if that wasn't necessary and some day within the IPv6 paradigm maybe it won't be but that's conjecture for the future.
Being able to push IMAP connections via BES will be nice too, but it's a half-answer if S/MIME doesn't work on them. BB10 appears to "cheat" in the S/MIME context in this regard with Exchange in that it looks like its relying on the Exchange server to split the attachments for the client; since the client itself knows how to do that (it couldn't read HTML-formatted email without that capacity) it's even more-silly to have implemented it this way. Incidentally another bogon in the BB10 email paradigm that you don't care about in a Microsoft-only world is that their client doesn't emit both a plain text copy of the message body and an HTML-encoded copy. Everyone else figured out that this was pretty-much a requirement two decades ago back when NCSA Mosaic first showed up but BlackBerry, well, no. As a result this is what BB10 thinks is a cited article I sent to myself last night from a text email client:
Code:Return-Path: <[email protected]>Received: from localhost (localhost [127.0.0.1]) by fs.denninger.net (8.14.9/8.14.8) with ESMTP id t3LN8bBo082441 for <[email protected]>; Tue, 21 Apr 2015 18:08:37 -0500 (CDT) (envelope-from [email protected]) Received: from localhost [127.0.0.1]; by Spamblock-sys (LOCAL/AUTH) Tue Apr 21 18:08:37 2015 Mime-Version: 1.0 X-client-id: 222 X-mailer: BlackBerry Email (10.3.2.500) Message-id: <[email protected]> Date: Tue, 21 Apr 2015 18:08:30 -0500 Subject: A LOT of employees plan to buy a smartwatch for work (AAPL) From: [email protected] To: Karl Denninger <[email protected]> Content-Type: multipart/mixed; boundary="=_3df410f53ae0bca519a6a93c3c7f55ba" X-Antivirus: avast! (VPS 150421-1, 04/21/2015), Inbound message X-Antivirus-Status: Clean This is a multi-part message in MIME format. --=_3df410f53ae0bca519a6a93c3c7f55ba Content-Type: multipart/alternative; boundary="=_6bba4f969a5da8a604f5ff46864292d5" --=_6bba4f969a5da8a604f5ff46864292d5 Content-Transfer-Encoding: base64 Content-Type: text/html; charset=utf-8 PGh0bWw+PGhlYWQ+PC9oZWFkPjxib2R5IGxhbmc9ImVuLVVTIj48ZGl2IHN0eWxlPSJ3aGl0ZS1z cGFjZTpwcmUtd3JhcDsgd29yZC13cmFwOiBicmVhay13b3JkOyI+QXBwbGUgV2F0Y2ggaXMgYWxy ZWFkeSBoYXZpbmcgYSByaXBwbGUgZWZmZWN0LiBBIGxvdCBvZiBlbXBsb3llZXMgd2hvIGFscmVh ZHkgdXNlIG1vYmlsZSBkZXZpY2VzIGZvciB3b3JrIHBsYW4gdG8gYnV5IGFuIEFwcGxlIFdhdGNo IG9yIHNpbWlsYXIgZGV2aWNlLCBhY2NvcmRpbmcgdG8gYSBnbG9iYWwgc3VydmV5IGJ5IG1vYmls ZSBkZXZpY2UgbWFuYWdlbWVudCBjb21wYW55IE1vYmlsZUlyb24uIEl0IGFza2VkIG92ZXIgMyw1 MDAgd29ya2VycyBhYm91dCB0aGVpciBwbGFucyB0byBidXkgYSBkZXZpY2UgbGlrZSB0aGUgQXBw bGUgV2F0Y2guIDQyJSBwZXJjZW50IHNhaWQgdGhleSBlaXRoZXIgYWxyZWFkeSBvd24gYSBzbWFy dHdhdGNoIG9yIHBsYW4gdG8gYnV5IG9uZS4gV2hpbGUgdGhhdCBkb2Vzbid0IG1lYW4gdGhhdCB0 aGV5IHdpbGwgYWxsIG9wdCBmb3IgdGhlIEFwcGxlIFdhdGNoLCBpdCBkb2VzIGJvZGUgd2VsbCBm b3IgdGhlIGNhdGVnb3J5IG9mIHNtYXJ0d2F0Y2hlcyBhbHRvZ2V0aGVyLiBBbW9uZyB0aGUgd29y ay1yZWxhdGVkIGFjdGl2aXRpZXMgdGhhdCB0aGUgcmVzcG9uZGVudHMgZW52aXNpb25lZCB1c2lu ZyBhIHNtYXJ0d2F0Y2ggZm9yOiBUYWtpbmcgcGhvbmUgY2FsbHMgNTglIFJlYWRpbmcgZW1haWwg NTYlIFdyaXRpbmcgZW1haWwgNDUlIEdldHRpbmcgYWxlcnRzLCBzdWNoIGFzIG1lZXRpbmcgcmVt aW5kZXJzIDQ0JSBBY2Nlc3NpbmcgY2FsZW5kYXIgNDAlIFJlYWRpbmcgZG9jdW1lbnRzIDM3JSBT dXJmaW5nIGNvbXBhbnkgaW50cmFuZXQgMzAlIE9uZSBwb3RlbnRpYWwuLi48YnI+X19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fXzxicj48YnI+aHR0cDovL2ZlZWRwcm94eS5nb29nbGUu Y29tL35yL2J1c2luZXNzaW5zaWRlci9+My9vVE9tTEcyakg3RS9lbXBsb3llZXMtdG8tYnV5LWEt c21hcnR3YXRjaC1mb3Itd29yay0yMDE1LTQ8L2Rpdj48YnI+PGRpdiBzdHlsZT0iY29sb3I6IHJn YigzOCwgMzgsIDM4KTsgZm9udC1mYW1pbHk6IENhbGlicmksICdTbGF0ZSBQcm8nLCBzYW5zLXNl cmlmLCBzYW5zLXNlcmlmOyI+PGJyPi0mbmJzcDtLYXJsPGJyPihvbiZuYnNwO1BEQSk8L2Rpdj48 L2JvZHk+PC9odG1sPg== --=_6bba4f969a5da8a604f5ff46864292d5-- --=_3df410f53ae0bca519a6a93c3c7f55ba--
(Of course it displays just fine in Thunderbird, Outlook or for that matter the BB10 client itself.... just don't try to read it from the command line!)04-22-15 07:20 AMLike 0 -
Having a very large historical mail folder structure is only unusual if you don't have much history in a given organization. In this case the organization is and was literally mine. It isn't often that I have a reason to go back and look at such things, but when I do it's nice to have access to them in a seamless fashion.
As far as "push" notifications go I find it particularly amusing that someone would think EAS is a "true" one and IMAP/IDLE is not. Both paradigms have the same basic problem in a mobile environment, namely that datagram transport doesn't work in the general case behind NAT since it's connectionless. For this reason both abuse a held-open TCP connection for this purpose. It would be nice if that wasn't necessary and some day within the IPv6 paradigm maybe it won't be but that's conjecture for the future.
Push-IMAP - Wikipedia, the free encyclopedia
https://groups.google.com/forum/#!to...ap/7_1q7TYsTzI
From reading those it looks like IDLE is just a component in push notification IMAP, but it itself is not Push. I'm not an expert at IDLE, just telling you what I was told and what I am reading online.
EAS on the other hand, which is a product specific implementation of the ActiveSync protocol (although people seem to use the two terms interchangeable which is incorrect), allows a device to register itself with the server to receive push notifications from the server without any additional components or complexities.
I am also curious what you meant by "Exchange connector front end", since there is no Exchange product that acts as a front-end to a 3rd party email system. Did you mean a 3rd party ActiveSync front-end? If so perhaps some of your issues are with that product's implementation of the ActiveSync protocol, not to be confused with EAS which is how Microsoft implements ActiveSync in Exchange.
Likewise each end point manufacturer (I.E. BlackBerry) chooses what features of ActiveSync to implement and how. It could be how they chose to implement something that is causing you some of the pain you are experiencing.
Sorry you are having issues with your chosen configuration. My only suggestion to you is to try breaking up your mail store into multiple pieces, look to not syncing more than X amount of days of your email, use the Data Control options I posted about above (which it appears you didn't know about since you cited IMAP could "give the client headers only"), or all of the above.04-22-15 01:05 PMLike 0 -
In my experience users just don't manage email like you do, and that includes supporting large corporations and also large government entities where people have spent decades working in the same job. Normally users archive off old email, most likely due to storage constraints of their primary mailbox and/or performance issues.
It's not storage constraints, it's performance related -- which has everything to do with garbage implementations.
Again I was just restating what I had been told, in that true push notification IMAP was beyond IDLE. Doing some searches I found the following:
Push-IMAP - Wikipedia, the free encyclopedia
https://groups.google.com/forum/#!to...ap/7_1q7TYsTzI
From reading those it looks like IDLE is just a component in push notification IMAP, but it itself is not Push. I'm not an expert at IDLE, just telling you what I was told and what I am reading online.
EAS on the other hand, which is a product specific implementation of the ActiveSync protocol (although people seem to use the two terms interchangeable which is incorrect), allows a device to register itself with the server to receive push notifications from the server without any additional components or complexities.
When you issue an IMAP "IDLE" command you give the server a folder to monitor. When it sees changed content it tells the client with a very brief message; the client then requests the change state on the folder and acts as appropriate.
Neither is true "push" because of NAT constraints, specifically, the impossibility of mapping datagrams through a NAT device on indefinite time horizons. That is, I can send a UDP datagram through a NAT device for port 53 (DNS), for example, and get a response -- but if the server tries to send a second response to my client 10 minutes later the mapping will be gone in the NAT device and the packet goes on the floor without being delivered to the client.
UDP is designed to be able to serve any number of potential clients with one socket descriptor and resource being open at any given time. NAT devices, however, cannot possibly maintain an infinite number of datagram IPort tuple mappings (RAM is finite, of course) nor do they have any way to know with certainty when an old mapping should be discarded (say, because the client behind it was turned off, left the current cell or Wifi transmitter range, etc.)
The way EAS and IMAP/IDLE get around this problem with NAT is by abusing TCP connections, leaving them open. Since each TCP connection carries state and the NAT device thus knows the connection is "current" this "works" after a fashion but leads to a secondary potential problem on the server end because each client with such a connection open consumes a stream descriptor (UDP would not but can't be used for the reason above.) EAS is in fact implemented more-or-less on top of HTTP, if you want to get into the weeds on the details of how it works. The problem with this abuse is that stream descriptors are a finite resource on the server end; IMAP resolves that by allowing an IDLE-issuing device to specify a maximum timeout after which it will either renew the request or drop it, and also allows the client to recognize a broken connection immediately should the server drop it (say, because it becomes resource constrained.) Since IMAP is lightweight on establishment of the connection, including login, in the first place (SSL negotiation is the most-costly part of connection setup for most servers) this makes for a very reliable and reasonably light-weight protocol when serving thousands of simultaneous users. EAS has a problem in this regard in that the setup of a connection is relatively-speaking "heavy" in the amount of processing and data exchange required to get back to a "idle yet monitoring" state when a connection has been is dropped.
Get out Wireshark or other similar tools and watch the protocols for a while and you'll see the difference. It's quite substantial and large EAS communities have materially greater processing and data requirements compared against those running IMAP/IDLE.
I am also curious what you meant by "Exchange connector front end", since there is no Exchange product that acts as a front-end to a 3rd party email system. Did you mean a 3rd party ActiveSync front-end? If so perhaps some of your issues are with that product's implementation of the ActiveSync protocol, not to be confused with EAS which is how Microsoft implements ActiveSync in Exchange.
Likewise each end point manufacturer (I.E. BlackBerry) chooses what features of ActiveSync to implement and how. It could be how they chose to implement something that is causing you some of the pain you are experiencing.
Of course that theory sometimes breaks in practice (witness BlackBerry emitting bogus MIME messages when you sign a S/MIME email; base64 specifies a maximum line length that they blithely ignore when emitting those messages....)04-22-15 02:01 PMLike 0 -
Posted via CB1004-23-15 06:15 AMLike 0 - This is a textbook case for the use of PST files. There's no reason for emails taking up space in an active inbox after a year. Create a PST file or whatever type of local storage file your desktop email client provides, or just convert the old emails into PDF and be done with it.
Posted via CB10
Again, the issue is NOT emails that are in the INBOX.04-23-15 07:05 AMLike 0 - This thread amuses me. Suggesting that something is terrible because it fails to do something that, pretty much, no-one ever does. It sounds like you need to design a protocol for yourself only.
Posted via CB1004-23-15 02:16 PMLike 0 - You implied that I wanted decades of email (or thousands of folders) SYNCed. I want no such thing; I want access to them on demand but I have no desire to have ANY of it on the device until and unless I want to look at it. IMAP allows this, EAS does not because the folder index has to be maintained on the mobile device itself.
- I have an utterly huge email store going back (literally) to the 1980s.
- Until I pointed my Passport at my native email store. All of it.
I'm not sure how else I was supposed to read that. Also I never said you were trying to sync all your folders, just that having thousands of folders in your mailbox is not a good idea and was likely responsible for most of your syncing issues.
Yeah, performance issues that come about because of crappy implementations!
It's not storage constraints, it's performance related -- which has everything to do with garbage implementations.
There's a secondary problem with extremely large messages in the "Sent" folder, which the phone insists on syncing and cannot be shut off.
The way EAS and IMAP/IDLE get around this problem with NAT is by abusing TCP connections, leaving them open. Since each TCP connection carries state and the NAT device thus knows the connection is "current" this "works" after a fashion but leads to a secondary potential problem on the server end because each client with such a connection open consumes a stream descriptor (UDP would not but can't be used for the reason above.) EAS is in fact implemented more-or-less on top of HTTP, if you want to get into the weeds on the details of how it works. The problem with this abuse is that stream descriptors are a finite resource on the server end; IMAP resolves that by allowing an IDLE-issuing device to specify a maximum timeout after which it will either renew the request or drop it, and also allows the client to recognize a broken connection immediately should the server drop it (say, because it becomes resource constrained.) Since IMAP is lightweight on establishment of the connection, including login, in the first place (SSL negotiation is the most-costly part of connection setup for most servers) this makes for a very reliable and reasonably light-weight protocol when serving thousands of simultaneous users. EAS has a problem in this regard in that the setup of a connection is relatively-speaking "heavy" in the amount of processing and data exchange required to get back to a "idle yet monitoring" state when a connection has been is dropped.
Get out Wireshark or other similar tools and watch the protocols for a while and you'll see the difference. It's quite substantial and large EAS communities have materially greater processing and data requirements compared against those running IMAP/IDLE.
You believe IMAP with IDLE is more elegant and efficient than Exchange, and I'm not going to argue with you because you have clearly made up your mind. I prefer a single protocol for syncing all aspects of my mailbox (including the calendar and contacts) and the rest of the industry (except for Google who dislikes all thinks Microsoft) seems to agree with me. I also like that I as an end user can log into OWA and turn on ActiveSync logging through the Control Panel if I want to.
Well, no, the problems exist in both Microsoft's Exchange server and the 3rd party protocol stuff (yes I've tested against both) because it's a limitation of how EAS works. Microsoft of course sets those rules since ActiveSync is under their exclusive control.
Well yes and no; they choose a given protocol version but having done so they're bound to implement it correctly -- at least in theory.
Of course that theory sometimes breaks in practice (witness BlackBerry emitting bogus MIME messages when you sign a S/MIME email; base64 specifies a maximum line length that they blithely ignore when emitting those messages....)
Comparison of Exchange ActiveSync clients - Wikipedia, the free encyclopedia
As you can see the different clients are all over the map, even between versions of software. Even Microsoft did not implement all of the ActiveSync features in Windows Phone, although they are improving over time.
And then there are client side bugs with the features the mobile do choose to implement, as you noted with your SMIME comment and BlackBerry 10 devices. I can't tell you how many times I have dealt with iOS updates and their screwing up calendaring or poorly designing a EAS feature so they cause a sever to get bogged down. Apple is the worst offender of not QA'ing their EAS code.
BTW I do want to clarify that I mis-spoke about EAS being an Exchange only implementation of ActiveSync. I was speaking to a colleague and he corrected me to say EAS is what Microsoft has licensed out to 3rd parties in addition to using it in Exchange. ActiveSync (non-EAS) was the old syncing software for mobile devices that was abandoned by Microsoft. He did agree that only Microsoft fully and completely implements the EAS, which is understandable considering they write it, and that putting "Exchange" in front of it does cause confusion. My bad for saying otherwise.
At this point I think we are going to have to agree to disagree in regards to the issue being with how you choose to structure your mailbox versus the issue being a fault of the code. I don't disagree that Microsoft could use this feedback to see if they could optimize their code a little better for the fringe cases such as yourself, and I will pass on your folder structure scenario and related performance issues to the Exchange Product Group the next time I see one of them. Meanwhile if you do choose to revisit ActiveSync, I highly encourage you to consider moving some of those folders out of your primary mailbox (an Archive mailbox is a great location if you store your email on Exchange again any time soon).
To recap my original issue with your subject line: a 50GB mailbox is something I consider to be a ridiculously large email store, and there are plenty of Exchange 2013 implementations (including Office 365) that have those with users syncing via ActiveSync with no issue. So a large mailbox itself is not an issue. Nor are large emails as most of the time those are with attachments that ActiveSync doesn't download by default. Even if the messages bodies are excessively large there is a way to handle that as well.
Have a great weekend!Last edited by HotFix; 04-25-15 at 05:32 AM.
deiop likes this.04-24-15 07:15 PMLike 1 - UDP is connectionless and does not guarantee packet delivery. Exchange used UDP for new mail notifications with Outlook back in the old days, and given networking issues within large enterprises, much less the wild wild west that is the Internet, I'm glad Microsoft isn't relying on it for new mail notifications with ActiveSync. Using UDP across the Internet adding carrier networks on top of that) would make push mail notifications worthless IMHO.
The reason it is unsuitable for a push notification system is not that it is connectionless and stateless; that's a feature, not a bug. Being connectionless means that the protocol must handle the case where messages are lost (DNS does this just fine) because under congested conditions this will happen.
No, the issue with UDP is that when address translation is in use the mappings are volatile and thus for any other than a query/response model unless the notifying agent is on the client side side of the NAT gateway it will not work. As the point of push notifications is for the server to initiate the notification (rather than the client polling for it) UDP is an inappropriate protocol for this purpose on the wider Internet.
You believe IMAP with IDLE is more elegant and efficient than Exchange, and I'm not going to argue with you because you have clearly made up your mind. I prefer a single protocol for syncing all aspects of my mailbox (including the calendar and contacts) and the rest of the industry (except for Google who dislikes all thinks Microsoft) seems to agree with me.
The fact is that there are IETF standards for IMAP, CalDAV and CardDAV. Being IETF standards you have the benefit of a formal documentary process and a formal change process, including (normally) backward compatibility when revisions are made. This means that various entities that produce software implementations have a canonical reference as to correct (and incorrect) behavior and when incorrect behavior is observed it can be determined exactly who screwed the pooch and how. This leads to demonstrably superior results in a multi-vendor environment.
Witness the original BB10 screw-up with SMTP transmission; in the days of 10.0.10 there were a wide variety of SMTP servers that could not receive email from BB10 phones. It was extremely easy, using the IETF reference documents on SMTP, to determine that the reason for this was that BB10 was not emitting CR/LF pairs at the end of lines, but instead was emitting bare LF characters.
Or look at the current issue with BB10 devices emitting invalid MIME attachments -- specifically, for S/MIME signatures. The IETF standards say that a MIME component line that is base64 encoded may not exceed 76 characters. That exchange is willing to "eat" such a line and that Outlook will "eat" it as well, processing it, is immaterial. Without IETF standards it would be very easy for BlackBerry (or anyone else) to point fingers and claim "well, it works with Microsoft!"
Indeed, it is this very fact -- that it does, and that S/MIME has been reliant on Exchange -- that has allowed this bogus implementation to remain unaddressed for more than two years in the BB10 codebase.
And then there are client side bugs with the features the mobile do choose to implement, as you noted with your SMIME comment and BlackBerry 10 devices. I can't tell you how many times I have dealt with iOS updates and their screwing up calendaring or poorly designing a EAS feature so they cause a sever to get bogged down. Apple is the worst offender of not QA'ing their EAS code.
Meanwhile if you do choose to revisit ActiveSync, I highly encourage you to consider moving some of those folders out of your primary mailbox (an Archive mailbox is a great location if you store your email on Exchange again any time soon).
I find it amusing that the "answer" you and others have propounded to such a structure, which has worked perfectly fine with open standard (e.g. IMAP) clients for two decades, is to stick emails in a proprietary archive format such as a .PST file. It is especially amusing considering the amount of money I have made through the last two decades unscrambling someone's .PST file when they call me in a panic as something has gone wrong and correspondence they need has become inaccessible. Several of those organizations have seen the wisdom, after such an event has cost them a ton of money and occasionally led to data that cannot be retrieved in a reasonable fashion and at reasonable cost, or at all, of moving to a mail system that is not subject to such events.
The point of my complaint here is simply that such a hierarchy structure has no business being stored as a complete "index" on a mobile device since it is both unnecessary and produces undesirable behavior (specifically, traffic volume and potential "break points" given the relative data store size available on both a mobile and server device.)
Hopefully BlackBerry will divorce its S/MIME implementation from the Exchange requirement entirely; Exchange is not necessary for S/MIME in any way and is simply a dodge as BlackBerry is using Exchange to separate out the signature (and encrypted portion) attachments rather than doing it in the client. This decision might be defensible if the client didn't know how to deal with attachments itself, but the client does know how to do so (as do essentially all modern email clients) and as such there is no defensible reason for them to have undertaken this decision in the first place.
This decision is especially egregious in that BlackBerry's email client and handset software does not require back-end services for either certificate verification (in fact it insists that the certificates in the chain up to the CA be on the handset) and further, the handset also knows how to handle OCSP revocation checks on its own.04-25-15 08:10 AMLike 0 - Hi All,
I am using BB Z10 with 10.3.1.1779 OS Version; I also setup and established Outlook.com with Exchange ActiveSync on this device since around one year.
Everything was fine until some weeks ago suddenly the device accessing and downloading lots of data from Outlook.com that filled the storage up to some gigabytes, this was very strange since although downloading but no e-mail appears beyond 30 days that I setup on the device.
The device kept downloading lots data and filled the storage for even more gigabytes, the only thing that can solve this problem is by deleting Outlook.com mail account and re-add it.
Does anyone have the same problem? Can anyone explain about this phenomenon?
Also, why is this device not able to sync the e-mail draft? If I wrote e-mail draft on device, it does not sync at Outlook.com server or web interface.
Thank you.05-16-15 07:03 AMLike 0
- Forum
- BlackBerry 10 Phones & OS
- BlackBerry 10 OS
Be Aware, Exchange, Ridiculously Large Email Stores and BB10
« Why do multiple account contacts sync more like contact cuisinart?
|
10.3.1.634 for all devices (Developer OS!) »
Similar Threads
-
Time to spend some $ and update CB App
By Mr4aces in forum Site and App Feedback & HelpReplies: 71Last Post: 10-07-15, 06:43 PM -
Saving messages/email?
By SharonVA in forum BlackBerry ClassicReplies: 9Last Post: 04-28-15, 01:04 PM -
Are or when are the blue or bronze going to be available ?
By CrackBerry Question in forum Ask a QuestionReplies: 3Last Post: 04-20-15, 08:03 PM -
A battery that indicates plenty of charge goes suddenly blank and turns off.
By CrackBerry Question in forum Ask a QuestionReplies: 1Last Post: 04-20-15, 05:28 PM -
Should BlackBerry just have tweaked and the Z3?
By m3mb3rsh1p in forum BlackBerry LeapReplies: 0Last Post: 04-20-15, 02:57 PM
LINK TO POST COPIED TO CLIPBOARD