05-16-15 01:20 PM
28 12
tools
  1. tickerguy's Avatar
    Here's a new one I ran into today, and it wasn't fun.

    I have an utterly huge email store going back (literally) to the 1980s. It is all visible under my desktop Thunderbird client via Dovecot (a really amazing IMAP server, by the way, that can handle ridiculous levels of load and I have deployed at clients with huge user bases and even bigger mail stores.)

    Anyway, I recently implemented an Exchange connector front end to this since I have clients that want it, but don't want to give up a working email system (don't get me started on Exchange!)

    So I did, and life appeared to be good.

    Until I pointed my Passport at my native email store.

    All of it.

    Now I wasn't ridiculous about it -- I only told it to sync the INBOX and a half-dozen folders. But -- it went and got state information for all of them.

    What's worse, it kept maintaining that internally on the Passport.

    Now the good news is that the phone didn't wig out and crash.

    The bad news is that the PIM and Hub CPU and battery consumption went to somewhere north of Mars and stayed there with the server being repeatedly hit for state changes.

    In a word, meh.

    Not that I think anything else would do better; I run IMAP on my PC and Laptop for a reason, and this is the reason. I can subscribe to a handful of folders for push notification but not have the damn server hitting me up every 30 seconds with this and that, and as a result my CPU usage is nil yet I still have access to everything.

    But that's not an option if you want BES management of BB10, nor if you want S/MIME even after 10.3.2. You run exchange or you don't get it, period.

    I'm not saying it's good or bad, just saying it is what it is and you need to be thinking about how you're going to compartmentalize email in some sort of fashion if you have utterly huge archives that you want access to. This much I can tell you -- the BB10 phones can't handle being terminals into a mail system with a couple of thousand folders accessible at a given time on Exchange.

    Access to this sort of data store works perfectly fine on IMAP without a care in the world but then you give up S/MIME and BES management.

    Having to make that choice sucks; BlackBerry needs to open up S/MIME so it works on IMAP servers and having BES manage them would be nice too.
    chgaida likes this.
    04-20-15 09:25 PM
  2. tickerguy's Avatar
    PS: Another way to get in big trouble is have a shared email store with some really, really large sent files. Like, for example, if you sent multi-megabyte logs around to clients..... yeah, that can be bad since the "Sent" folder is sync'd to the phone.
    04-20-15 09:58 PM
  3. deiop's Avatar
    That is a problem of your active sync implementation. With IBM traveler and exchange you can configure and sync only the folders you want. This is not a device issue.

    The active sync server / service has to support active sync 14.1 to work probably with your BlackBerry device.

    And as far as I know BES12 support for imap is coming with one of the next releases.


    Posted via CB10
    04-21-15 12:27 AM
  4. tickerguy's Avatar
    That is a problem of your active sync implementation. With IBM traveler and exchange you can configure and sync only the folders you want. This is not a device issue.

    The active sync server / service has to support active sync 14.1 to work probably with your BlackBerry device.

    And as far as I know BES12 support for imap is coming with one of the next releases.


    Posted via CB10
    It does support 14.1 and yes, I can select which folders to sync (and obviously I don't say "sync all" as there's not that much memory available on the phone!) That's not the issue; the issue is that the catalog is transferred and the PIM manager on the phone has a problem with the size of it. It doesn't blow up but it creates a significant load (and thus battery drain) issue.

    There are not a lot of folks out there with a few thousand folders. I'm one of them, and I can get around this by moving those unlikely to be accessed to a different account (thus removing them from the "view" of the mobile) but it's just one of those limitations of Exchange. The phone has no problem on the same mailstore under IMAP because IMAP doesn't pull a catalog until and unless you go to browse the folder list.

    There's a secondary problem with extremely large messages in the "Sent" folder, which the phone insists on syncing and cannot be shut off. I can get around that by having a separate mapping for the "Sent" folder on the phone as opposed to desktop clients; this becomes an issue when you start doing things like sending data dumps to clients that are several megabytes in size (or crash dumps, etc.) Since those go in your "Sent" folder and Exchange insists on syncing that if you throw a few of those out from your PC you will get a surprise with regard to what it does on your BlackBerry!

    The lack of S/MIME support on IMAP accounts bites, as does the lack of BES support for them on BlackBerry devices. IMAP is a nearly-perfect protocol for mobile devices when it comes to email exchange, and coupled with CalDAV and CardDAV (which BlackBerry supports native) for contacts and calendars it would be pretty simply to link them. You can do it now on the personal side and I have since the original 10.0.10.85 firmware with good effect.

    I'm looking forward to a PROPER S/MIME implementation (there is a very severe problem with the current code on the phone in that it emits improperly formatted MIME messages, even though Exchange, when signing or encryption is turned on) and, hopefully, BlackBerry will support IMAP for it as well as adding IMAP + Caldav/CardDav to BES. For those who aren't married to the Microsoft paradigm and back end (think those firms with cloud-based infrastructure rather than large-company in-house deployments) this would be a godsend.
    04-21-15 12:30 AM
  5. HotFix's Avatar
    (don't get me started on Exchange!)
    Ok... I'll bite. What's wrong with Exchange?

    For the record IMAP was not designed for lightweight communication with mobile devices with push notifications. ActiveSync was. Google and others have extended IMAP and bolted on other protocols to make up for its short comings, so obviously it can be made to work, but again that's a retrofit of an old protocol versus one designed for mobile devices from the ground up.

    It sounds like your issue is you have thousands of folders, which as you note is not common, and you believe there is a performance issues with your device trying to download the "catalog" of them.

    I could see any device needing to download the list of folders so you the end user could choose which ones to sync. I'm unaware of this being an issue with ActiveSync, but then again I am not aware of any customer with thousands of folders in their mailbox.

    It also sounds like you front ended your IMAP server that emulates Exchange, and not Exchange itself. If so how do you know the issue isn't with the front end you placed in front of your IMAP server.

    Posted via CB10 on Z30STA100-5/10.3.1.2726
    Supa_Fly1 and deiop like this.
    04-21-15 08:12 PM
  6. HotFix's Avatar
    And for the record Microsoft hosts 50 GB mailboxes, which I consider "large email stores", and users don't have any issues using ActiveSync against them. So the size of the mailbox isn't the issie.

    Posted via CB10 on Z30STA100-5/10.3.1.2726
    Supa_Fly1 and deiop like this.
    04-21-15 08:14 PM
  7. ppeters914's Avatar
    I cannot fathom a legitimate reason for retaining 20-30 year old email nor having thousands of folders.

    Posted via CB10 / AT&T /Z10 STL100-3 /10.3.1.2582
    rthonpm likes this.
    04-21-15 09:00 PM
  8. tickerguy's Avatar
    Ok... I'll bite. What's wrong with Exchange?

    For the record IMAP was not designed for lightweight communication with mobile devices with push notifications. ActiveSync was. Google and others have extended IMAP and bolted on other protocols to make up for its short comings, so obviously it can be made to work, but again that's a retrofit of an old protocol versus one designed for mobile devices from the ground up.
    Actually, IDLE preceded Google by quite some time, and it is far more lightweight than EAS for that purpose (immediate notification of new inbound messages.)

    Exchange has its purposes, but like most things Microsoft overpromises and underdelivers, along with gratuitously breaking things that are otherwise expected to interoperate with that beyond its borders. If all your world is Microsoft none of this matters much at all, but as soon as it's not the issues start to show up.

    My point in this was simply that IMAP doesn't come up with an issue such as this because it doesn't have a presumption that knowledge of the hierarchy matters except at the point where the client wishes to ask for it (e.g. to subscribe to a particular folder for synchronization purposes), and then only to the extent the client wants (e.g. "give the client headers only", etc). This also means that it doesn't enforce a sync of anything you do not wish immediate notification on (think the "Sent" folder for how this could go bad on you in a multi-client device, one mail-store environment.)

    I've always found the Exchange partisanship amusing given that it's basically a proprietary Microsoft thing, particularly in a world that is supposed to be built around open interchange and IETF standards.... I just wanted to note for the peanut gallery my bemusement at it choking on something that I've handled using standard protocols now for a couple of decades on clients with far less horsepower than today's high end phone devices without so much as a wimper of protest.
    chgaida likes this.
    04-21-15 09:37 PM
  9. HotFix's Avatar
    Actually, IDLE preceded Google by quite some time, and it is far more lightweight than EAS for that purpose (immediate notification of new inbound messages.)

    Exchange has its purposes, but like most things Microsoft overpromises and underdelivers, along with gratuitously breaking things that are otherwise expected to interoperate with that beyond its borders. If all your world is Microsoft none of this matters much at all, but as soon as it's not the issues start to show up.

    My point in this was simply that IMAP doesn't come up with an issue such as this because it doesn't have a presumption that knowledge of the hierarchy matters except at the point where the client wishes to ask for it (e.g. to subscribe to a particular folder for synchronization purposes), and then only to the extent the client wants (e.g. "give the client headers only", etc). This also means that it doesn't enforce a sync of anything you do not wish immediate notification on (think the "Sent" folder for how this could go bad on you in a multi-client device, one mail-store environment.)

    I've always found the Exchange partisanship amusing given that it's basically a proprietary Microsoft thing, particularly in a world that is supposed to be built around open interchange and IETF standards.... I just wanted to note for the peanut gallery my bemusement at it choking on something that I've handled using standard protocols now for a couple of decades on clients with far less horsepower than today's high end phone devices without so much as a wimper of protest.
    I was referencing that Google dropped ActiveSync for IMAP and chose to use that extended functionality (which was jot a part of the original IMAP protocol). I didn't mean to imply they invented push notification for IMAP.

    Also I think you are confusing IDLE, which isn't a push notification for Lemonade which is, at least that is how to was explained to me.

    IDLE is more lightweight than EAS? What are you basing this statement on? Even if it were true, EAS incorporates everything else needed to sync the contents of a mailbox including calendar and contacts, without having to bolt on other components, so it is a more complete solution:
    http://blogs.technet.com/b/exchange/...0/3403386.aspx

    Exchange over promises and under delivers? OK...LOL.

    As for downloading headers only, go turn that feature on in your ActiveSync account. By default it is set to Full Messages, but you can change that in BlackBerry 10 to be Headers Only or a couple of different size limits:
    Be Aware, Exchange, Ridiculously Large Email Stores and BB10-img_20150421_225214.png
    Given this understanding I don't see how syncing the Sent Items is a problem unless you are trying to sync all items with no time limit, and then that's on you for having years and years in your Sent items and choosing that setting. It certainly doesn't have anything to do with their size since you can control that behavior.

    As for Exchange being proprietary... I guess that in the sense that any corporate email solution that isn't 100% open source is. Otherwise Microsoft makes the ActiveSync protocol available to the public, and flows RFCs for other things like SMTP.

    So far the only potentially legitimate complaint I see is that you don't like how ActiveSync tries to download your entire folder hierarchy of "thousands" of folders. I can't speak to the potential performance problems as I know no one who tries to manage their email that way for me to analyze and test with.

    Posted via CB10 on Z30STA100-5/10.3.1.2726
    Last edited by HotFix; 04-21-15 at 11:09 PM.
    04-21-15 10:52 PM
  10. tickerguy's Avatar
    IDLE is more lightweight than EAS? What are you basing this statement on?

    Intimate knowledge of the protocols involved and what the back end must process in order to return the requested functionality, including developing code to run against it in both instances, beginning with a very high-volume production environment with many thousands of active users (specifically, my ISP) in the 1990s. I've only been at this for more than 20 years now in a professional capacity. You?
    Even if it were true, EAS incorporates everything else needed to sync the contents of a mailbox including calendar and contacts, without having to bolt on other components, so it is a more complete solution.
    Except calendars and contacts aren't in your mailbox, of course. Never mind that the protocol to do one is hardly ideal for the other and that there are standards-based ways to do both, with BB10 supporting them since 10.0.10.85 (with dead-nuts reliability too, I might add.)

    Microsoft's Exchange puts everything in one box which is great so long as everything's Microsoft and nothing ever goes wrong with it (in which case it all goes wrong at once.) It's not so good when that's not the case for one reason or another (like, for instance, all the world not being a Windows machine on the user desktop end.)

    Have a nice evening.
    04-21-15 11:06 PM
  11. HotFix's Avatar

    Intimate knowledge of the protocols involved and what the back end must process in order to return the requested functionality, including developing code to run against it in both instances, beginning with a very high-volume production environment with many thousands of active users (specifically, my ISP) in the 1990s. I've only been at this for more than 20 years now in a professional capacity. You?

    Except calendars and contacts aren't in your mailbox, of course. Never mind that the protocol to do one is hardly ideal for the other and that there are standards-based ways to do both, with BB10 supporting them since 10.0.10.85 (with dead-nuts reliability too, I might add.)

    Microsoft's Exchange puts everything in one box which is great so long as everything's Microsoft and nothing ever goes wrong with it (in which case it all goes wrong at once.) It's not so good when that's not the case for one reason or another (like, for instance, all the world not being a Windows machine on the user desktop end.)

    Have a nice evening.
    Ok... I should just take your word for it since you know... not exactly what I would call a technical explanation to my question.

    For the record I was editing my comment to provide better clarification (typing a lengthy response on a mobile device is not conducive to proof reading), adding an example of how you could have header only emails with ActiveSync, and examples (such as adding a functional comparison of EAS to IMAP features from the Exchange team Blog) when you had responded. I also believe there is a difference between IDLE and "lemonade" since it appears as if IDLE itself is not a push notification service like EAS.

    I am disappointed you decided to turn this into a sword waving contest because I questioned you your statement...
    Not that it should matter but I have been working with email (predominantly Exchange) for ~20 years myself, and I currently directly support an environment of over 100K users that spans the globe, and I also indirectly support many other large and diverse environments. One of the previous environments I supported was a very pro Mac and Linux enterprise, and there were no real issues with those platforms using Exchange as the corporate email platform.

    Sigh... yes contacts and calendars are stored in a users mailbox in Exchange. I thought we were talking about that since you were referring EAS and Exchange (as it is in the threads Subject line), and not a generic implementation of ActiveSync.

    You clearly have a bias against Exchange, and that's fine as everyone is entitled to their own opinion. However if you are going to make statements comparing two components as fact, then be prepared to back them up with data and not you pedigree.

    As for using you pedigree as a justification for your statements, I have a hard time trusting the technical word of someone that chooses to use thousands of folders to organize their mail, not to mention keeping 3 decades of email in one mailbox, but then they complain that some email component doesn't work the way they think it should with that uncommon choice of a configuration. I wouldn't blame any email product or protocol for not working well with that configuration. I am glad you found one protocol that does work for you though.

    You have a great evening as well.

    Posted via CB10 on Z30STA100-5/10.3.1.2726
    04-21-15 11:50 PM
  12. kevets's Avatar
    I think the mailboxes work well if you stick to 30 days to sync to the phone.

    If someone is syncing all of their mailbox and expect anything to keep up I am going to say to get their head checked. Run a remote search and that can work in a pinch.

    Assistant is becoming very powerful for keeping up with clients. I can usually type one of my clients into it and I see not only my remember notes, but contacts, emails, attachments, calendar appointments all related to the client. That type of stuff is unique to BB10 I think.

    Posted via CB10
    04-22-15 12:17 AM
  13. deiop's Avatar
    Thank you Hotfix for your words. I would have written the same if English were my main language.

    To add some more things:

    We are using IBM Domino and Traveler with BES 12, my maildatabase has about 32 GB and 68000 documents stored in it. I use about 90 folders, and in my opinion even 90 is too much and one day when I find the time I will create a new folder structure with less folders. In my humble opinion less folder (well structured) is more efficient.

    Whatever, beside that document numbers I have a well filled calender and about 2500 addresses in my addressbook.

    And guess : everything is syncing fine with my blackberry. Even the attachments in my synced sent box are NOT synced to the device, if I need them, I tap on them to get it downloaded. I sync 90 days, but I think 30 days should be enough because remote search works well and if I need an older email I just search for them.

    The most sync problems we had during the last years were not issued by the device but on the EAS side, named traveler in our case.

    The issues we had were reported to IBM and extremly fast fixed by them.
    We always got a separate Hotfix for the issues a day ore two after reporting and a couple of weeks later the official service pack arrived. Well done IBM.

    On the other side s/mime does not work with IBM Traveler, but this is something IBM is working on and will be there some day. Notes native encryption is working fine so when your s/mime certificate is imported to your notes.id , you can decrypt s/mime mails. In our case it is not needed because we are using a central s/mime and PGP gateway (zertificon).


    Long words short: We support 8900 users with there mailboxes and different syn techniques like EAS, IMAP, POP3, etc etc,
    And EAS in most cases is running fine and working for 99% of all users.

    Last word: and I already told you in another thread that IMAP support for bes 12 is comming.... :-)

    Posted via CB10
    04-22-15 01:37 AM
  14. tickerguy's Avatar
    Taking things out of context (or flatly inventing what was not said) appears to be a sport with some folks.

    Having a very large historical mail folder structure is only unusual if you don't have much history in a given organization. In this case the organization is and was literally mine. It isn't often that I have a reason to go back and look at such things, but when I do it's nice to have access to them in a seamless fashion.

    As far as "push" notifications go I find it particularly amusing that someone would think EAS is a "true" one and IMAP/IDLE is not. Both paradigms have the same basic problem in a mobile environment, namely that datagram transport doesn't work in the general case behind NAT since it's connectionless. For this reason both abuse a held-open TCP connection for this purpose. It would be nice if that wasn't necessary and some day within the IPv6 paradigm maybe it won't be but that's conjecture for the future.

    Being able to push IMAP connections via BES will be nice too, but it's a half-answer if S/MIME doesn't work on them. BB10 appears to "cheat" in the S/MIME context in this regard with Exchange in that it looks like its relying on the Exchange server to split the attachments for the client; since the client itself knows how to do that (it couldn't read HTML-formatted email without that capacity) it's even more-silly to have implemented it this way. Incidentally another bogon in the BB10 email paradigm that you don't care about in a Microsoft-only world is that their client doesn't emit both a plain text copy of the message body and an HTML-encoded copy. Everyone else figured out that this was pretty-much a requirement two decades ago back when NCSA Mosaic first showed up but BlackBerry, well, no. As a result this is what BB10 thinks is a cited article I sent to myself last night from a text email client:

    Code:
    Return-Path: <karl@denninger.net>Received: from localhost (localhost [127.0.0.1])
        by fs.denninger.net (8.14.9/8.14.8) with ESMTP id t3LN8bBo082441
        for <karl@denninger.net>; Tue, 21 Apr 2015 18:08:37 -0500 (CDT)
        (envelope-from karl@denninger.net)
    Received: from localhost [127.0.0.1];
        by Spamblock-sys (LOCAL/AUTH) Tue Apr 21 18:08:37 2015
    Mime-Version: 1.0
    X-client-id: 222
    X-mailer: BlackBerry Email (10.3.2.500)
    Message-id: <20150421230830.8958030.79048.222@denninger.net>
    Date: Tue, 21 Apr 2015 18:08:30 -0500
    Subject: A LOT of employees plan to buy a smartwatch for work (AAPL)
    From: karl@denninger.net
    To: Karl Denninger <karl@denninger.net>
    Content-Type: multipart/mixed;
     boundary="=_3df410f53ae0bca519a6a93c3c7f55ba"
    X-Antivirus: avast! (VPS 150421-1, 04/21/2015), Inbound message
    X-Antivirus-Status: Clean
    
    This is a multi-part message in MIME format.
    --=_3df410f53ae0bca519a6a93c3c7f55ba
    Content-Type: multipart/alternative;
     boundary="=_6bba4f969a5da8a604f5ff46864292d5"
    
    --=_6bba4f969a5da8a604f5ff46864292d5
    Content-Transfer-Encoding: base64
    Content-Type: text/html; charset=utf-8
    
    PGh0bWw+PGhlYWQ+PC9oZWFkPjxib2R5IGxhbmc9ImVuLVVTIj48ZGl2IHN0eWxlPSJ3aGl0ZS1z
    cGFjZTpwcmUtd3JhcDsgd29yZC13cmFwOiBicmVhay13b3JkOyI+QXBwbGUgV2F0Y2ggaXMgYWxy
    ZWFkeSBoYXZpbmcgYSByaXBwbGUgZWZmZWN0LiBBIGxvdCBvZiBlbXBsb3llZXMgd2hvIGFscmVh
    ZHkgdXNlIG1vYmlsZSBkZXZpY2VzIGZvciB3b3JrIHBsYW4gdG8gYnV5IGFuIEFwcGxlIFdhdGNo
    IG9yIHNpbWlsYXIgZGV2aWNlLCBhY2NvcmRpbmcgdG8gYSBnbG9iYWwgc3VydmV5IGJ5IG1vYmls
    ZSBkZXZpY2UgbWFuYWdlbWVudCBjb21wYW55IE1vYmlsZUlyb24uIEl0IGFza2VkIG92ZXIgMyw1
    MDAgd29ya2VycyBhYm91dCB0aGVpciBwbGFucyB0byBidXkgYSBkZXZpY2UgbGlrZSB0aGUgQXBw
    bGUgV2F0Y2guIDQyJSBwZXJjZW50IHNhaWQgdGhleSBlaXRoZXIgYWxyZWFkeSBvd24gYSBzbWFy
    dHdhdGNoIG9yIHBsYW4gdG8gYnV5IG9uZS4gV2hpbGUgdGhhdCBkb2Vzbid0IG1lYW4gdGhhdCB0
    aGV5IHdpbGwgYWxsIG9wdCBmb3IgdGhlIEFwcGxlIFdhdGNoLCBpdCBkb2VzIGJvZGUgd2VsbCBm
    b3IgdGhlIGNhdGVnb3J5IG9mIHNtYXJ0d2F0Y2hlcyBhbHRvZ2V0aGVyLiBBbW9uZyB0aGUgd29y
    ay1yZWxhdGVkIGFjdGl2aXRpZXMgdGhhdCB0aGUgcmVzcG9uZGVudHMgZW52aXNpb25lZCB1c2lu
    ZyBhIHNtYXJ0d2F0Y2ggZm9yOiBUYWtpbmcgcGhvbmUgY2FsbHMgNTglIFJlYWRpbmcgZW1haWwg
    NTYlIFdyaXRpbmcgZW1haWwgNDUlIEdldHRpbmcgYWxlcnRzLCBzdWNoIGFzIG1lZXRpbmcgcmVt
    aW5kZXJzIDQ0JSBBY2Nlc3NpbmcgY2FsZW5kYXIgNDAlIFJlYWRpbmcgZG9jdW1lbnRzIDM3JSBT
    dXJmaW5nIGNvbXBhbnkgaW50cmFuZXQgMzAlIE9uZSBwb3RlbnRpYWwuLi48YnI+X19fX19fX19f
    X19fX19fX19fX19fX19fX19fX19fX19fXzxicj48YnI+aHR0cDovL2ZlZWRwcm94eS5nb29nbGUu
    Y29tL35yL2J1c2luZXNzaW5zaWRlci9+My9vVE9tTEcyakg3RS9lbXBsb3llZXMtdG8tYnV5LWEt
    c21hcnR3YXRjaC1mb3Itd29yay0yMDE1LTQ8L2Rpdj48YnI+PGRpdiBzdHlsZT0iY29sb3I6IHJn
    YigzOCwgMzgsIDM4KTsgZm9udC1mYW1pbHk6IENhbGlicmksICdTbGF0ZSBQcm8nLCBzYW5zLXNl
    cmlmLCBzYW5zLXNlcmlmOyI+PGJyPi0mbmJzcDtLYXJsPGJyPihvbiZuYnNwO1BEQSk8L2Rpdj48
    L2JvZHk+PC9odG1sPg==
    --=_6bba4f969a5da8a604f5ff46864292d5--
    
    --=_3df410f53ae0bca519a6a93c3c7f55ba--
    That very readable on such a client..... NOT!

    (Of course it displays just fine in Thunderbird, Outlook or for that matter the BB10 client itself.... just don't try to read it from the command line!)
    04-22-15 08:20 AM
  15. HotFix's Avatar
    Taking things out of context (or flatly inventing what was not said) appears to be a sport with some folks.
    Let me know where I "flatly invented" what you said, and I will apologize for it. So far I see myself referencing what you wrote (decades of email, thousands of folders, Exchange ActiveSync (EAS), etc...).

    Having a very large historical mail folder structure is only unusual if you don't have much history in a given organization. In this case the organization is and was literally mine. It isn't often that I have a reason to go back and look at such things, but when I do it's nice to have access to them in a seamless fashion.
    In my experience users just don't manage email like you do, and that includes supporting large corporations and also large government entities where people have spent decades working in the same job. Normally users archive off old email, most likely due to storage constraints of their primary mailbox and/or performance issues. This is such a common practice that Microsoft created the option of an Exchange Archive mailbox so you could move the really old stuff out of your primary mailbox, but still keep it online and indexed should you need it.

    As far as "push" notifications go I find it particularly amusing that someone would think EAS is a "true" one and IMAP/IDLE is not. Both paradigms have the same basic problem in a mobile environment, namely that datagram transport doesn't work in the general case behind NAT since it's connectionless. For this reason both abuse a held-open TCP connection for this purpose. It would be nice if that wasn't necessary and some day within the IPv6 paradigm maybe it won't be but that's conjecture for the future.
    Again I was just restating what I had been told, in that true push notification IMAP was beyond IDLE. Doing some searches I found the following:
    Push-IMAP - Wikipedia, the free encyclopedia
    https://groups.google.com/forum/#!to...ap/7_1q7TYsTzI
    From reading those it looks like IDLE is just a component in push notification IMAP, but it itself is not Push. I'm not an expert at IDLE, just telling you what I was told and what I am reading online.

    EAS on the other hand, which is a product specific implementation of the ActiveSync protocol (although people seem to use the two terms interchangeable which is incorrect), allows a device to register itself with the server to receive push notifications from the server without any additional components or complexities.

    I am also curious what you meant by "Exchange connector front end", since there is no Exchange product that acts as a front-end to a 3rd party email system. Did you mean a 3rd party ActiveSync front-end? If so perhaps some of your issues are with that product's implementation of the ActiveSync protocol, not to be confused with EAS which is how Microsoft implements ActiveSync in Exchange.

    Likewise each end point manufacturer (I.E. BlackBerry) chooses what features of ActiveSync to implement and how. It could be how they chose to implement something that is causing you some of the pain you are experiencing.

    Sorry you are having issues with your chosen configuration. My only suggestion to you is to try breaking up your mail store into multiple pieces, look to not syncing more than X amount of days of your email, use the Data Control options I posted about above (which it appears you didn't know about since you cited IMAP could "give the client headers only"), or all of the above.
    04-22-15 02:05 PM
  16. tickerguy's Avatar
    Let me know where I "flatly invented" what you said, and I will apologize for it. So far I see myself referencing what you wrote (decades of email, thousands of folders, Exchange ActiveSync (EAS), etc...).
    You implied that I wanted decades of email (or thousands of folders) SYNCed. I want no such thing; I want access to them on demand but I have no desire to have ANY of it on the device until and unless I want to look at it. IMAP allows this, EAS does not because the folder index has to be maintained on the mobile device itself.
    In my experience users just don't manage email like you do, and that includes supporting large corporations and also large government entities where people have spent decades working in the same job. Normally users archive off old email, most likely due to storage constraints of their primary mailbox and/or performance issues.
    Yeah, performance issues that come about because of crappy implementations!

    It's not storage constraints, it's performance related -- which has everything to do with garbage implementations.
    Again I was just restating what I had been told, in that true push notification IMAP was beyond IDLE. Doing some searches I found the following:
    Push-IMAP - Wikipedia, the free encyclopedia
    https://groups.google.com/forum/#!to...ap/7_1q7TYsTzI
    From reading those it looks like IDLE is just a component in push notification IMAP, but it itself is not Push. I'm not an expert at IDLE, just telling you what I was told and what I am reading online.

    EAS on the other hand, which is a product specific implementation of the ActiveSync protocol (although people seem to use the two terms interchangeable which is incorrect), allows a device to register itself with the server to receive push notifications from the server without any additional components or complexities.
    Both work the same way in essence.

    When you issue an IMAP "IDLE" command you give the server a folder to monitor. When it sees changed content it tells the client with a very brief message; the client then requests the change state on the folder and acts as appropriate.

    Neither is true "push" because of NAT constraints, specifically, the impossibility of mapping datagrams through a NAT device on indefinite time horizons. That is, I can send a UDP datagram through a NAT device for port 53 (DNS), for example, and get a response -- but if the server tries to send a second response to my client 10 minutes later the mapping will be gone in the NAT device and the packet goes on the floor without being delivered to the client.

    UDP is designed to be able to serve any number of potential clients with one socket descriptor and resource being open at any given time. NAT devices, however, cannot possibly maintain an infinite number of datagram IPort tuple mappings (RAM is finite, of course) nor do they have any way to know with certainty when an old mapping should be discarded (say, because the client behind it was turned off, left the current cell or Wifi transmitter range, etc.)

    The way EAS and IMAP/IDLE get around this problem with NAT is by abusing TCP connections, leaving them open. Since each TCP connection carries state and the NAT device thus knows the connection is "current" this "works" after a fashion but leads to a secondary potential problem on the server end because each client with such a connection open consumes a stream descriptor (UDP would not but can't be used for the reason above.) EAS is in fact implemented more-or-less on top of HTTP, if you want to get into the weeds on the details of how it works. The problem with this abuse is that stream descriptors are a finite resource on the server end; IMAP resolves that by allowing an IDLE-issuing device to specify a maximum timeout after which it will either renew the request or drop it, and also allows the client to recognize a broken connection immediately should the server drop it (say, because it becomes resource constrained.) Since IMAP is lightweight on establishment of the connection, including login, in the first place (SSL negotiation is the most-costly part of connection setup for most servers) this makes for a very reliable and reasonably light-weight protocol when serving thousands of simultaneous users. EAS has a problem in this regard in that the setup of a connection is relatively-speaking "heavy" in the amount of processing and data exchange required to get back to a "idle yet monitoring" state when a connection has been is dropped.

    Get out Wireshark or other similar tools and watch the protocols for a while and you'll see the difference. It's quite substantial and large EAS communities have materially greater processing and data requirements compared against those running IMAP/IDLE.
    I am also curious what you meant by "Exchange connector front end", since there is no Exchange product that acts as a front-end to a 3rd party email system. Did you mean a 3rd party ActiveSync front-end? If so perhaps some of your issues are with that product's implementation of the ActiveSync protocol, not to be confused with EAS which is how Microsoft implements ActiveSync in Exchange.
    Well, no, the problems exist in both Microsoft's Exchange server and the 3rd party protocol stuff (yes I've tested against both) because it's a limitation of how EAS works. Microsoft of course sets those rules since ActiveSync is under their exclusive control.
    Likewise each end point manufacturer (I.E. BlackBerry) chooses what features of ActiveSync to implement and how. It could be how they chose to implement something that is causing you some of the pain you are experiencing.
    Well yes and no; they choose a given protocol version but having done so they're bound to implement it correctly -- at least in theory.

    Of course that theory sometimes breaks in practice (witness BlackBerry emitting bogus MIME messages when you sign a S/MIME email; base64 specifies a maximum line length that they blithely ignore when emitting those messages....)
    04-22-15 03:01 PM
  17. rthonpm's Avatar
    I cannot fathom a legitimate reason for retaining 20-30 year old email nor having thousands of folders.

    Posted via CB10 / AT&T /Z10 STL100-3 /10.3.1.2582
    This is a textbook case for the use of PST files. There's no reason for emails taking up space in an active inbox after a year. Create a PST file or whatever type of local storage file your desktop email client provides, or just convert the old emails into PDF and be done with it.

    Posted via CB10
    04-23-15 07:15 AM
  18. tickerguy's Avatar
    This is a textbook case for the use of PST files. There's no reason for emails taking up space in an active inbox after a year. Create a PST file or whatever type of local storage file your desktop email client provides, or just convert the old emails into PDF and be done with it.

    Posted via CB10
    And again, people can't read or are intentionally stuffing words in someone's mouth that were never spoken.

    Again, the issue is NOT emails that are in the INBOX.
    04-23-15 08:05 AM
  19. cjameslockwood's Avatar
    This thread amuses me. Suggesting that something is terrible because it fails to do something that, pretty much, no-one ever does. It sounds like you need to design a protocol for yourself only.

    Posted via CB10
    04-23-15 03:16 PM
  20. tickerguy's Avatar
    Why would I need to design something when an IETF-standard protocol that does that same thing under those same circumstances, and does it very well, has existed and been in commercial use for more than 20 years?
    04-23-15 03:18 PM
  21. deiop's Avatar
    What about switching to Android f.e. If Blackberry is not right for your needs?

    Posted via CB10
    04-24-15 01:19 AM
  22. HotFix's Avatar
    You implied that I wanted decades of email (or thousands of folders) SYNCed. I want no such thing; I want access to them on demand but I have no desire to have ANY of it on the device until and unless I want to look at it. IMAP allows this, EAS does not because the folder index has to be maintained on the mobile device itself.
    I didn't think I was implying or "flatly invented" you were trying to sync decades of email when I read these statements in your posts:
    1. I have an utterly huge email store going back (literally) to the 1980s.
    2. Until I pointed my Passport at my native email store. All of it.

    I'm not sure how else I was supposed to read that. Also I never said you were trying to sync all your folders, just that having thousands of folders in your mailbox is not a good idea and was likely responsible for most of your syncing issues.

    Yeah, performance issues that come about because of crappy implementations!

    It's not storage constraints, it's performance related -- which has everything to do with garbage implementations.
    Performance issues can come about because of bad code (which I respectfully disagree with you on Exchange being bad code), and/or poor server side and client side implementations. I believe the way you are choosing to store and organize your email is the issue, not the software product that works fine for just about everyone else. You have another product/protocol that works better for you? Great. That doesn't mean Exchange is bad software, it just isn't right for how you choose to do things.

    There's a secondary problem with extremely large messages in the "Sent" folder, which the phone insists on syncing and cannot be shut off.
    BTW You never acknowledged where I showed you how to set ActiveSync to sync headers only or only parts of emails to address that specific issue. I hope you saw that and that the change helps you should you end up using ActiveSync again.

    The way EAS and IMAP/IDLE get around this problem with NAT is by abusing TCP connections, leaving them open. Since each TCP connection carries state and the NAT device thus knows the connection is "current" this "works" after a fashion but leads to a secondary potential problem on the server end because each client with such a connection open consumes a stream descriptor (UDP would not but can't be used for the reason above.) EAS is in fact implemented more-or-less on top of HTTP, if you want to get into the weeds on the details of how it works. The problem with this abuse is that stream descriptors are a finite resource on the server end; IMAP resolves that by allowing an IDLE-issuing device to specify a maximum timeout after which it will either renew the request or drop it, and also allows the client to recognize a broken connection immediately should the server drop it (say, because it becomes resource constrained.) Since IMAP is lightweight on establishment of the connection, including login, in the first place (SSL negotiation is the most-costly part of connection setup for most servers) this makes for a very reliable and reasonably light-weight protocol when serving thousands of simultaneous users. EAS has a problem in this regard in that the setup of a connection is relatively-speaking "heavy" in the amount of processing and data exchange required to get back to a "idle yet monitoring" state when a connection has been is dropped.

    Get out Wireshark or other similar tools and watch the protocols for a while and you'll see the difference. It's quite substantial and large EAS communities have materially greater processing and data requirements compared against those running IMAP/IDLE.
    UDP is connectionless and does not guarantee packet delivery. Exchange used UDP for new mail notifications with Outlook back in the old days, and given networking issues within large enterprises, much less the wild wild west that is the Internet, I'm glad Microsoft isn't relying on it for new mail notifications with ActiveSync. Using UDP across the Internet adding carrier networks on top of that) would make push mail notifications worthless IMHO.

    You believe IMAP with IDLE is more elegant and efficient than Exchange, and I'm not going to argue with you because you have clearly made up your mind. I prefer a single protocol for syncing all aspects of my mailbox (including the calendar and contacts) and the rest of the industry (except for Google who dislikes all thinks Microsoft) seems to agree with me. I also like that I as an end user can log into OWA and turn on ActiveSync logging through the Control Panel if I want to.

    Well, no, the problems exist in both Microsoft's Exchange server and the 3rd party protocol stuff (yes I've tested against both) because it's a limitation of how EAS works. Microsoft of course sets those rules since ActiveSync is under their exclusive control.
    Setting the standard for how the ActiveSync protocol works and what features it has is one thing, having some sort of control over how 3rd party vendors choose to implement it (and with what features) in their software is another. I have seen repeated ActiveSync issues on these forums with 3rd party implementations, enough to know that it's not a Microsoft control issue but rather a 3rd party implementation issue.

    Well yes and no; they choose a given protocol version but having done so they're bound to implement it correctly -- at least in theory.

    Of course that theory sometimes breaks in practice (witness BlackBerry emitting bogus MIME messages when you sign a S/MIME email; base64 specifies a maximum line length that they blithely ignore when emitting those messages....)
    Your first statement there is unfortunately is not a realistic view of how mobile device manufacturers implement EAS (I wish it was though). Reference:
    Comparison of Exchange ActiveSync clients - Wikipedia, the free encyclopedia
    As you can see the different clients are all over the map, even between versions of software. Even Microsoft did not implement all of the ActiveSync features in Windows Phone, although they are improving over time.

    And then there are client side bugs with the features the mobile do choose to implement, as you noted with your SMIME comment and BlackBerry 10 devices. I can't tell you how many times I have dealt with iOS updates and their screwing up calendaring or poorly designing a EAS feature so they cause a sever to get bogged down. Apple is the worst offender of not QA'ing their EAS code.

    BTW I do want to clarify that I mis-spoke about EAS being an Exchange only implementation of ActiveSync. I was speaking to a colleague and he corrected me to say EAS is what Microsoft has licensed out to 3rd parties in addition to using it in Exchange. ActiveSync (non-EAS) was the old syncing software for mobile devices that was abandoned by Microsoft. He did agree that only Microsoft fully and completely implements the EAS, which is understandable considering they write it, and that putting "Exchange" in front of it does cause confusion. My bad for saying otherwise.

    At this point I think we are going to have to agree to disagree in regards to the issue being with how you choose to structure your mailbox versus the issue being a fault of the code. I don't disagree that Microsoft could use this feedback to see if they could optimize their code a little better for the fringe cases such as yourself, and I will pass on your folder structure scenario and related performance issues to the Exchange Product Group the next time I see one of them. Meanwhile if you do choose to revisit ActiveSync, I highly encourage you to consider moving some of those folders out of your primary mailbox (an Archive mailbox is a great location if you store your email on Exchange again any time soon).

    To recap my original issue with your subject line: a 50GB mailbox is something I consider to be a ridiculously large email store, and there are plenty of Exchange 2013 implementations (including Office 365) that have those with users syncing via ActiveSync with no issue. So a large mailbox itself is not an issue. Nor are large emails as most of the time those are with attachments that ActiveSync doesn't download by default. Even if the messages bodies are excessively large there is a way to handle that as well.

    Have a great weekend!
    Last edited by HotFix; 04-25-15 at 06:32 AM.
    deiop likes this.
    04-24-15 08:15 PM
  23. tickerguy's Avatar
    UDP is connectionless and does not guarantee packet delivery. Exchange used UDP for new mail notifications with Outlook back in the old days, and given networking issues within large enterprises, much less the wild wild west that is the Internet, I'm glad Microsoft isn't relying on it for new mail notifications with ActiveSync. Using UDP across the Internet adding carrier networks on top of that) would make push mail notifications worthless IMHO.
    You use UDP across the Internet every single time you're on it and it works perfectly well. It's used for DNS, specifically.

    The reason it is unsuitable for a push notification system is not that it is connectionless and stateless; that's a feature, not a bug. Being connectionless means that the protocol must handle the case where messages are lost (DNS does this just fine) because under congested conditions this will happen.

    No, the issue with UDP is that when address translation is in use the mappings are volatile and thus for any other than a query/response model unless the notifying agent is on the client side side of the NAT gateway it will not work. As the point of push notifications is for the server to initiate the notification (rather than the client polling for it) UDP is an inappropriate protocol for this purpose on the wider Internet.
    You believe IMAP with IDLE is more elegant and efficient than Exchange, and I'm not going to argue with you because you have clearly made up your mind. I prefer a single protocol for syncing all aspects of my mailbox (including the calendar and contacts) and the rest of the industry (except for Google who dislikes all thinks Microsoft) seems to agree with me.
    I have "clearly made up my mind" because I have studied both protocols both in theory (e.g. as claimed) and as actually written with the code paths that must be traversed to perform the related functions.

    The fact is that there are IETF standards for IMAP, CalDAV and CardDAV. Being IETF standards you have the benefit of a formal documentary process and a formal change process, including (normally) backward compatibility when revisions are made. This means that various entities that produce software implementations have a canonical reference as to correct (and incorrect) behavior and when incorrect behavior is observed it can be determined exactly who screwed the pooch and how. This leads to demonstrably superior results in a multi-vendor environment.

    Witness the original BB10 screw-up with SMTP transmission; in the days of 10.0.10 there were a wide variety of SMTP servers that could not receive email from BB10 phones. It was extremely easy, using the IETF reference documents on SMTP, to determine that the reason for this was that BB10 was not emitting CR/LF pairs at the end of lines, but instead was emitting bare LF characters.

    Or look at the current issue with BB10 devices emitting invalid MIME attachments -- specifically, for S/MIME signatures. The IETF standards say that a MIME component line that is base64 encoded may not exceed 76 characters. That exchange is willing to "eat" such a line and that Outlook will "eat" it as well, processing it, is immaterial. Without IETF standards it would be very easy for BlackBerry (or anyone else) to point fingers and claim "well, it works with Microsoft!"

    Indeed, it is this very fact -- that it does, and that S/MIME has been reliant on Exchange -- that has allowed this bogus implementation to remain unaddressed for more than two years in the BB10 codebase.

    And then there are client side bugs with the features the mobile do choose to implement, as you noted with your SMIME comment and BlackBerry 10 devices. I can't tell you how many times I have dealt with iOS updates and their screwing up calendaring or poorly designing a EAS feature so they cause a sever to get bogged down. Apple is the worst offender of not QA'ing their EAS code.
    Apple and BlackBerry are "offenders" only because there is no formal, open, public IETF standard involved. Such is not the case for IMAP, CardDav and CalDav. I will note for the peanut gallery that CardDav has a number of server implementations that are bogus in some fashion in their behavior and BB10, which is standard-compliant, has had "fun" with them in some cases (E.g. duplication of contacts.) It is the fact of the specification being an IETF standard that makes possible the determination of exactly which end of the connection is at fault and why in this case.
    Meanwhile if you do choose to revisit ActiveSync, I highly encourage you to consider moving some of those folders out of your primary mailbox (an Archive mailbox is a great location if you store your email on Exchange again any time soon).
    Actually, an individual "mailbox" is the worst possible place to be storing a large volume of mail across multiple correspondents for a whole host of reasons. The point of a mail directory structure is that it makes the storage of mail in a hierarchy (as you do for, say, spreadsheets or word processing documents) a matter of organization that makes sense given the specifics of the situation. This paradigm, including multi-level folder hierarchies, has been proved superior through the test of time for all sorts of filing and organization when it comes to computer data storage, and email archives are no exception to that rule.

    I find it amusing that the "answer" you and others have propounded to such a structure, which has worked perfectly fine with open standard (e.g. IMAP) clients for two decades, is to stick emails in a proprietary archive format such as a .PST file. It is especially amusing considering the amount of money I have made through the last two decades unscrambling someone's .PST file when they call me in a panic as something has gone wrong and correspondence they need has become inaccessible. Several of those organizations have seen the wisdom, after such an event has cost them a ton of money and occasionally led to data that cannot be retrieved in a reasonable fashion and at reasonable cost, or at all, of moving to a mail system that is not subject to such events.

    The point of my complaint here is simply that such a hierarchy structure has no business being stored as a complete "index" on a mobile device since it is both unnecessary and produces undesirable behavior (specifically, traffic volume and potential "break points" given the relative data store size available on both a mobile and server device.)

    Hopefully BlackBerry will divorce its S/MIME implementation from the Exchange requirement entirely; Exchange is not necessary for S/MIME in any way and is simply a dodge as BlackBerry is using Exchange to separate out the signature (and encrypted portion) attachments rather than doing it in the client. This decision might be defensible if the client didn't know how to deal with attachments itself, but the client does know how to do so (as do essentially all modern email clients) and as such there is no defensible reason for them to have undertaken this decision in the first place.

    This decision is especially egregious in that BlackBerry's email client and handset software does not require back-end services for either certificate verification (in fact it insists that the certificates in the chain up to the CA be on the handset) and further, the handset also knows how to handle OCSP revocation checks on its own.
    04-25-15 09:10 AM
  24. Hennry Satria's Avatar
    Hi All,

    I am using BB Z10 with 10.3.1.1779 OS Version; I also setup and established Outlook.com with Exchange ActiveSync on this device since around one year.

    Everything was fine until some weeks ago suddenly the device accessing and downloading lots of data from Outlook.com that filled the storage up to some gigabytes, this was very strange since although downloading but no e-mail appears beyond 30 days that I setup on the device.

    The device kept downloading lots data and filled the storage for even more gigabytes, the only thing that can solve this problem is by deleting Outlook.com mail account and re-add it.

    Does anyone have the same problem? Can anyone explain about this phenomenon?

    Also, why is this device not able to sync the e-mail draft? If I wrote e-mail draft on device, it does not sync at Outlook.com server or web interface.

    Thank you.
    05-16-15 08:03 AM
  25. byex's Avatar
    Just archive your email. Anything prior to 5-10 years ago. Problem fixed.

    Posted via CB10
    05-16-15 08:57 AM
28 12

Similar Threads

  1. Time to spend some $ and update CB App
    By Mr4aces in forum CrackBerry Apps
    Replies: 72
    Last Post: 10-07-15, 07:43 PM
  2. Saving messages/email?
    By SharonVA in forum BlackBerry Classic
    Replies: 9
    Last Post: 04-28-15, 02:04 PM
  3. Are or when are the blue or bronze going to be available ?
    By CrackBerry Question in forum Ask a Question
    Replies: 3
    Last Post: 04-20-15, 09:03 PM
  4. Replies: 1
    Last Post: 04-20-15, 06:28 PM
  5. Should BlackBerry just have tweaked and the Z3?
    By m3mb3rsh1p in forum BlackBerry Leap
    Replies: 0
    Last Post: 04-20-15, 03:57 PM
LINK TO POST COPIED TO CLIPBOARD