E-mail Push (EAS / IMAP IDLE) problems connected via WIFI
- Got my Z10 a few days ago. Love it. But I'm having a problem with the push/pull/sync that I can't figure out.
I have setup four e-mail accounts on my Z10: Videotron IMAP, Yahoo IMAP, Live.com (EAS), and GMail (IMAP).
I've been testing the e-mail push sending e-mails to and from the various accounts, and what I have discovered is that when I am connected via wifi (at my home), the e-mails won't come in via push. Every now and then one WILL come in pretty quickly, but the vast majority of them will not come in until I do a refresh or I guess the poll interval gets hit. I have no wireless problems with this router. The home network is completely stable. My ISP is Videotron. My mobile carrier is Koodo.
So I tested the process without the wifi connection, on the mobile network (LTE), and from each e-mail account the e-mails come in almost instantaneously! Beautiful. Thus, I assume that the IMAP Idle is working properly (because I set the polling interval to a high number just for testing purposes) - though Yahoo is funkier than the others.
If I do a refresh, the e-mails will all come in. I assume that is a result of polling.
Any suggestions? Thank you.02-27-13 12:06 AMLike 0 - Both IMAP IDLE and EAS work by keeping a long-lived connection open from the device to the server. High network latency can interfere with that, causing your device to drop the connection. That could be a problem on your network, or with your ISP. The fact that this happens with all of your accounts suggests it's not a node somewhere else that's the problem. This is not a speed or reliability issue, but a long routing time, which can cause the same symptoms.
If you can, I'd suggest swapping out your router and taking other devices off the network to see if it's a problem in your house. If that doesn't solve it, give Videotron CS a call and see if they can find a problem with your connection from their centre.02-27-13 12:38 AMLike 0 - Thanks for the reply.
Well, I just did deeper testing of the servers. I used telnet and openssl to connect to and log into the videotron, yahoo, and gmail servers directly. I also discovered that the login banner of capabilities may be wrong. To wit, this is the login message from imap.videotron.ca:
Code:* OK [CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS CHILDREN BINARY UNSELECT SORT LANGUAGE XSENDER X-NETSCAPE XSERVERINFO AUTH=PLAIN] Messaging Multiplexor (Sun Java(tm) System Messaging Server 6.3-5.02 (built Oct 12 2007))
I then logged into the rim.imap.mail.yahoo.com server, which is what the Z10 entered in the server field for my Yahoo e-mail account. The banner does not show IDLE and it does not respond to the command. Therefore the Z10/Yahoo e-mail combination is stuck with polling only. That explains why even when I was connected via LTE it still was the laggard of the 4 servers I tested.
Next I tested the imap.gmail.com server with openssl. As we already knew, I confirmed that it definitely supports imap idle.
I did not test the microsoft live.com server because it uses Exchange ActiveSync and therefore one would think it would work the best with the Z10 - but I still get long delays with e-mails sent to that address if I'm connected via wifi. Connected via LTE, the push definitely worked.
DannySmurf, you spoke of network latency and long routing time, which makes sense. I have a DLINK DIR825 router, very little wireless (or other) traffic (two clients - an itouch and a laptop), and the router is located about 2 feet from the Z10. Could it be as simple as a setting in the router? I googled that subject but found nothing. I agree that it is likely something to do with either my system here (cablemodem, router, ?) or with Videotron (ISP). Since I only got the phone on Saturday night, and I've never had a smartphone, Blackberry, etc., it's taken me a few days to learn enough about the O/S, push, polling, imap, activesync, etc., to get a handle on what's what. In the coming days I will try to test the Z10 while connected to other routers/wifi networks using just gmail and videotron e-mail servers. Oh, and thanks for the laugh about calling videotron's CS for help.Last edited by slickvguy; 02-27-13 at 03:15 AM.
02-27-13 01:54 AMLike 0 - OmnitechDragon SlayerI then logged into the rim.imap.mail.yahoo.com server, which is what the Z10 entered in the server field for my Yahoo e-mail account. The banner does not show IDLE and it does not respond to the command. Therefore the Z10/Yahoo e-mail combination is stuck with polling only. That explains why even when I was connected via LTE it still was the laggard of the 4 servers I tested.
I did not test the microsoft live.com server because it uses Exchange ActiveSync and therefore one would think it would work the best with the Z10 - but I still get long delays with e-mails sent to that address if I'm connected via wifi. Connected via LTE, the push definitely worked.
I agree that it is likely something to do with either my system here (cablemodem, router, ?) or with Videotron (ISP). Since I only got the phone on Saturday night, and I've never had a smartphone, Blackberry, etc., it's taken me a few days to learn enough about the O/S, push, polling, imap, activesync, etc., to get a handle on what's what. In the coming days I will try to test the Z10 while connected to other routers/wifi networks using just gmail and videotron e-mail servers.
Therefore I'd give about a 95% likelihood that the problem is that your network devices there are dropping the IMAP IDLE session before it has a chance to send a keepalive packet to keep the session open. (IMAP IDLE keepalive packets might be sent every 15 minutes or so, depending on the implementation)
The test I would try is do a manual IMAP sync/refresh, then within say 15-30 seconds, send an incoming message to that account, and see if it shows up on the device right away. The refresh should reset the firewall/router session table so the incoming traffic/notification should be able to traverse the existing socket.
If that works, I'd say that pretty much confirms it. You may not be able to configure things like session timeout on a lot of home routers, but if that test is positive that is the next place I would look if I were you. (The reason it works over the cellular network is that their network is already setup to not drop those long-lived IMAP sessions. Apparently Rogers in Canada were not properly setup in that way during the initial rollout of the Z10 and they subsequently made changes on their network after a lot of people complained about IMAP PUSH issues.)
HTH02-27-13 03:37 AMLike 0 - Thank you Omnitech. I now understand. It makes sense and I believe you are correct because I tried two different things, both resulted in the push working over wifi immediately. The first, was turning airplane mode on and off (after reading on other threads that that had worked for some people). I assume that caused the same/similar reset that you described? The second, was exactly what you had advised: do a manual refresh. Now I need to find out what is causing this to happen and if there is anything I can do about it. I am renting the router, so I can easily change it if I want to. I'm using Arch linux on my main PC - could that have something to do with it?
The wifi push issue is not that critical while I'm at home (I work from home). I can view incoming e-mails in my Thunderbird client and the web clients. I was just using my wifi here to test the e-mail push and was dismayed at the long delays. Now that I know that the LTE works fine, I'm relieved, because that's far more important to me (when I'm on the road).
When you said "network devices are dropping the IMAP IDLE session before a keep-alive is sent", were you referring to just the router or the other wireless devices? My daughter has a touch and a laptop - those are the only wireless devices on the network besides the Z10. What I am asking is if it is strictly a router issue - or could it be one of those devices causing the problem?
I find it very odd that the Videotron server's capability string does not include "IDLE", yet when you issue the IDLE command it accepts it ("+ idling").
Another question: the other sync functions (e.g. when you delete an e-mail) - are they supposed to occur at the same time as the push e-mail? Are those things also a function of IMAP IDLE orr is that only done at the sync intervals and a separate process? Because I notice that those kinds of things happen later (sometimes just a short while after, other times longer delay).
p.s. While connecting to the yahoo server with openssl I saw " XAPPLEPUSHSERVICE" in the capability message. Would be nice if they had a XBBPUSHSERVICE02-27-13 04:28 AMLike 0 - OmnitechDragon SlayerThat's interesting that "applepushservice", hadn't heard about that before. It appears to be a proprietary thing that Apple requires the server side to use an Apple-provided certificate in order to participate in:
Apple Push Service
Apple Push Notification Service - Wikipedia, the free encyclopedia
And it seems to have a sort of unique device identifier, perhaps not unlike a classic Blackberry "PIN", and appears to require special routing via the carrier, sort-of like "BIS-lite". Very interesting.
Theoretically that could give Apple a significant competitive advantage if it were actually true push, but if the wikipedia article is correct that it needs to keep a persistent connection open, it's not any better in that regard than IMAP or EAS.
In regards to your network, the usual culprit is any sort of firewall or NAT device that uses a "stateful" architecture, which means when the firewall or NAT device sees ie an outgoing connection attempt, it examines the request and assuming it passes the firewall rules, creates a dynamic "session" which listens for replies to the request, eliminating the need for creating static port-by-port firewall rules on the incoming side to allow replies to outgoing requests.
Typically each "session" will stay active for some predetermined length of time, to allow time for replies to the outgoing request to reach the initiator. Typical session timeout values vary from perhaps 60 seconds to 60 minutes. You may want to go through your firewall reference guide and see if it allows you to adjust the session timeout value.
In the case of IMAP IDLE I would say 60 minutes would be a good value. But if I were you I would try not to set that globally because it compromises security slightly and greatly increases the risk that the firewall will run out of resources if it is busy and holding all the sessions open for a long time. In the kinds of firewalls I usually deal with I can set the session timeout on a per-protocol or per-rule basis.
Regarding whether multiple actions are pipelined on an IMAP IDLE request, I think it depends on various optional IMAP functions. There are not-yet-standardized IMAP extensions (often referred to as the "Lemonade profile") which among other things provide standard ways to group multiple actions together. Reading material:
RFC 3501 - IMAP4v1 (note all the links to updates at the top)
RFC 2177 - IMAP4 IDLE command
Lemonade Profile - Wikipedia, the free encyclopedia02-27-13 07:44 AMLike 0 - Thanks for the additional info and links.
I do not have a separate firewall. It's just a typical home network setup. A DIR-825 run of the mill wifi router w/ NAT connected to a cablemodem. My PC connected directly to the port and three wireless devices (an itouch, a Dell laptop, and the Z10). When you log into the router's s/w, there are quite a few settings (more than I've seen on other home routers) for firewall (TCP endpoint filtering is set to "Port And Address Restricted" - other choices are Endpoint Independent or Address Restricted), routing, port forwarding, filters, etc. (Hmm...I wonder if using DD-WRT might help?). It seems quite configurable.
When I did further testing yesterday using refresh and other things, I noticed that sometimes even after a refresh there is still no push! e.g. The live.com account (which as you know uses EAS not IMAP), did not immediately send the e-mails it received even after a refresh (there was still a delay of I'd say about 5 minutes).
I spent some time looking at the "Internet Sessions" table of the router at various times after doing different things. I noticed that even after a refresh or a manual wifi disconnect/connect, there is not a session for some of the servers.
Code:This is after a manual Z10 refresh in the Hub: 192.168.0.101:16012 16012 173.194.76.109:993 TCP EST OUT 119 ;GOOGLE 192.168.0.101:34602 34602 216.9.242.66:52164 TCP EST OUT 112 ;RIMBB-IPV4-02 192.168.0.101:15620 15620 98.138.215.4:993 TCP EST OUT 119 ;A-YAHOO-US9 192.168.0.101:37327 37327 206.53.159.203:443 TCP EST OUT 118 ;RIMBB-IPV4-03
Code:After I logged into live.com, then the e-mails got pushed, and this was in the table: 192.168.0.101:19466 19466 207.46.11.152:443 TCP EST OUT 69 ; MICROSOFT port 443 (EAS) After deleting a sent e-mail item on the z10 that had been sent from videotron 192.168.0.101:24468 24468 24.201.245.33:143 TCP EST OUT 119 ; Videotron imap
Is it possible that the Z10's s/w is not initiating the connections when it should be? e.g. Why wouldn't the manual Hub refresh immediately establish a session for ALL FOUR of the e-mail accounts/servers - at least temporarily? I can understand if the sessions wouldn't stay open long enough - but they should still be established after a refresh, no?
I will do more reading and testing.02-27-13 04:52 PMLike 0 - OmnitechDragon SlayerWell your firewall is in your router, basically.
When I did further testing yesterday using refresh and other things, I noticed that sometimes even after a refresh there is still no push! e.g. The live.com account (which as you know uses EAS not IMAP), did not immediately send the e-mails it received even after a refresh (there was still a delay of I'd say about 5 minutes).
I spent some time looking at the "Internet Sessions" table of the router at various times after doing different things. I noticed that even after a refresh or a manual wifi disconnect/connect, there is not a session for some of the servers.
Code:This is after a manual Z10 refresh in the Hub: 192.168.0.101:16012 16012 173.194.76.109:993 TCP EST OUT 119 ;GOOGLE 192.168.0.101:34602 34602 216.9.242.66:52164 TCP EST OUT 112 ;RIMBB-IPV4-02 192.168.0.101:15620 15620 98.138.215.4:993 TCP EST OUT 119 ;A-YAHOO-US9 192.168.0.101:37327 37327 206.53.159.203:443 TCP EST OUT 118 ;RIMBB-IPV4-03
What is the text after the semicolon characters in the session-list? Doesn't look like a hostname, doesn't look like a domain..
Are the numbers preceding the semicolon chars the session lifetime, do they count-down?
Port 993 is IMAP over TLS(SSL).
Port 143 is straight IMAP.
Port 443 is SSL, could be various things.
Well 206.53.159.203 definitely belongs to Blackberry so at least we have some idea what that is. Could be a million things. Would be good to hear from the horse's mouth what sort of "phoning home" Blackberry 10 does, though.
Once in a while I would see a Microsoft or Videotron session - but not when you'd expect it. NONE of those sessions to the e-mail servers last long. The timeout numbers are all low. Sometimes the timeout would count down and then get reinitialized to a higher number, but most of the time the timeout would just count down and close. Some more data:
Code:After I logged into live.com, then the e-mails got pushed, and this was in the table: 192.168.0.101:19466 19466 207.46.11.152:443 TCP EST OUT 69 ; MICROSOFT port 443 (EAS) After deleting a sent e-mail item on the z10 that had been sent from videotron 192.168.0.101:24468 24468 24.201.245.33:143 TCP EST OUT 119 ; Videotron imap
What figure does the timeout start out at? Looks like something around 120 seconds?
Are the internet sessions to the servers supposed to stay open all the time if it's working properly? Or are we talking about two different types of "sessions"?
Is it possible that the Z10's s/w is not initiating the connections when it should be? e.g. Why wouldn't the manual Hub refresh immediately establish a session for ALL FOUR of the e-mail accounts/servers - at least temporarily? I can understand if the sessions wouldn't stay open long enough - but they should still be established after a refresh, no?
99.9% likelihood the firewall is just timing out the sessions early, as I mentioned previously. Very common issue. If those session timeouts are only 120 seconds, that's pretty short, and that's probably why. IMAP IDLE needs to maintain a persistent connection typically anywhere from 15 to 59 minutes long.
IMAP IDLE implementations will typically send a "keepalive" packet something like every 15-30 minutes. If the firewall's session watchdog closes the session before the keepalive packet is sent, there will be no state-change notifications until the client opens a new session because the server has no way to initiate a new connection from the outside. (Most firewalls deny incoming traffic by default, and in your case you're behind a NAT so that wouldn't work anyway since they can't route over the internet to a private IP address-space anyway - connection needs to be initiated from behind the firewall/NAT.)
So what you need to look for is some setting that would allow you to adjust the session timeout values. I downloaded the manual (see below) and saw NO references in that document to either a configurable session-timeout value OR the "TCP endpoint filtering" settings you mentioned above.
http://www.dlink.com/us/en/support/p...11301130EN.pdf02-27-13 10:18 PMLike 0 - Wow this thread has been a plethora of information.
I was the OP on one of the other threads with this exact same issue. What I've been doing every morning is toggling airplane mode and that seems to have some sort of fix. But the "59min" time out seems very plausible from what I've been experiencing.
I am on Rogers Internet with the wireless modem that they provide and my z10 is on the Bell network. Interestingly before I read this thread I had come to the conclusion in my head that somehow my wifi network was becoming "Stale". Other networks that I'm on don't ever present me with this problem.
If there is ever a solution I'm dying to know!! For now my Gmail accounts are set to 15 min sync intervals so at the very least I should not receive mail that is more than 15 mins old, but still... This should be okay.
Posted via CB1002-27-13 10:59 PMLike 0 - And yes, I can confirm that this is happening on Gmail, EAS and Hotmail. Curses to my Rogers internet!
Posted via CB1002-27-13 11:00 PMLike 0 - What is the text after the semicolon characters in the session-list? Doesn't look like a hostname, doesn't look like a domain..
Are the numbers preceding the semicolon chars the session lifetime, do they count-down?
From the router's online help:
Code:Time Out The number of seconds of idle time until the router considers the session terminated. The initial value of Time Out depends on the type and state of the connection. 300 seconds - UDP connections. 240 seconds - Reset or closed TCP connections. The connection does not close instantly so that lingering packets can pass or the connection can be re-established. 7800 seconds - Reset or closed TCP connections. The connection does not close instantly so that lingering packets can pass or the connection can be re-established.
What figure does the timeout start out at? Looks like something around 120 seconds?
I'm playing around with it right now as we speak. Logging into servers. Issuing the IDLE command, doing various things w/ the e-mails/sync/etc, and watching the session tables and how the servers respond. When I connect to the servers from my PC (plugged into the router port), one difference is that you see the session immediately and it persists. You don't see a higher timeout than with the Z10/wifi, but it the session keeps refreshing or extending itself, if you know what I mean. It stays alive. When I issue the IDLE command and send a new e-mail to that server, the client shows the updated EXISTS, RECENT, etc., info being sent from the server.
Another thing (in case anyone using videotron imap stumbles onto this thread). RFC2177 says the following:
If the server does not advertise the IDLE capability, the client MUST NOT use the IDLE command and must poll for mailbox updates.
This part from RFC2177 was interesting too:
The server MAY consider a client inactive if it has an IDLE command
running, and if such a server has an inactivity timeout it MAY log
the client off implicitly at the end of its timeout period. Because
of that, clients using IDLE are advised to terminate the IDLE and
re-issue it at least every 29 minutes to avoid being logged off.
This still allows a client to receive immediate mailbox updates even
though it need only "poll" at half hour intervals.
Do you know how I would be able to monitor the (decrypted) communication between the Z10 and the router? . I use Arch linux. Something on BackTrack perhaps?02-27-13 11:22 PMLike 0 - OmnitechDragon SlayerEauRouge: probably has nothing to do with "Rogers Internet". More than likely an issue with your network hardware, as outlined above.
You can still just set your IMAP or EAS accounts to just poll ie every 15 minutes, should work fine. You just lose "instant" email delivery and at the cost of potentially higher battery usage on the handheld.02-28-13 12:01 AMLike 0 - OmnitechDragon SlayerFrom the router's online help:
Code:Time Out The number of seconds of idle time until the router considers the session terminated. The initial value of Time Out depends on the type and state of the connection. 300 seconds - UDP connections. 240 seconds - Reset or closed TCP connections. The connection does not close instantly so that lingering packets can pass or the connection can be re-established. 7800 seconds - Reset or closed TCP connections. The connection does not close instantly so that lingering packets can pass or the connection can be re-established.
You'd assume from the above that it would start at either 240 or 7800. But I've never seen a timeout number even close to 240. Your 120 guess is closer to what I normally see. But what happens is that they frequently won't count down to 0, instead they will get bumped back up, but NOT to 240 or 120. I'll do a screen refresh and it'll jump from let's say 45 to 85. Weird. Also, I sometimes see more than one entry for the same ip address and port combination (different NAT number) and different timeouts. Multiple connections, same IP address and port.
Is there no control for adjusting the timeout value? If not, I wonder if you put a specific rule in there that specifies IMAP as the protocol if the device might adjust that. (Though honestly I think we may be expecting far grander things from an inexpensive home router than we might have a right to expect. )
When I connect to the servers from my PC (plugged into the router port), one difference is that you see the session immediately and it persists. You don't see a higher timeout than with the Z10/wifi, but it the session keeps refreshing or extending itself, if you know what I mean. It stays alive. When I issue the IDLE command and send a new e-mail to that server, the client shows the updated EXISTS, RECENT, etc., info being sent from the server.
Although Videotron does not state that it does IDLE in the string it sends to the client upon logging in, I know it does do IDLE. I was concerned that the Z10 would not attempt to use push with it. But when I sent it the CAPABILITY command, lo and behold the very last item is "IDLE". Brilliant.
It's easy enough to verify - but you have to be careful about security since you will no longer be firewalled or NAT'd. You can either:
- Disable the stateful firewall in your D-Link temporarily (may not be feasible if you can't also disable NAT)
- Find a way to connect your Z10 directly to the cablemodem. (ie plug a plain WiFi access-point into the cablemodem and link the Z10 to that)
My ISP and/or the Z10 s/w could be the culprit. Maybe the client (Z10) is not terminating the IDLE and re-issuing it and then the SERVER is disconnecting? Anyway, I searched all day long for solutions, but didn't come up with anything. If it was for my PC as the client, then I'd be able to change /procs/sys/net/ipv4/tcp_keepalive_time and I think that would solve it.
Changing the TCP keepalive time on your PC isn't going to help if the firewall is killing the session at the border. According to the following document (not exactly authoritative, just easy to lookup) the minimum TCP keepalive value should already be 2 hours anyway:
Keepalive - Wikipedia, the free encyclopedia
The multiple connections to the same port are normal, many or most IP applications commonly open multiple connections to the same host to do what they need to do, including web browsers. There may be a separate socket/thread doing an IMAP watchdog/keepalive, a different one in the process of doing a sync of some kind, a different one sending a message, etc.
Now you're getting into a bit dicey hacking territory, trying to break into public-key encrypted communications. Suffice to say there are ways but you sorta have to be a bad guy to do them with a clear conscience, and have access to extraordinary resources, and I'm not feeling in a particularly educational mood on that subject at the moment.
Anyway, the SSL link is actually not between the router and the Z10, it's between the server and the Z10 via the router. The easiest way to troubleshoot that would be to just turn off encryption for IMAP on that connection. If you can find an IMAP server to test with that will do an unencrypted IMAP session, that would be ideal. Though I'm not sure they're very easy to find, as that would mean auth credentials being sent in the clear etc.02-28-13 12:45 AMLike 0 - Progress! (Not on a solution - but on what is taking place).
I have confirmed that the router is using 120 (not 240) for the initial timeout value every single time for every TCP session I have observed regardless of IP address or port.
I used my PC (directly wired to router) to connect to imap.google.com via openssl. Each time I sent a command to the server, it reset the timeout to 120. If the session timed out (went to 0 and then disappeared from the table), if I issued another command, it would start another session, at 120. Then I sent the IDLE command (which reset the timeout to 120) and used my Z10 to send an e-mail to my gmail account before the 120 had elapsed. As expected, I got the EXISTS message from the server indicating the new message count. Then I tried it again, with no e-mails sent, let the timeout elapse while in IDLE mode, sent the DONE, and I got an "ok" from the server. But....when I did the same thing, but allowed the timeout to elapse BEFORE sending the e-mail to my gmail account (and waited to verify that the e-mail had been received by keeping an eye on my browser tab's title text ("Inbox (x)") and seeing that the (x) incremented) - no EXISTS message was sent by the server. Most importantly, when I then sent a DONE from the client, IT ERRED OUT! Each time I tried I had the same result. In hindsight, I guess that this seems predictable and obvious, but it's one thing to think it quite another to actually see it in realtime. So the server was fine with the timing out in IDLE mode as long as no e-mail was received, but when the server received an e-mail and the connection had timed out, it crapped out. Unless I'm wrong, this is what seems to be happening over and over with the Z10/router/ISP/mail server dance.
Omnitech (or anyone else), can you please verify and confirm to me that your router uses a much higher timeout for the session when the connection from the client (Z10) to the e-mail server is established? I find it very odd that such a popular home router always using 120 would cause such problems! I've never had trouble using any other software. I guess the other software continuously establishes new sessions or keeps prolonging the same session (continuously restarts the timeout clock, as it were)? But in this case, the server in imap IDLE mode EXPECTS that the same session will be available, the client isn't sending a damn thing to keep that session alive, and when it isn't there, voila no e-mail push.02-28-13 12:50 AMLike 0 - I can't figure out what the distinction is between those last 2 lines in the help file list other than the timeout value - did you mis-copy something there?
No, there is no way within the router's s/w user interface to adjust the timeout. I need a way to fake it out somehow. That 7800 is in the guide. Hopefully, there is a way to get it to use that (or at least a number that is great enough).
Yes, I'm using Telnet or OpenSSL to log into the servers.
Backtrack is a liveCD, you mean one of the tools they have on there?
I had thought of using a direct connection to test. I don't even have to use the Z10. I guess I can connect my PC directly to the cablemodem and test the push, but I wouldn't be able to monitor it. I'll see if I can't try another router too. Will report back any findings. Thank you for your help.02-28-13 01:10 AMLike 0 - OmnitechDragon Slayer
So the server was fine with the timing out in IDLE mode as long as no e-mail was received, but when the server received an e-mail and the connection had timed out, it crapped out. Unless I'm wrong, this is what seems to be happening over and over with the Z10/router/ISP/mail server dance.
Omnitech (or anyone else), can you please verify and confirm to me that your router uses a much higher timeout for the session when the connection from the client (Z10) to the e-mail server is established? I find it very odd that such a popular home router always using 120 would cause such problems! I've never had trouble using any other software. I guess the other software continuously establishes new sessions or keeps prolonging the same session (continuously restarts the timeout clock, as it were)? But in this case, the server in imap IDLE mode EXPECTS that the same session will be available, the client isn't sending a damn thing to keep that session alive, and when it isn't there, voila no e-mail push.
Re: the firewall timeouts in general: IMAP IDLE is relatively unusual in how it expects to maintain a very "long-lived" but mostly idle connection like that. In a non-mobile environment, it's not a big deal to keep sending periodic packets to keep connections open, but that's often not feasible on a mobile device because even to send one tiny IP packet you have to wakeup the radio and use significant battery energy.
The only other situation where I have personally run into this much is when I want to maintain for some reason a very long-lived SSH connection to something, ie for days on end. In that case I set an option in OpenSSH to send periodic keepalive packets, which have the effect of keeping the firewall session open so the SSH session doesn't drop. The general issue is the same as the IMAP IDLE issue here.
And it's not technically true that the client doesn't send "a darn thing" to keep the session alive - what is probably happening (as stated previously) is that your firewall session timeout is just shorter than the interval between those keepalive packets. As you discovered, the IMAP standard suggests closing the session entirely after no more than 59 minutes and re-opening a new one anyway.
The firewalls that I normally use (Netscreen/Juniper) have a much longer default session timeout for TCP connections, about 30 mins or so. (Though that varies on a per-protocol basis anywhere from 1 minute to 120 minutes) So I'd guess that by default I'd probably never see that issue here, though I personally don't normally have anything running on IMAP IDLE continuously anyway.
On an inexpensive home router, it's not uncommon to have shorter timeouts because it preserves resources on a device with limited horsepower and memory, and it also helps keep them less vulnerable to certain types of DoS attacks designed to open a bunch of simultaneous connections and starve the device of resources. (On a fancier security device, there are typically certain algorithms designed to minimize the impact of such attacks in a more elegant way without just clamping-down all the session timeouts globally)
But there are a bunch of devices out there that are a few years old that have a lot of those fancy capabilities, ie old Netscreen 5XT or 5GT devices you can often find on ebay for less than $100.
Another option could be to try DD-WRT, I'd guess it's either got session timeout configurability or is at least smart enough to set a long timeout if you put a rule in there that specifies IMAP as the protocol/service.
Let us know how you do. If you were close to me I'd give you a good deal on one of my old Netscreens or maybe something else from the archive pile.02-28-13 01:49 AMLike 0 - Thats a lot of technical info! Some of it goes above me. Wanted to ask what is the ideal setting for push and sync options so as to save up on the data being consumed due to the constant connection. Another new problem for me is that i cannot seem to set up my previously deleted yahoo account back again. It gives an error msg after i enter the email and password " the server imap.yahoo.com for your account is not responding. Pls cheack your settings and try again". I didn't do any changes...so am wondering where have i gone wrong. Help, anyone?02-28-13 01:59 AMLike 0
- Omnitech, I'm in Canada.
FWIW, I found a revised copy of the help guide on dlink's website. Here's what the Time Out section was supposed to say:
Time Out
The number of seconds of idle time until the router considers the session terminated.
The initial value of Time Out depends on the type and state of the connection.
300 seconds
UDP connections.
240 seconds
Reset or closed TCP connections.
The connection does not close instantly so that lingering packets can pass or the connection can be re-established.
7800 seconds
Established or closing TCP connections.
what is probably happening (as stated previously) is that your firewall session timeout is just shorter than the interval between those keepalive packets. As you discovered, the IMAP standard suggests closing the session entirely after no more than 59 minutes and re-opening a new one anyway.
re: wireshark: but it's on MY network (my WPA2 password), so why would that be a problem? I'm not trying to sniff someone else's communication - just my own.
I use Thunderbird on my linux box.02-28-13 02:25 AMLike 0 - Thats a lot of technical info! Some of it goes above me. Wanted to ask what is the ideal setting for push and sync options so as to save up on the data being consumed due to the constant connection. Another new problem for me is that i cannot seem to set up my previously deleted yahoo account back again. It gives an error msg after i enter the email and password " the server imap.yahoo.com for your account is not responding. Pls cheack your settings and try again". I didn't do any changes...so am wondering where have i gone wrong. Help, anyone?
FYI, on my Z10, the server is rim.imap.mail.yahoo.com (not imap.yahoo.com). That might be the issue. Not sure.02-28-13 02:28 AMLike 0 - Thanks for replying. It is rim.imap.yahoo.com...i didn't copy that properly...sorry. I didn't start a new thread since i needed to know about the push/sync settings. Don't mind starting a new one!02-28-13 02:34 AMLike 0
- Hm so would changing the router help?
Im currently on a modem/router combo from my ISP (Cisco DPC 3825) but I've heard of people calling in and getting the router portion disabled and only using it as a modem. Then I could connect any router to this02-28-13 07:25 AMLike 0 - Debian User - Idle TCP connections freeze
Also, might be useful to you guys. I don't understand code very well02-28-13 07:54 AMLike 0
- Forum
- BlackBerry 10 Phones & OS
- BlackBerry Z10
E-mail Push (EAS / IMAP IDLE) problems connected via WIFI
Similar Threads
-
BIS connection via WIFI problem - SIM Unlock HELP!!
By brake911 in forum General Carrier DiscussionReplies: 9Last Post: 12-06-09, 11:56 AM -
Junk Mail pushed through IMAP
By nextgates in forum General BlackBerry News, Discussion & RumorsReplies: 7Last Post: 05-14-09, 11:38 AM -
Problem connecting with WiFi since upgrading firmware
By listentothis in forum BlackBerry Curve SeriesReplies: 3Last Post: 01-20-09, 09:34 PM -
Problems connecting to WiFi
By dianap in forum BlackBerry Curve SeriesReplies: 3Last Post: 06-09-08, 02:00 PM
LINK TO POST COPIED TO CLIPBOARD