03-25-13 05:12 AM
40 12
tools
  1. slickvguy's Avatar
    Got my Z10 a few days ago. Love it. But I'm having a problem with the push/pull/sync that I can't figure out.

    I have setup four e-mail accounts on my Z10: Videotron IMAP, Yahoo IMAP, Live.com (EAS), and GMail (IMAP).

    I've been testing the e-mail push sending e-mails to and from the various accounts, and what I have discovered is that when I am connected via wifi (at my home), the e-mails won't come in via push. Every now and then one WILL come in pretty quickly, but the vast majority of them will not come in until I do a refresh or I guess the poll interval gets hit. I have no wireless problems with this router. The home network is completely stable. My ISP is Videotron. My mobile carrier is Koodo.

    So I tested the process without the wifi connection, on the mobile network (LTE), and from each e-mail account the e-mails come in almost instantaneously! Beautiful. Thus, I assume that the IMAP Idle is working properly (because I set the polling interval to a high number just for testing purposes) - though Yahoo is funkier than the others.

    If I do a refresh, the e-mails will all come in. I assume that is a result of polling.

    Any suggestions? Thank you.
    02-27-13 01:06 AM
  2. DannySmurf's Avatar
    Both IMAP IDLE and EAS work by keeping a long-lived connection open from the device to the server. High network latency can interfere with that, causing your device to drop the connection. That could be a problem on your network, or with your ISP. The fact that this happens with all of your accounts suggests it's not a node somewhere else that's the problem. This is not a speed or reliability issue, but a long routing time, which can cause the same symptoms.

    If you can, I'd suggest swapping out your router and taking other devices off the network to see if it's a problem in your house. If that doesn't solve it, give Videotron CS a call and see if they can find a problem with your connection from their centre.
    02-27-13 01:38 AM
  3. slickvguy's Avatar
    Thanks for the reply.

    Well, I just did deeper testing of the servers. I used telnet and openssl to connect to and log into the videotron, yahoo, and gmail servers directly. I also discovered that the login banner of capabilities may be wrong. To wit, this is the login message from imap.videotron.ca:

    Code:
    * OK [CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS CHILDREN BINARY UNSELECT SORT LANGUAGE XSENDER
     X-NETSCAPE XSERVERINFO AUTH=PLAIN] Messaging Multiplexor (Sun Java(tm) System Messaging Server 6.3-5.02 (built Oct 12 2007))
    It does not list IDLE as a capability. I was prepared to conclude that videotron does not support IDLE, except I knew that when I used the LTE network (not wifi), e-mails sent to that server definitely were immediately pushed to the Z10. So I issued a few commands and then the IDLE command - and it accepted the IDLE! Unless I'm missing something, it seems that the videotron server does indeed support imap idle (push). Good to know.

    I then logged into the rim.imap.mail.yahoo.com server, which is what the Z10 entered in the server field for my Yahoo e-mail account. The banner does not show IDLE and it does not respond to the command. Therefore the Z10/Yahoo e-mail combination is stuck with polling only. That explains why even when I was connected via LTE it still was the laggard of the 4 servers I tested.

    Next I tested the imap.gmail.com server with openssl. As we already knew, I confirmed that it definitely supports imap idle.

    I did not test the microsoft live.com server because it uses Exchange ActiveSync and therefore one would think it would work the best with the Z10 - but I still get long delays with e-mails sent to that address if I'm connected via wifi. Connected via LTE, the push definitely worked.

    DannySmurf, you spoke of network latency and long routing time, which makes sense. I have a DLINK DIR825 router, very little wireless (or other) traffic (two clients - an itouch and a laptop), and the router is located about 2 feet from the Z10. Could it be as simple as a setting in the router? I googled that subject but found nothing. I agree that it is likely something to do with either my system here (cablemodem, router, ?) or with Videotron (ISP). Since I only got the phone on Saturday night, and I've never had a smartphone, Blackberry, etc., it's taken me a few days to learn enough about the O/S, push, polling, imap, activesync, etc., to get a handle on what's what. In the coming days I will try to test the Z10 while connected to other routers/wifi networks using just gmail and videotron e-mail servers. Oh, and thanks for the laugh about calling videotron's CS for help.
    Last edited by slickvguy; 02-27-13 at 04:15 AM.
    02-27-13 02:54 AM
  4. Omnitech's Avatar
    I then logged into the rim.imap.mail.yahoo.com server, which is what the Z10 entered in the server field for my Yahoo e-mail account. The banner does not show IDLE and it does not respond to the command. Therefore the Z10/Yahoo e-mail combination is stuck with polling only. That explains why even when I was connected via LTE it still was the laggard of the 4 servers I tested.
    It's a well-known fact at this point that Yahoo does not push to the Z10. It's not that Yahoo can't support push, but Yahoo has proprietary ways of doing push. One is a completely proprietary thing that I think does something over UDP, the other are non-standard extensions to IMAP (something related to "P-IMAP" as I recall) and those extensions/bastardizations of the protocol are probably not supported in the Z10 email app right now.


    I did not test the microsoft live.com server because it uses Exchange ActiveSync and therefore one would think it would work the best with the Z10 - but I still get long delays with e-mails sent to that address if I'm connected via wifi. Connected via LTE, the push definitely worked.
    None of the Microsoft webmail variants (live.com, hotmail.com, outlook.com) support IMAP. They support EAS and POP3 only. (EAS is really quite a bit more sophisticated than IMAP anyway, so it's no great loss IMHO)


    I agree that it is likely something to do with either my system here (cablemodem, router, ?) or with Videotron (ISP). Since I only got the phone on Saturday night, and I've never had a smartphone, Blackberry, etc., it's taken me a few days to learn enough about the O/S, push, polling, imap, activesync, etc., to get a handle on what's what. In the coming days I will try to test the Z10 while connected to other routers/wifi networks using just gmail and videotron e-mail servers.
    IMAP IDLE requires a long-lived TCP socket/connection to remain open for as long as 59 minutes at a time in order for the server to be able to signal the client when a state change occurs. Otherwise if the connection drops and a stateful firewall session closes the server has no way to re-contact the client because it either has no way to route back through a NAT barrier or it will hit an inbound firewall rule and get dropped.

    Therefore I'd give about a 95% likelihood that the problem is that your network devices there are dropping the IMAP IDLE session before it has a chance to send a keepalive packet to keep the session open. (IMAP IDLE keepalive packets might be sent every 15 minutes or so, depending on the implementation)

    The test I would try is do a manual IMAP sync/refresh, then within say 15-30 seconds, send an incoming message to that account, and see if it shows up on the device right away. The refresh should reset the firewall/router session table so the incoming traffic/notification should be able to traverse the existing socket.

    If that works, I'd say that pretty much confirms it. You may not be able to configure things like session timeout on a lot of home routers, but if that test is positive that is the next place I would look if I were you. (The reason it works over the cellular network is that their network is already setup to not drop those long-lived IMAP sessions. Apparently Rogers in Canada were not properly setup in that way during the initial rollout of the Z10 and they subsequently made changes on their network after a lot of people complained about IMAP PUSH issues.)

    HTH
    02-27-13 04:37 AM
  5. slickvguy's Avatar
    Thank you Omnitech. I now understand. It makes sense and I believe you are correct because I tried two different things, both resulted in the push working over wifi immediately. The first, was turning airplane mode on and off (after reading on other threads that that had worked for some people). I assume that caused the same/similar reset that you described? The second, was exactly what you had advised: do a manual refresh. Now I need to find out what is causing this to happen and if there is anything I can do about it. I am renting the router, so I can easily change it if I want to. I'm using Arch linux on my main PC - could that have something to do with it?

    The wifi push issue is not that critical while I'm at home (I work from home). I can view incoming e-mails in my Thunderbird client and the web clients. I was just using my wifi here to test the e-mail push and was dismayed at the long delays. Now that I know that the LTE works fine, I'm relieved, because that's far more important to me (when I'm on the road).

    When you said "network devices are dropping the IMAP IDLE session before a keep-alive is sent", were you referring to just the router or the other wireless devices? My daughter has a touch and a laptop - those are the only wireless devices on the network besides the Z10. What I am asking is if it is strictly a router issue - or could it be one of those devices causing the problem?

    I find it very odd that the Videotron server's capability string does not include "IDLE", yet when you issue the IDLE command it accepts it ("+ idling").

    Another question: the other sync functions (e.g. when you delete an e-mail) - are they supposed to occur at the same time as the push e-mail? Are those things also a function of IMAP IDLE orr is that only done at the sync intervals and a separate process? Because I notice that those kinds of things happen later (sometimes just a short while after, other times longer delay).

    p.s. While connecting to the yahoo server with openssl I saw " XAPPLEPUSHSERVICE" in the capability message. Would be nice if they had a XBBPUSHSERVICE
    02-27-13 05:28 AM
  6. Omnitech's Avatar
    That's interesting that "applepushservice", hadn't heard about that before. It appears to be a proprietary thing that Apple requires the server side to use an Apple-provided certificate in order to participate in:

    Apple Push Service
    Apple Push Notification Service - Wikipedia, the free encyclopedia

    And it seems to have a sort of unique device identifier, perhaps not unlike a classic Blackberry "PIN", and appears to require special routing via the carrier, sort-of like "BIS-lite". Very interesting.

    Theoretically that could give Apple a significant competitive advantage if it were actually true push, but if the wikipedia article is correct that it needs to keep a persistent connection open, it's not any better in that regard than IMAP or EAS.

    In regards to your network, the usual culprit is any sort of firewall or NAT device that uses a "stateful" architecture, which means when the firewall or NAT device sees ie an outgoing connection attempt, it examines the request and assuming it passes the firewall rules, creates a dynamic "session" which listens for replies to the request, eliminating the need for creating static port-by-port firewall rules on the incoming side to allow replies to outgoing requests.

    Typically each "session" will stay active for some predetermined length of time, to allow time for replies to the outgoing request to reach the initiator. Typical session timeout values vary from perhaps 60 seconds to 60 minutes. You may want to go through your firewall reference guide and see if it allows you to adjust the session timeout value.

    In the case of IMAP IDLE I would say 60 minutes would be a good value. But if I were you I would try not to set that globally because it compromises security slightly and greatly increases the risk that the firewall will run out of resources if it is busy and holding all the sessions open for a long time. In the kinds of firewalls I usually deal with I can set the session timeout on a per-protocol or per-rule basis.

    Regarding whether multiple actions are pipelined on an IMAP IDLE request, I think it depends on various optional IMAP functions. There are not-yet-standardized IMAP extensions (often referred to as the "Lemonade profile") which among other things provide standard ways to group multiple actions together. Reading material:

    RFC 3501 - IMAP4v1 (note all the links to updates at the top)
    RFC 2177 - IMAP4 IDLE command
    Lemonade Profile - Wikipedia, the free encyclopedia
    02-27-13 08:44 AM
  7. slickvguy's Avatar
    Thanks for the additional info and links.

    I do not have a separate firewall. It's just a typical home network setup. A DIR-825 run of the mill wifi router w/ NAT connected to a cablemodem. My PC connected directly to the port and three wireless devices (an itouch, a Dell laptop, and the Z10). When you log into the router's s/w, there are quite a few settings (more than I've seen on other home routers) for firewall (TCP endpoint filtering is set to "Port And Address Restricted" - other choices are Endpoint Independent or Address Restricted), routing, port forwarding, filters, etc. (Hmm...I wonder if using DD-WRT might help?). It seems quite configurable.

    When I did further testing yesterday using refresh and other things, I noticed that sometimes even after a refresh there is still no push! e.g. The live.com account (which as you know uses EAS not IMAP), did not immediately send the e-mails it received even after a refresh (there was still a delay of I'd say about 5 minutes).

    I spent some time looking at the "Internet Sessions" table of the router at various times after doing different things. I noticed that even after a refresh or a manual wifi disconnect/connect, there is not a session for some of the servers.

    Code:
    This is after a manual Z10 refresh in the Hub:
    
    192.168.0.101:16012	16012	173.194.76.109:993	TCP	EST	OUT	119 ;GOOGLE
    192.168.0.101:34602	34602	216.9.242.66:52164	TCP	EST	OUT	112 ;RIMBB-IPV4-02
    192.168.0.101:15620	15620	98.138.215.4:993	TCP	EST	OUT	119 ;A-YAHOO-US9
    192.168.0.101:37327	37327	206.53.159.203:443	TCP	EST	OUT	118 ;RIMBB-IPV4-03
    Notice that there is no connection to the Microsoft or Videotron servers. Not sure what those RIM connections are, but they remain nearly constantly "on", even when there are no other open sessions from the Z10. (Is it for BBM? Firmware upgrades? Something else?). Once in a while I would see a Microsoft or Videotron session - but not when you'd expect it. NONE of those sessions to the e-mail servers last long. The timeout numbers are all low. Sometimes the timeout would count down and then get reinitialized to a higher number, but most of the time the timeout would just count down and close. Some more data:

    Code:
    After I logged into live.com, then the e-mails got pushed, and this was in the table:
    192.168.0.101:19466	19466	207.46.11.152:443	TCP	EST	OUT	69 ; MICROSOFT port 443 (EAS)
    
    After deleting a sent e-mail item on the z10 that had been sent from videotron
    192.168.0.101:24468	24468	24.201.245.33:143	TCP	EST	OUT	119 ; Videotron imap
    Are the internet sessions to the servers supposed to stay open all the time if it's working properly? Or are we talking about two different types of "sessions"?

    Is it possible that the Z10's s/w is not initiating the connections when it should be? e.g. Why wouldn't the manual Hub refresh immediately establish a session for ALL FOUR of the e-mail accounts/servers - at least temporarily? I can understand if the sessions wouldn't stay open long enough - but they should still be established after a refresh, no?

    I will do more reading and testing.
    02-27-13 05:52 PM
  8. Omnitech's Avatar
    I do not have a separate firewall.
    Well your firewall is in your router, basically.

    (TCP endpoint filtering is set to "Port And Address Restricted" - other choices are Endpoint Independent or Address Restricted), routing, port forwarding, filters, etc. (Hmm...I wonder if using DD-WRT might help?). It seems quite configurable.
    I'm not sure what those "endpoint filtering" settings are, I'd probably have to look at the UI or docs. DD-WRT is definitely highly configurable but it can also be a morass, and a pain to update because after all this time they still don't seem to have figured out how to update the firmware without blowing off the entire configuration and not having any reliable way to re-load it - so you end up having to re-configure everything from scratch again.


    When I did further testing yesterday using refresh and other things, I noticed that sometimes even after a refresh there is still no push! e.g. The live.com account (which as you know uses EAS not IMAP), did not immediately send the e-mails it received even after a refresh (there was still a delay of I'd say about 5 minutes).
    I'm not as familiar with the mechanism EAS uses for push - I believe the Microsoft terminology is "Direct Push" but for example I don't know if it runs on a separate set of ports or what.


    I spent some time looking at the "Internet Sessions" table of the router at various times after doing different things. I noticed that even after a refresh or a manual wifi disconnect/connect, there is not a session for some of the servers.

    Code:
    This is after a manual Z10 refresh in the Hub:
    
    192.168.0.101:16012	16012	173.194.76.109:993	TCP	EST	OUT	119 ;GOOGLE
    192.168.0.101:34602	34602	216.9.242.66:52164	TCP	EST	OUT	112 ;RIMBB-IPV4-02
    192.168.0.101:15620	15620	98.138.215.4:993	TCP	EST	OUT	119 ;A-YAHOO-US9
    192.168.0.101:37327	37327	206.53.159.203:443	TCP	EST	OUT	118 ;RIMBB-IPV4-03

    What is the text after the semicolon characters in the session-list? Doesn't look like a hostname, doesn't look like a domain..
    Are the numbers preceding the semicolon chars the session lifetime, do they count-down?

    Port 993 is IMAP over TLS(SSL).
    Port 143 is straight IMAP.
    Port 443 is SSL, could be various things.


    Notice that there is no connection to the Microsoft or Videotron servers. Not sure what those RIM connections are, but they remain nearly constantly "on", even when there are no other open sessions from the Z10. (Is it for BBM? Firmware upgrades? Something else?).

    Well 206.53.159.203 definitely belongs to Blackberry so at least we have some idea what that is. Could be a million things. Would be good to hear from the horse's mouth what sort of "phoning home" Blackberry 10 does, though.


    Once in a while I would see a Microsoft or Videotron session - but not when you'd expect it. NONE of those sessions to the e-mail servers last long. The timeout numbers are all low. Sometimes the timeout would count down and then get reinitialized to a higher number, but most of the time the timeout would just count down and close. Some more data:

    Code:
    After I logged into live.com, then the e-mails got pushed, and this was in the table:
    192.168.0.101:19466	19466	207.46.11.152:443	TCP	EST	OUT	69 ; MICROSOFT port 443 (EAS)
    
    After deleting a sent e-mail item on the z10 that had been sent from videotron
    192.168.0.101:24468	24468	24.201.245.33:143	TCP	EST	OUT	119 ; Videotron imap

    What figure does the timeout start out at? Looks like something around 120 seconds?


    Are the internet sessions to the servers supposed to stay open all the time if it's working properly? Or are we talking about two different types of "sessions"?

    Is it possible that the Z10's s/w is not initiating the connections when it should be? e.g. Why wouldn't the manual Hub refresh immediately establish a session for ALL FOUR of the e-mail accounts/servers - at least temporarily? I can understand if the sessions wouldn't stay open long enough - but they should still be established after a refresh, no?
    I don't think the issue is with the Z10 - if that were the case it wouldn't work over 3G/4G either.

    99.9% likelihood the firewall is just timing out the sessions early, as I mentioned previously. Very common issue. If those session timeouts are only 120 seconds, that's pretty short, and that's probably why. IMAP IDLE needs to maintain a persistent connection typically anywhere from 15 to 59 minutes long.

    IMAP IDLE implementations will typically send a "keepalive" packet something like every 15-30 minutes. If the firewall's session watchdog closes the session before the keepalive packet is sent, there will be no state-change notifications until the client opens a new session because the server has no way to initiate a new connection from the outside. (Most firewalls deny incoming traffic by default, and in your case you're behind a NAT so that wouldn't work anyway since they can't route over the internet to a private IP address-space anyway - connection needs to be initiated from behind the firewall/NAT.)

    So what you need to look for is some setting that would allow you to adjust the session timeout values. I downloaded the manual (see below) and saw NO references in that document to either a configurable session-timeout value OR the "TCP endpoint filtering" settings you mentioned above.

    http://www.dlink.com/us/en/support/p...11301130EN.pdf
    02-27-13 11:18 PM
  9. EauRouge's Avatar
    Wow this thread has been a plethora of information.

    I was the OP on one of the other threads with this exact same issue. What I've been doing every morning is toggling airplane mode and that seems to have some sort of fix. But the "59min" time out seems very plausible from what I've been experiencing.

    I am on Rogers Internet with the wireless modem that they provide and my z10 is on the Bell network. Interestingly before I read this thread I had come to the conclusion in my head that somehow my wifi network was becoming "Stale". Other networks that I'm on don't ever present me with this problem.

    If there is ever a solution I'm dying to know!! For now my Gmail accounts are set to 15 min sync intervals so at the very least I should not receive mail that is more than 15 mins old, but still... This should be okay.

    Posted via CB10
    02-27-13 11:59 PM
  10. EauRouge's Avatar
    And yes, I can confirm that this is happening on Gmail, EAS and Hotmail. Curses to my Rogers internet!

    Posted via CB10
    02-28-13 12:00 AM
  11. slickvguy's Avatar
    What is the text after the semicolon characters in the session-list? Doesn't look like a hostname, doesn't look like a domain..
    Are the numbers preceding the semicolon chars the session lifetime, do they count-down?
    The text after the semi-colon is just my comment (some of which I copied from the result of an ip lookup when I was checking to see which ip went with which server). I should have pasted the header from the internet sessions table. It's "Local / NAT / Internet Protocol / State / Dir / Time Out". The timeout is in seconds.

    From the router's online help:
    Code:
    Time Out
    The number of seconds of idle time until the router considers the session terminated.
    The initial value of Time Out depends on the type and state of the connection.
    300 seconds  - UDP connections. 
    240 seconds  - Reset or closed TCP connections. The connection does not close instantly so that lingering packets can pass or the connection can be re-established. 
    7800 seconds - Reset or closed TCP connections. The connection does not close instantly so that lingering packets can pass or the connection can be re-established.
    What figure does the timeout start out at? Looks like something around 120 seconds?
    You'd assume from the above that it would start at either 240 or 7800. But I've never seen a timeout number even close to 240. Your 120 guess is closer to what I normally see. But what happens is that they frequently won't count down to 0, instead they will get bumped back up, but NOT to 240 or 120. I'll do a screen refresh and it'll jump from let's say 45 to 85. Weird. Also, I sometimes see more than one entry for the same ip address and port combination (different NAT number) and different timeouts. Multiple connections, same IP address and port.

    I'm playing around with it right now as we speak. Logging into servers. Issuing the IDLE command, doing various things w/ the e-mails/sync/etc, and watching the session tables and how the servers respond. When I connect to the servers from my PC (plugged into the router port), one difference is that you see the session immediately and it persists. You don't see a higher timeout than with the Z10/wifi, but it the session keeps refreshing or extending itself, if you know what I mean. It stays alive. When I issue the IDLE command and send a new e-mail to that server, the client shows the updated EXISTS, RECENT, etc., info being sent from the server.

    Another thing (in case anyone using videotron imap stumbles onto this thread). RFC2177 says the following:
    If the server does not advertise the IDLE capability, the client MUST NOT use the IDLE command and must poll for mailbox updates.
    Although Videotron does not state that it does IDLE in the string it sends to the client upon logging in, I know it does do IDLE. I was concerned that the Z10 would not attempt to use push with it. But when I sent it the CAPABILITY command, lo and behold the very last item is "IDLE". Brilliant.

    This part from RFC2177 was interesting too:
    The server MAY consider a client inactive if it has an IDLE command
    running, and if such a server has an inactivity timeout it MAY log
    the client off implicitly at the end of its timeout period. Because
    of that, clients using IDLE are advised to terminate the IDLE and
    re-issue it at least every 29 minutes to avoid being logged off.
    This still allows a client to receive immediate mailbox updates even
    though it need only "poll" at half hour intervals.
    Bottom line: assuming that the timeout is the problem...I see no way around it. And are we really so sure that the ROUTER is the problem? I'll see if I can borrow someone else's wifi router for testing purposes. Yes, I agree, that it's a timeout issue. But there are a few participants in this dance. My ISP and/or the Z10 s/w could be the culprit. Maybe the client (Z10) is not terminating the IDLE and re-issuing it and then the SERVER is disconnecting? Anyway, I searched all day long for solutions, but didn't come up with anything. If it was for my PC as the client, then I'd be able to change /procs/sys/net/ipv4/tcp_keepalive_time and I think that would solve it. The router's online help says that it does 240 or 7800 and that the initial value depends on the type and state of the connection. I see no way to change this in the router and I see no way to change it in the Z10 OS (unless you've managed to root it? j/k)

    Do you know how I would be able to monitor the (decrypted) communication between the Z10 and the router? . I use Arch linux. Something on BackTrack perhaps?
    02-28-13 12:22 AM
  12. Omnitech's Avatar
    EauRouge: probably has nothing to do with "Rogers Internet". More than likely an issue with your network hardware, as outlined above.

    You can still just set your IMAP or EAS accounts to just poll ie every 15 minutes, should work fine. You just lose "instant" email delivery and at the cost of potentially higher battery usage on the handheld.
    02-28-13 01:01 AM
  13. Omnitech's Avatar
    From the router's online help:
    Code:
    Time Out
    The number of seconds of idle time until the router considers the session terminated.
    The initial value of Time Out depends on the type and state of the connection.
    300 seconds  - UDP connections. 
    240 seconds  - Reset or closed TCP connections. The connection does not close instantly so that lingering packets can pass or the connection can be re-established. 
    7800 seconds - Reset or closed TCP connections. The connection does not close instantly so that lingering packets can pass or the connection can be re-established.

    You'd assume from the above that it would start at either 240 or 7800. But I've never seen a timeout number even close to 240. Your 120 guess is closer to what I normally see. But what happens is that they frequently won't count down to 0, instead they will get bumped back up, but NOT to 240 or 120. I'll do a screen refresh and it'll jump from let's say 45 to 85. Weird. Also, I sometimes see more than one entry for the same ip address and port combination (different NAT number) and different timeouts. Multiple connections, same IP address and port.
    I can't figure out what the distinction is between those last 2 lines in the help file list other than the timeout value - did you mis-copy something there?

    Is there no control for adjusting the timeout value? If not, I wonder if you put a specific rule in there that specifies IMAP as the protocol if the device might adjust that. (Though honestly I think we may be expecting far grander things from an inexpensive home router than we might have a right to expect. )



    When I connect to the servers from my PC (plugged into the router port), one difference is that you see the session immediately and it persists. You don't see a higher timeout than with the Z10/wifi, but it the session keeps refreshing or extending itself, if you know what I mean. It stays alive. When I issue the IDLE command and send a new e-mail to that server, the client shows the updated EXISTS, RECENT, etc., info being sent from the server.
    Where are you monitoring that, via a telnet session or something?



    Although Videotron does not state that it does IDLE in the string it sends to the client upon logging in, I know it does do IDLE. I was concerned that the Z10 would not attempt to use push with it. But when I sent it the CAPABILITY command, lo and behold the very last item is "IDLE". Brilliant.
    Perhaps the reason the RFC has the rule it does is to avoid scenarios where a server implementation for some reason cannot disable IMAP IDLE, but the system adminstrator may wish to prohibit it to manage network/server resource usage.


    Bottom line: assuming that the timeout is the problem...I see no way around it. And are we really so sure that the ROUTER is the problem?
    I'd give it a very very high likelihood of that being the case.

    It's easy enough to verify - but you have to be careful about security since you will no longer be firewalled or NAT'd. You can either:

    • Disable the stateful firewall in your D-Link temporarily (may not be feasible if you can't also disable NAT)
    • Find a way to connect your Z10 directly to the cablemodem. (ie plug a plain WiFi access-point into the cablemodem and link the Z10 to that)


    My ISP and/or the Z10 s/w could be the culprit. Maybe the client (Z10) is not terminating the IDLE and re-issuing it and then the SERVER is disconnecting? Anyway, I searched all day long for solutions, but didn't come up with anything. If it was for my PC as the client, then I'd be able to change /procs/sys/net/ipv4/tcp_keepalive_time and I think that would solve it.
    I really doubt it has to do w/ the ISP or the Z10. If it works over 3G/4G, that pretty much eliminates the Z10. And I already know that lots of people have IMAP IDLE working with the Z10, so once again I wouldn't waste any energy pursuing that angle.

    Changing the TCP keepalive time on your PC isn't going to help if the firewall is killing the session at the border. According to the following document (not exactly authoritative, just easy to lookup) the minimum TCP keepalive value should already be 2 hours anyway:

    Keepalive - Wikipedia, the free encyclopedia

    The multiple connections to the same port are normal, many or most IP applications commonly open multiple connections to the same host to do what they need to do, including web browsers. There may be a separate socket/thread doing an IMAP watchdog/keepalive, a different one in the process of doing a sync of some kind, a different one sending a message, etc.


    Do you know how I would be able to monitor the (decrypted) communication between the Z10 and the router? . I use Arch linux. Something on BackTrack perhaps?
    Backtrack is a liveCD, you mean one of the tools they have on there?

    Now you're getting into a bit dicey hacking territory, trying to break into public-key encrypted communications. Suffice to say there are ways but you sorta have to be a bad guy to do them with a clear conscience, and have access to extraordinary resources, and I'm not feeling in a particularly educational mood on that subject at the moment.

    Anyway, the SSL link is actually not between the router and the Z10, it's between the server and the Z10 via the router. The easiest way to troubleshoot that would be to just turn off encryption for IMAP on that connection. If you can find an IMAP server to test with that will do an unencrypted IMAP session, that would be ideal. Though I'm not sure they're very easy to find, as that would mean auth credentials being sent in the clear etc.
    02-28-13 01:45 AM
  14. slickvguy's Avatar
    Progress! (Not on a solution - but on what is taking place).

    I have confirmed that the router is using 120 (not 240) for the initial timeout value every single time for every TCP session I have observed regardless of IP address or port.

    I used my PC (directly wired to router) to connect to imap.google.com via openssl. Each time I sent a command to the server, it reset the timeout to 120. If the session timed out (went to 0 and then disappeared from the table), if I issued another command, it would start another session, at 120. Then I sent the IDLE command (which reset the timeout to 120) and used my Z10 to send an e-mail to my gmail account before the 120 had elapsed. As expected, I got the EXISTS message from the server indicating the new message count. Then I tried it again, with no e-mails sent, let the timeout elapse while in IDLE mode, sent the DONE, and I got an "ok" from the server. But....when I did the same thing, but allowed the timeout to elapse BEFORE sending the e-mail to my gmail account (and waited to verify that the e-mail had been received by keeping an eye on my browser tab's title text ("Inbox (x)") and seeing that the (x) incremented) - no EXISTS message was sent by the server. Most importantly, when I then sent a DONE from the client, IT ERRED OUT! Each time I tried I had the same result. In hindsight, I guess that this seems predictable and obvious, but it's one thing to think it quite another to actually see it in realtime. So the server was fine with the timing out in IDLE mode as long as no e-mail was received, but when the server received an e-mail and the connection had timed out, it crapped out. Unless I'm wrong, this is what seems to be happening over and over with the Z10/router/ISP/mail server dance.

    Omnitech (or anyone else), can you please verify and confirm to me that your router uses a much higher timeout for the session when the connection from the client (Z10) to the e-mail server is established? I find it very odd that such a popular home router always using 120 would cause such problems! I've never had trouble using any other software. I guess the other software continuously establishes new sessions or keeps prolonging the same session (continuously restarts the timeout clock, as it were)? But in this case, the server in imap IDLE mode EXPECTS that the same session will be available, the client isn't sending a damn thing to keep that session alive, and when it isn't there, voila no e-mail push.
    02-28-13 01:50 AM
  15. slickvguy's Avatar
    I can't figure out what the distinction is between those last 2 lines in the help file list other than the timeout value - did you mis-copy something there?
    lol. I thought the exact same thing when I read it. No mis-copy. Probably a mistake on dlink's part. Doesn't make sense.

    No, there is no way within the router's s/w user interface to adjust the timeout. I need a way to fake it out somehow. That 7800 is in the guide. Hopefully, there is a way to get it to use that (or at least a number that is great enough).

    Yes, I'm using Telnet or OpenSSL to log into the servers.

    Backtrack is a liveCD, you mean one of the tools they have on there?
    Yes, that's what I meant. Like WireShark? I've used a few applications on BT in the past to, ahem, "investigate" wifi. We'll leave it at that. But as per my previous post, I no longer see a need to do it (though it would be fun to watch the back and forth).

    I had thought of using a direct connection to test. I don't even have to use the Z10. I guess I can connect my PC directly to the cablemodem and test the push, but I wouldn't be able to monitor it. I'll see if I can't try another router too. Will report back any findings. Thank you for your help.
    02-28-13 02:10 AM
  16. Omnitech's Avatar
    I have confirmed that the router is using 120 (not 240) for the initial timeout value every single time for every TCP session I have observed regardless of IP address or port.
    OK.



    So the server was fine with the timing out in IDLE mode as long as no e-mail was received, but when the server received an e-mail and the connection had timed out, it crapped out. Unless I'm wrong, this is what seems to be happening over and over with the Z10/router/ISP/mail server dance.
    Well I'm not sure what you mean specifically by "crapped out", I assume you mean the server didn't send any response back to the client since the TCP session was closed and there was no link to send that communication back on. As I predicted would happen.


    Omnitech (or anyone else), can you please verify and confirm to me that your router uses a much higher timeout for the session when the connection from the client (Z10) to the e-mail server is established? I find it very odd that such a popular home router always using 120 would cause such problems! I've never had trouble using any other software. I guess the other software continuously establishes new sessions or keeps prolonging the same session (continuously restarts the timeout clock, as it were)? But in this case, the server in imap IDLE mode EXPECTS that the same session will be available, the client isn't sending a damn thing to keep that session alive, and when it isn't there, voila no e-mail push.
    First of all, I can't buy a Z10 here (USA) yet so I am in no position to test this personally. Can I ask what part of the world you're in BTW?

    Re: the firewall timeouts in general: IMAP IDLE is relatively unusual in how it expects to maintain a very "long-lived" but mostly idle connection like that. In a non-mobile environment, it's not a big deal to keep sending periodic packets to keep connections open, but that's often not feasible on a mobile device because even to send one tiny IP packet you have to wakeup the radio and use significant battery energy.

    The only other situation where I have personally run into this much is when I want to maintain for some reason a very long-lived SSH connection to something, ie for days on end. In that case I set an option in OpenSSH to send periodic keepalive packets, which have the effect of keeping the firewall session open so the SSH session doesn't drop. The general issue is the same as the IMAP IDLE issue here.

    And it's not technically true that the client doesn't send "a darn thing" to keep the session alive - what is probably happening (as stated previously) is that your firewall session timeout is just shorter than the interval between those keepalive packets. As you discovered, the IMAP standard suggests closing the session entirely after no more than 59 minutes and re-opening a new one anyway.

    The firewalls that I normally use (Netscreen/Juniper) have a much longer default session timeout for TCP connections, about 30 mins or so. (Though that varies on a per-protocol basis anywhere from 1 minute to 120 minutes) So I'd guess that by default I'd probably never see that issue here, though I personally don't normally have anything running on IMAP IDLE continuously anyway.

    On an inexpensive home router, it's not uncommon to have shorter timeouts because it preserves resources on a device with limited horsepower and memory, and it also helps keep them less vulnerable to certain types of DoS attacks designed to open a bunch of simultaneous connections and starve the device of resources. (On a fancier security device, there are typically certain algorithms designed to minimize the impact of such attacks in a more elegant way without just clamping-down all the session timeouts globally)

    But there are a bunch of devices out there that are a few years old that have a lot of those fancy capabilities, ie old Netscreen 5XT or 5GT devices you can often find on ebay for less than $100.

    Another option could be to try DD-WRT, I'd guess it's either got session timeout configurability or is at least smart enough to set a long timeout if you put a rule in there that specifies IMAP as the protocol/service.

    Let us know how you do. If you were close to me I'd give you a good deal on one of my old Netscreens or maybe something else from the archive pile.
    02-28-13 02:49 AM
  17. Omnitech's Avatar
    BTW re: Wireshark - yes you would be able to see traffic, but the traffic you would see would be encrypted. The reason we rely on public-key encryption for security is because it is not trivial to break into on the wire.
    02-28-13 02:53 AM
  18. Omnitech's Avatar
    Caveat: there are ways of course to log POP/SMTP/IMAP sessions directly at the application - ie client or server. If you have an IMAP client on your Linux box (Evolution? Mutt?) it should have the ability to log email sessions.
    02-28-13 02:56 AM
  19. nmonbb's Avatar
    Thats a lot of technical info! Some of it goes above me. Wanted to ask what is the ideal setting for push and sync options so as to save up on the data being consumed due to the constant connection. Another new problem for me is that i cannot seem to set up my previously deleted yahoo account back again. It gives an error msg after i enter the email and password " the server imap.yahoo.com for your account is not responding. Pls cheack your settings and try again". I didn't do any changes...so am wondering where have i gone wrong. Help, anyone?
    02-28-13 02:59 AM
  20. slickvguy's Avatar
    Omnitech, I'm in Canada.

    FWIW, I found a revised copy of the help guide on dlink's website. Here's what the Time Out section was supposed to say:

    Time Out
    The number of seconds of idle time until the router considers the session terminated.
    The initial value of Time Out depends on the type and state of the connection.
    300 seconds
    UDP connections.
    240 seconds
    Reset or closed TCP connections.
    The connection does not close instantly so that lingering packets can pass or the connection can be re-established.
    7800 seconds
    Established or closing TCP connections.
    OK, but it's not true! lol. I've never seen 240 seconds - it's always 120. And it says 7800 for established or closing TCP connections! Not true! Hmm...I'm renting this router from my ISP...it wouldn't surprise me if my ISP modified the firmware to make it 120. Would it be to their advantage? If it would cut down on network traffic, that would make sense, right? lol. I can give back this router anytime I want, but it's dual band (2.4ghz/5ghz two separate radios), so I kinda like it. I'll try others and see what th story is. I won't install dd-wrt because I don't own the router.

    what is probably happening (as stated previously) is that your firewall session timeout is just shorter than the interval between those keepalive packets. As you discovered, the IMAP standard suggests closing the session entirely after no more than 59 minutes and re-opening a new one anyway.
    I'd like to see evidence of these keep-alive packets and see how often they are sent. You really think they would be sent as often as even 240 seconds? In looking around the internet, I see that so many routers (home and office) have FAR longer timeout values. This 120 just makes no sense.

    re: wireshark: but it's on MY network (my WPA2 password), so why would that be a problem? I'm not trying to sniff someone else's communication - just my own.

    I use Thunderbird on my linux box.
    02-28-13 03:25 AM
  21. slickvguy's Avatar
    Thats a lot of technical info! Some of it goes above me. Wanted to ask what is the ideal setting for push and sync options so as to save up on the data being consumed due to the constant connection. Another new problem for me is that i cannot seem to set up my previously deleted yahoo account back again. It gives an error msg after i enter the email and password " the server imap.yahoo.com for your account is not responding. Pls cheack your settings and try again". I didn't do any changes...so am wondering where have i gone wrong. Help, anyone?
    nmonbb, please start a new thread. Your questions have nothing to do with the main topic of this thread. Let's keep this one on track.

    FYI, on my Z10, the server is rim.imap.mail.yahoo.com (not imap.yahoo.com). That might be the issue. Not sure.
    02-28-13 03:28 AM
  22. nmonbb's Avatar
    nmonbb, please start a new thread. Your questions have nothing to do with the main topic of this thread. Let's keep this one on track.

    FYI, on my Z10, the server is rim.imap.mail.yahoo.com (not imap.yahoo.com). That might be the issue. Not sure.
    Thanks for replying. It is rim.imap.yahoo.com...i didn't copy that properly...sorry. I didn't start a new thread since i needed to know about the push/sync settings. Don't mind starting a new one!
    02-28-13 03:34 AM
  23. young guy's Avatar
    I was having the same issue as the op. I have DD-WRT on my router, changed the default tcp timeout to 360 seconds. Now my hotmail and gmail are coming in on my Z10 instantly via wifi. Thanks for the tip.
    02-28-13 04:16 AM
  24. EauRouge's Avatar
    Hm so would changing the router help?

    Im currently on a modem/router combo from my ISP (Cisco DPC 3825) but I've heard of people calling in and getting the router portion disabled and only using it as a modem. Then I could connect any router to this
    02-28-13 08:25 AM
  25. EauRouge's Avatar
    Debian User - Idle TCP connections freeze


    Also, might be useful to you guys. I don't understand code very well
    02-28-13 08:54 AM
40 12

Similar Threads

  1. BIS connection via WIFI problem - SIM Unlock HELP!!
    By brake911 in forum General Carrier Discussion
    Replies: 9
    Last Post: 12-06-09, 12:56 PM
  2. Junk Mail pushed through IMAP
    By nextgates in forum General BlackBerry News, Discussion & Rumors
    Replies: 7
    Last Post: 05-14-09, 12:38 PM
  3. Problem connecting with WiFi since upgrading firmware
    By listentothis in forum BlackBerry Curve Series
    Replies: 3
    Last Post: 01-20-09, 10:34 PM
  4. Problems connecting to WiFi
    By dianap in forum BlackBerry Curve Series
    Replies: 3
    Last Post: 06-09-08, 03:00 PM
LINK TO POST COPIED TO CLIPBOARD