Best in class:Save 49% on 15 months of ExpressVPN
-
Thanks for the reply.
Well, I just did deeper testing of the servers. I used telnet and openssl to connect to and log into the videotron, yahoo, and gmail servers directly. I also discovered that the login banner of capabilities may be wrong. To wit, this is the login message from imap.videotron.ca:
It does not list IDLE as a capability. I was prepared to conclude that videotron does not support IDLE, except I knew that when I used the LTE network (not wifi), e-mails sent to that server definitely were immediately pushed to the Z10. So I issued a few commands and then the IDLE command - and it accepted the IDLE! Unless I'm missing something, it seems that the videotron server does indeed support imap idle (push). Good to know.Code:* OK [CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS CHILDREN BINARY UNSELECT SORT LANGUAGE XSENDER X-NETSCAPE XSERVERINFO AUTH=PLAIN] Messaging Multiplexor (Sun Java(tm) System Messaging Server 6.3-5.02 (built Oct 12 2007))
I then logged into the rim.imap.mail.yahoo.com server, which is what the Z10 entered in the server field for my Yahoo e-mail account. The banner does not show IDLE and it does not respond to the command. Therefore the Z10/Yahoo e-mail combination is stuck with polling only. That explains why even when I was connected via LTE it still was the laggard of the 4 servers I tested.
Next I tested the imap.gmail.com server with openssl. As we already knew, I confirmed that it definitely supports imap idle.
I did not test the microsoft live.com server because it uses Exchange ActiveSync and therefore one would think it would work the best with the Z10 - but I still get long delays with e-mails sent to that address if I'm connected via wifi. Connected via LTE, the push definitely worked.
DannySmurf, you spoke of network latency and long routing time, which makes sense. I have a DLINK DIR825 router, very little wireless (or other) traffic (two clients - an itouch and a laptop), and the router is located about 2 feet from the Z10. Could it be as simple as a setting in the router? I googled that subject but found nothing. I agree that it is likely something to do with either my system here (cablemodem, router, ?) or with Videotron (ISP). Since I only got the phone on Saturday night, and I've never had a smartphone, Blackberry, etc., it's taken me a few days to learn enough about the O/S, push, polling, imap, activesync, etc., to get a handle on what's what. In the coming days I will try to test the Z10 while connected to other routers/wifi networks using just gmail and videotron e-mail servers. Oh, and thanks for the laugh about calling videotron's CS for help.
Last edited by slickvguy; 02-27-13 at 04:15 AM.
02-27-13 02:54 AM - Thank you Omnitech. I now understand. It makes sense and I believe you are correct because I tried two different things, both resulted in the push working over wifi immediately. The first, was turning airplane mode on and off (after reading on other threads that that had worked for some people). I assume that caused the same/similar reset that you described? The second, was exactly what you had advised: do a manual refresh. Now I need to find out what is causing this to happen and if there is anything I can do about it. I am renting the router, so I can easily change it if I want to. I'm using Arch linux on my main PC - could that have something to do with it?
The wifi push issue is not that critical while I'm at home (I work from home). I can view incoming e-mails in my Thunderbird client and the web clients. I was just using my wifi here to test the e-mail push and was dismayed at the long delays. Now that I know that the LTE works fine, I'm relieved, because that's far more important to me (when I'm on the road).
When you said "network devices are dropping the IMAP IDLE session before a keep-alive is sent", were you referring to just the router or the other wireless devices? My daughter has a touch and a laptop - those are the only wireless devices on the network besides the Z10. What I am asking is if it is strictly a router issue - or could it be one of those devices causing the problem?
I find it very odd that the Videotron server's capability string does not include "IDLE", yet when you issue the IDLE command it accepts it ("+ idling").
Another question: the other sync functions (e.g. when you delete an e-mail) - are they supposed to occur at the same time as the push e-mail? Are those things also a function of IMAP IDLE orr is that only done at the sync intervals and a separate process? Because I notice that those kinds of things happen later (sometimes just a short while after, other times longer delay).
p.s. While connecting to the yahoo server with openssl I saw " XAPPLEPUSHSERVICE" in the capability message. Would be nice if they had a XBBPUSHSERVICE
02-27-13 05:28 AM -
- Well your firewall is in your router, basically.
I'm not sure what those "endpoint filtering" settings are, I'd probably have to look at the UI or docs. DD-WRT is definitely highly configurable but it can also be a morass, and a pain to update because after all this time they still don't seem to have figured out how to update the firmware without blowing off the entire configuration and not having any reliable way to re-load it - so you end up having to re-configure everything from scratch again.
I'm not as familiar with the mechanism EAS uses for push - I believe the Microsoft terminology is "Direct Push" but for example I don't know if it runs on a separate set of ports or what.
What is the text after the semicolon characters in the session-list? Doesn't look like a hostname, doesn't look like a domain..
Are the numbers preceding the semicolon chars the session lifetime, do they count-down?
Port 993 is IMAP over TLS(SSL).
Port 143 is straight IMAP.
Port 443 is SSL, could be various things.
Well 206.53.159.203 definitely belongs to Blackberry so at least we have some idea what that is. Could be a million things. Would be good to hear from the horse's mouth what sort of "phoning home" Blackberry 10 does, though.
What figure does the timeout start out at? Looks like something around 120 seconds?
I don't think the issue is with the Z10 - if that were the case it wouldn't work over 3G/4G either.
99.9% likelihood the firewall is just timing out the sessions early, as I mentioned previously. Very common issue. If those session timeouts are only 120 seconds, that's pretty short, and that's probably why. IMAP IDLE needs to maintain a persistent connection typically anywhere from 15 to 59 minutes long.
IMAP IDLE implementations will typically send a "keepalive" packet something like every 15-30 minutes. If the firewall's session watchdog closes the session before the keepalive packet is sent, there will be no state-change notifications until the client opens a new session because the server has no way to initiate a new connection from the outside. (Most firewalls deny incoming traffic by default, and in your case you're behind a NAT so that wouldn't work anyway since they can't route over the internet to a private IP address-space anyway - connection needs to be initiated from behind the firewall/NAT.)
So what you need to look for is some setting that would allow you to adjust the session timeout values. I downloaded the manual (see below) and saw NO references in that document to either a configurable session-timeout value OR the "TCP endpoint filtering" settings you mentioned above.
http://www.dlink.com/us/en/support/p...11301130EN.pdf02-27-13 11:18 PM - The text after the semi-colon is just my comment (some of which I copied from the result of an ip lookup when I was checking to see which ip went with which server). I should have pasted the header from the internet sessions table. It's "Local / NAT / Internet Protocol / State / Dir / Time Out". The timeout is in seconds.What is the text after the semicolon characters in the session-list? Doesn't look like a hostname, doesn't look like a domain..
Are the numbers preceding the semicolon chars the session lifetime, do they count-down?
From the router's online help:
Code:Time Out The number of seconds of idle time until the router considers the session terminated. The initial value of Time Out depends on the type and state of the connection. 300 seconds - UDP connections. 240 seconds - Reset or closed TCP connections. The connection does not close instantly so that lingering packets can pass or the connection can be re-established. 7800 seconds - Reset or closed TCP connections. The connection does not close instantly so that lingering packets can pass or the connection can be re-established.
You'd assume from the above that it would start at either 240 or 7800. But I've never seen a timeout number even close to 240. Your 120 guess is closer to what I normally see. But what happens is that they frequently won't count down to 0, instead they will get bumped back up, but NOT to 240 or 120. I'll do a screen refresh and it'll jump from let's say 45 to 85. Weird. Also, I sometimes see more than one entry for the same ip address and port combination (different NAT number) and different timeouts. Multiple connections, same IP address and port.What figure does the timeout start out at? Looks like something around 120 seconds?
I'm playing around with it right now as we speak. Logging into servers. Issuing the IDLE command, doing various things w/ the e-mails/sync/etc, and watching the session tables and how the servers respond. When I connect to the servers from my PC (plugged into the router port), one difference is that you see the session immediately and it persists. You don't see a higher timeout than with the Z10/wifi, but it the session keeps refreshing or extending itself, if you know what I mean. It stays alive. When I issue the IDLE command and send a new e-mail to that server, the client shows the updated EXISTS, RECENT, etc., info being sent from the server.
Another thing (in case anyone using videotron imap stumbles onto this thread). RFC2177 says the following:
Although Videotron does not state that it does IDLE in the string it sends to the client upon logging in, I know it does do IDLE. I was concerned that the Z10 would not attempt to use push with it. But when I sent it the CAPABILITY command, lo and behold the very last item is "IDLE". Brilliant.If the server does not advertise the IDLE capability, the client MUST NOT use the IDLE command and must poll for mailbox updates.
This part from RFC2177 was interesting too:
Bottom line: assuming that the timeout is the problem...I see no way around it. And are we really so sure that the ROUTER is the problem? I'll see if I can borrow someone else's wifi router for testing purposes. Yes, I agree, that it's a timeout issue. But there are a few participants in this dance. My ISP and/or the Z10 s/w could be the culprit. Maybe the client (Z10) is not terminating the IDLE and re-issuing it and then the SERVER is disconnecting? Anyway, I searched all day long for solutions, but didn't come up with anything. If it was for my PC as the client, then I'd be able to change /procs/sys/net/ipv4/tcp_keepalive_time and I think that would solve it. The router's online help says that it does 240 or 7800 and that the initial value depends on the type and state of the connection. I see no way to change this in the router and I see no way to change it in the Z10 OS (unless you've managed to root it? j/k)The server MAY consider a client inactive if it has an IDLE command
running, and if such a server has an inactivity timeout it MAY log
the client off implicitly at the end of its timeout period. Because
of that, clients using IDLE are advised to terminate the IDLE and
re-issue it at least every 29 minutes to avoid being logged off.
This still allows a client to receive immediate mailbox updates even
though it need only "poll" at half hour intervals.
Do you know how I would be able to monitor the (decrypted) communication between the Z10 and the router? . I use Arch linux. Something on BackTrack perhaps?02-28-13 12:22 AM - I can't figure out what the distinction is between those last 2 lines in the help file list other than the timeout value - did you mis-copy something there?
Is there no control for adjusting the timeout value? If not, I wonder if you put a specific rule in there that specifies IMAP as the protocol if the device might adjust that. (Though honestly I think we may be expecting far grander things from an inexpensive home router than we might have a right to expect.)
Where are you monitoring that, via a telnet session or something?
Perhaps the reason the RFC has the rule it does is to avoid scenarios where a server implementation for some reason cannot disable IMAP IDLE, but the system adminstrator may wish to prohibit it to manage network/server resource usage.
I'd give it a very very high likelihood of that being the case.
It's easy enough to verify - but you have to be careful about security since you will no longer be firewalled or NAT'd. You can either:
- Disable the stateful firewall in your D-Link temporarily (may not be feasible if you can't also disable NAT)
- Find a way to connect your Z10 directly to the cablemodem. (ie plug a plain WiFi access-point into the cablemodem and link the Z10 to that)
I really doubt it has to do w/ the ISP or the Z10. If it works over 3G/4G, that pretty much eliminates the Z10. And I already know that lots of people have IMAP IDLE working with the Z10, so once again I wouldn't waste any energy pursuing that angle.
Changing the TCP keepalive time on your PC isn't going to help if the firewall is killing the session at the border. According to the following document (not exactly authoritative, just easy to lookup) the minimum TCP keepalive value should already be 2 hours anyway:
Keepalive - Wikipedia, the free encyclopedia
The multiple connections to the same port are normal, many or most IP applications commonly open multiple connections to the same host to do what they need to do, including web browsers. There may be a separate socket/thread doing an IMAP watchdog/keepalive, a different one in the process of doing a sync of some kind, a different one sending a message, etc.
Backtrack is a liveCD, you mean one of the tools they have on there?
Now you're getting into a bit dicey hacking territory, trying to break into public-key encrypted communications. Suffice to say there are ways but you sorta have to be a bad guy to do them with a clear conscience, and have access to extraordinary resources, and I'm not feeling in a particularly educational mood on that subject at the moment.
Anyway, the SSL link is actually not between the router and the Z10, it's between the server and the Z10 via the router. The easiest way to troubleshoot that would be to just turn off encryption for IMAP on that connection. If you can find an IMAP server to test with that will do an unencrypted IMAP session, that would be ideal. Though I'm not sure they're very easy to find, as that would mean auth credentials being sent in the clear etc.02-28-13 01:45 AM - OK.
Well I'm not sure what you mean specifically by "crapped out", I assume you mean the server didn't send any response back to the client since the TCP session was closed and there was no link to send that communication back on. As I predicted would happen.
First of all, I can't buy a Z10 here (USA) yet so I am in no position to test this personally. Can I ask what part of the world you're in BTW?
Re: the firewall timeouts in general: IMAP IDLE is relatively unusual in how it expects to maintain a very "long-lived" but mostly idle connection like that. In a non-mobile environment, it's not a big deal to keep sending periodic packets to keep connections open, but that's often not feasible on a mobile device because even to send one tiny IP packet you have to wakeup the radio and use significant battery energy.
The only other situation where I have personally run into this much is when I want to maintain for some reason a very long-lived SSH connection to something, ie for days on end. In that case I set an option in OpenSSH to send periodic keepalive packets, which have the effect of keeping the firewall session open so the SSH session doesn't drop. The general issue is the same as the IMAP IDLE issue here.
And it's not technically true that the client doesn't send "a darn thing" to keep the session alive - what is probably happening (as stated previously) is that your firewall session timeout is just shorter than the interval between those keepalive packets. As you discovered, the IMAP standard suggests closing the session entirely after no more than 59 minutes and re-opening a new one anyway.
The firewalls that I normally use (Netscreen/Juniper) have a much longer default session timeout for TCP connections, about 30 mins or so. (Though that varies on a per-protocol basis anywhere from 1 minute to 120 minutes) So I'd guess that by default I'd probably never see that issue here, though I personally don't normally have anything running on IMAP IDLE continuously anyway.
On an inexpensive home router, it's not uncommon to have shorter timeouts because it preserves resources on a device with limited horsepower and memory, and it also helps keep them less vulnerable to certain types of DoS attacks designed to open a bunch of simultaneous connections and starve the device of resources. (On a fancier security device, there are typically certain algorithms designed to minimize the impact of such attacks in a more elegant way without just clamping-down all the session timeouts globally)
But there are a bunch of devices out there that are a few years old that have a lot of those fancy capabilities, ie old Netscreen 5XT or 5GT devices you can often find on ebay for less than $100.
Another option could be to try DD-WRT, I'd guess it's either got session timeout configurability or is at least smart enough to set a long timeout if you put a rule in there that specifies IMAP as the protocol/service.
Let us know how you do. If you were close to me I'd give you a good deal on one of my old Netscreens or maybe something else from the archive pile.02-28-13 02:49 AM - Thats a lot of technical info! Some of it goes above me. Wanted to ask what is the ideal setting for push and sync options so as to save up on the data being consumed due to the constant connection. Another new problem for me is that i cannot seem to set up my previously deleted yahoo account back again. It gives an error msg after i enter the email and password " the server imap.yahoo.com for your account is not responding. Pls cheack your settings and try again". I didn't do any changes...so am wondering where have i gone wrong. Help, anyone?02-28-13 02:59 AM
- nmonbb, please start a new thread. Your questions have nothing to do with the main topic of this thread. Let's keep this one on track.Thats a lot of technical info! Some of it goes above me. Wanted to ask what is the ideal setting for push and sync options so as to save up on the data being consumed due to the constant connection. Another new problem for me is that i cannot seem to set up my previously deleted yahoo account back again. It gives an error msg after i enter the email and password " the server imap.yahoo.com for your account is not responding. Pls cheack your settings and try again". I didn't do any changes...so am wondering where have i gone wrong. Help, anyone?
FYI, on my Z10, the server is rim.imap.mail.yahoo.com (not imap.yahoo.com). That might be the issue. Not sure.02-28-13 03:28 AM
E-mail Push (EAS / IMAP IDLE) problems connected via WIFI
Similar Threads
-
BIS connection via WIFI problem - SIM Unlock HELP!!
By brake911 in forum General Carrier DiscussionReplies: 9Last Post: 12-06-09, 12:56 PM -
Junk Mail pushed through IMAP
By nextgates in forum General BlackBerry News, Discussion & RumorsReplies: 7Last Post: 05-14-09, 12:38 PM -
Problem connecting with WiFi since upgrading firmware
By listentothis in forum BlackBerry Curve SeriesReplies: 3Last Post: 01-20-09, 10:34 PM -
Problems connecting to WiFi
By dianap in forum BlackBerry Curve SeriesReplies: 3Last Post: 06-09-08, 03:00 PM