[Omnitech]

I missed the following earlier:

No packet buffers are required to keep a TCP connection alive. Not at the end points and not at NAT devices in between. Only the address/port association data. Which is finite and limited but not to the extent that packet buffers are.

The only device that does not use buffers for forwarding packets is an old-fashioned ethernet hub or "cut through" L2 switch, which always runs the risk of forwarding corrupt packets because it does not buffer the packet before it forwards it and thus cannot determine its integrity.

A stateful firewall uses buffers because it needs to understand what is in the packet before it decides what to do with it, and higher-layer protocols cannot be understood simply by reading packet headers.


Back to current:

And I don't agree with your assertion that Limited Connectivity is related to not setting the default route.

That was not the assertion I made. In fact, you describe it yourself, it pertains to the workaround that people discovered:

For most people the simple work around of turning wireless data off allowed the use of Wi-Fi to access all the remaining Internet.

People discovered that in many cases they could make WiFi work again when this "Limited Connectivity" condition arose by turning off cellular data, or simply toggling it off/on. This suggests to me that the OS likely was not properly updating the default route when the link state changed.

[muellerto]

But back to your problem. We know that turning data services off and back on is a cause. What happens when you loose data service due to lack of signal? Is there somewhere near you that you can take your phone where you have no cell coverage at all and tell us what happens then?

It never happened until now. Or I didn't notice a lack of signal. To find a place without network I would probably have to use a Faraday cage. (Once I had a safe ...) But good question. I try to find this out.

I've noticed no one has brought up the various cellular networks aspect (CDMA versus GSM etc)

I still use GSM (EDGE), small footprint ..., and because I never stream videos or other multimedia crap this does the job without any lack of comfort. But should this really have an impact? Could try 3G or LTE as well.

Yes, but he did say he leaves the phone part of the device on and connected to the network. The device isn't in complete airplane mode.

Yes, not completely. That's why I never mentioned Airplane mode.

I wonder if it's just email or the rest of the data (like BBM and/or the browser) which isn't working for the length of time he's trying to re-establish email settings ...

Just e-mail. Browser works without any problems. That's why I say, it is not a connection problem. The connection is well after being opened manually, the ideal case. But in this moment the Hub (the PIM sevices) should start up a loop for every "Push configured" e-mail account telling the IMAP server there's a client and then sending keep-alives every 15 minutes. I guess, the IMAP server never gets any information when my device is reconnected to the data network. This could also have it's reason in the mail provider and the server, perhaps it is a very specific problem with this constellation.

[belfastdispatcher]

It never happened until now. Or I didn't notice a lack of signal. To find a place without network I would probably have to use a Faraday cage. (Once I had a safe ...) But good question. I try to find this out.
I still use GSM (EDGE), small footprint ..., and because I never stream videos or other multimedia crap this does the job without any lack of comfort. But should this really have an impact? Could try 3G or LTE as well.
Yes, not completely. That's why I never mentioned Airplane mode.
Just e-mail. Browser works without any problems. That's why I say, it is not a connection problem. The connection is well after being opened manually, the ideal case. But in this moment the Hub (the PIM sevices) should start up a loop for every "Push configured" e-mail account telling the IMAP server there's a client and then sending keep-alives every 15 minutes. I guess, the IMAP server never gets any information when my device is reconnected to the data network. This could also have it's reason in the mail provider and the server, perhaps it is a very specific problem with this constellation.

You know, it might be because you're using EDGE, I also use IMAP IDLE and I find it almost unusable on both BB10 and iOS. Sometimes I get the email notification but it takes up to 5 minutes to actually open the email on 2G.

Also don't forget any call coming through will interrupt your data connection.




#believeinfilm

[Omnitech]

You know, it might be because you're using EDGE, I also use IMAP IDLE and I find it almost unusable on both BB10 and iOS. Sometimes I get the email notification but it takes up to 5 minutes to actually open the email on 2G.

Also don't forget any call coming through will interrupt your data connection.

Excellent points.

EDGE also has very high latency, around 600ms, which is worse than a dialup modem. That can negatively impact all sorts of modern internet content, and make trying to browse a complex website very frustrating, for example.

[muellerto]

You know, it might be because you're using EDGE, I also use IMAP IDLE and I find it almost unusable on both BB10 and iOS. Sometimes I get the email notification but it takes up to 5 minutes to actually open the email on 2G.

Never had this issue. If I got a notification I also had the mail body a few seconds later. But I have switched off downloading pictures.

Also don't forget any call coming through will interrupt your data connection.

Sigh.

OK, I try 3G now. Let's see ... Oh, 3G seems to work! The idle thing starts up again without any delay. All is as expected.

Now 4G (LTE) ... argh, this is not easy - the option is named "4G and 3G" and if you open a connection manually it comes up as 3G and then after 10 or 15 seconds it gets 4G. But this short period of 3G is already enough to get the notification ...

So it seems indeed substantially influenced by the type of the connection, I never thought that this could be the reason. That's why I never tried to change this ... So I let it on 3G.

Thanks to SmellWhole for bringing this aspect into the discussion. Thanks to all the others for the patience over the week.

(Well, but this all should indeed be written into a manual, "How to setup push email", by an educated person. We saw how many variables this topic has and how many ideas we discussed.)

[Richard Buckley]

Belfastdisptcher is right. Your issue may be more related to your use of EDGE.

Also, the capacity of a GSM cell is very limited when compared to a 3G or 4G cell. The philosophy of carriers when EDGE rolled out was that data was a loss leader. They had to offer data to get customers, but had to provide reliable voice to make money. Those cells are probably still running software written to that philosophy. If other people are making calls they would get priority.

Posted via CB10

[SmellWhole]

... People discovered that in many cases they could make WiFi work again when this "Limited Connectivity" condition arose by turning off cellular data, or simply toggling it off/on. This suggests to me that the OS likely was not properly updating the default route when the link state changed ...

I've seen the Limited Wifi Connection notification show up on BB10. Often when I've seen it, I've noticed that the phone has switched to 4G/LTE. In these cases I've done nothing, just left it as-is, and found that the phone has reconnected to wifi by itself a short while later. I'll make the following speculation about Limited Wifi Connectivity:

When the mobile network and wifi are both turned on, the phone chooses the stronger signal. Of course, if strength is equal, the phone would choose wifi before the mobile network. But let's say you're on wifi, 4GLTE is available at three bars where you're located, wifi is at three bars and drops for some reason to two bars. Perhaps then the phone would switch to the 4G/LTE signal which is stronger (at three bars) and display the Limited Wifi connectivity notification until wifi strength increases back to three or more bars. At least that's what has seemed to have happened whenever I've observed the Limited Connectivity notification.

... Just e-mail. Browser works without any problems. That's why I say, it is not a connection problem. The connection is well after being opened manually, the ideal case. But in this moment the Hub (the PIM sevices) should start up a loop for every "Push configured" e-mail account telling the IMAP server there's a client and then sending keep-alives every 15 minutes. I guess, the IMAP server never gets any information when my device is reconnected to the data network. This could also have it's reason in the mail provider and the server, perhaps it is a very specific problem with this constellation.

Though BB10 is not on BIS, the handsets still interact to some extent with the n.o.c. On BIS, some things (email, BBM, and Browser) would work seamlessly when moving back and forth between wifi while other things would require a sign out/in or connection reset. For example, on BIS when I was logged into yahoo instant messenger on 3G data and came home to wifi, I would continue to receive messages, but recipients would no longer receive my messages. This drove me crazy until I realized that only signing out and signing back in or turning all connections off and then back on (which essentially accomplished the same thing) would reset things properly (whether at the yahoo servers or BlackBerry n.o.c. idk) so that my sent yahoo instant messages would arrive at their destinations. I'm just pointing this out to show how it's possible for some parts of data to work while others don't when it comes to the various networks and connections.

[Richard Buckley]

The only device that does not use buffers for forwarding packets is an old-fashioned ethernet hub or "cut through" L2 switch, which always runs the risk of forwarding corrupt packets because it does not buffer the packet before it forwards it and thus cannot determine its integrity.

A stateful firewall uses buffers because it needs to understand what is in the packet before it decides what to do with it, and higher-layer protocols cannot be understood simply by reading packet headers.

Yes, it needs the buffer space to process the packet, but not to keep the session open, which is what I said.

So when a packet arrives there must be buffer space for it to be held long enough for routing, firewall and other processes to do what they need to. It is then sent on its way, rejected or dropped and that buffer space is available for the next packet. If there is no buffer space available that would be because the rate of packet arrival has over-run either the routers ability to process the packets fast enough. Or, more likely, the bandwidth of the of the channel the packets are being forwarded. If there is no buffer space the packet will be dropped. Dropping a packet is an important signal in TCP/IP networking. It means either there is no endpoint listening to connections at the address and port specified, or that the network was unable to deliver it for some reason.

A widely accepted rule of thumb is that you need 50ms of output queue time buffering. Any more than that can lead to problems (see discussions on buffer bloat). Normally this is only an issue on networks of less than 100 Mbbps where it is cheap enough to provide enough memory to be too much. This is especially true in SOHO or home network setting where buffer size can be a selling point. The LAN would be at least 100 Mbps, but more likely 1 Gbps but the upstream WAN connection to the internet is probably less than 1 Mbps.

When no packets are arriving for a session, such as when a device is holding a session open in IMAP/IDLE and most of the time there are no packets in transit, the router needs no packet buffer space to keep the session alive. It only needs to store the associative state. Less than 100 bytes in most devices. If packets for a session regularly arrive when there is no buffer space for them resulting in them being dropped, it means the network is overloaded at that point and packets are probably being dropped from the queue because the output network can't take them anyway.

[Omnitech]

I've seen the Limited Wifi Connection notification show up on BB10. Often when I've seen it, I've noticed that the phone has switched to 4G/LTE. In these cases I've done nothing, just left it as-is, and found that the phone has reconnected to wifi by itself a short while later. I'll make the following speculation about Limited Wifi Connectivity:

When the mobile network and wifi are both turned on, the phone chooses the stronger signal. Of course, if strength is equal, the phone would choose wifi before the mobile network. But let's say you're on wifi, 4GLTE is available at three bars where you're located, wifi is at three bars and drops for some reason to two bars. Perhaps then the phone would switch to the 4G/LTE signal which is stronger (at three bars) and display the Limited Wifi connectivity notification until wifi strength increases back to three or more bars. At least that's what has seemed to have happened whenever I've observed the Limited Connectivity notification.

The "Limited Connectivity" warning and the mechanism that is supposed to trigger it perform a useful function.

What the discussion has been about is when this "error" is triggered when in fact there is no problem with the link. (ie various other devices can use the same WiFi connection just fine)

In a nutshell, the OS is using an unreliable tactic to detect link state, and if it falsely determines that the link is "bad", it then sometimes causes connectivity problems because it will drop the WiFi link and return to carrier data when in fact the WiFi link is functional. (Which can cost users money because carrier data is typically billed based on traffic usage)

[Omnitech]

Yes, it needs the buffer space to process the packet, but not to keep the session open, which is what I said.

So when a packet arrives there must be buffer space for it to be held long enough for routing, firewall and other processes to do what they need to. It is then sent on its way, rejected or dropped and that buffer space is available for the next packet. If there is no buffer space available that would be because the rate of packet arrival has over-run either the routers ability to process the packets fast enough.

Or the device simply didn't allocate any buffers to receive any new packets, causing it to drop packets when it's not even congested.

In other words, broken.

[Richard Buckley]

Or the device simply didn't allocate any buffers to receive any new packets, causing it to drop packets when it's not even congested.

In other words, broken.

From Top Five TCP/IP Packet Loss Myths:

Myth � Loss does not occur in properly-operating networks.

Reality � TCP queue overflow may cause loss, even during normal operation, as part of TCP congestion control. See this report for more information. Loss rates of several percent are common with high bandwidth.

Myth � All packet loss is bad.

Reality � Packet loss plays at least two important, beneficial roles:

• Implicit signaling of congestion: TCP relies on packet loss as an implicit signal of network congestion. To fairly share available bandwidth, each TCP stream passing through a congestion point must dynamically discover other TCP streams sharing that same congestion point. For more on the impact this can have on latency, see the TCP Sender-Side Latency section of the Topics in High Performance Messaging whitepaper.
• Efficient discarding of stale, latency-sensitive data: for more, see the Optimal UDP Buffer Sizing section of the Topics in High Performance Messaging whitepaper.

[Omnitech]

From Top Five TCP/IP Packet Loss Myths:

Myth – Loss does not occur in properly-operating networks.

Reality – TCP queue overflow may cause loss, even during normal operation, as part of TCP congestion control. See this report for more information. Loss rates of several percent are common with high bandwidth.

Myth – All packet loss is bad.

Reality – Packet loss plays at least two important, beneficial roles:

• Implicit signaling of congestion: TCP relies on packet loss as an implicit signal of network congestion. To fairly share available bandwidth, each TCP stream passing through a congestion point must dynamically discover other TCP streams sharing that same congestion point. For more on the impact this can have on latency, see the TCP Sender-Side Latency section of the Topics in High Performance Messaging whitepaper.
• Efficient discarding of stale, latency-sensitive data: for more, see the Optimal UDP Buffer Sizing section of the Topics in High Performance Messaging whitepaper.

A) When there is no congestion and no contention for traffic, and a device drops packets on a regular basis, it's broken. No matter how useful congestion control is when there is, you know, actual congestion.

B) I never even remotely asserted that "all packet loss is bad".

C)

[Richard Buckley]

A) When there is no congestion and no contention for traffic, and a device drops packets on a regular basis, it's broken. No matter how useful congestion control is when there is, you know, actual congestion.

The packet dropping I have been discussing has all been about dealing with congestion. When there is no congestion, or as you put it when the traffic is idle, there is no need for large buffers either. Even if the memory is available to allocate them. If a router does allocate buffers that are too large and does not mitigate against the adverse affects then the impact on throughput and latency can be significantly detrimental.

When it comes right down to it on one hand we have people like Vint Cerf, Van Jacobson, Nick Weaver, and Jim Gettys, papers published in peer review journals, industry actions such as the DOCSIS upstream buffer control, and so forth. On the other hand we have:

C)

[Omnitech]

The packet dropping I have been discussing has all been about dealing with congestion.

Which as I have been saying all along, is irrelevant, because what I have been discussing all along is packet loss when there is no congestion.

When there is no congestion, or as you put it when the traffic is idle, there is no need for large buffers either.

You made up this "large buffers" red herring. "Not needing large buffers" ≠ "Zero buffers".

[Richard Buckley]

Which as I have been saying all along, is irrelevant, because what I have been discussing all along is packet loss when there is no congestion.




You made up this "large buffers" red herring. "Not needing large buffers" ≠ "Zero buffers".

Yes, specifically you suggested that packet loss could result in the application layer falling back from providing IMAP/IDLE service to providing polling service. Perhaps I should have pointed out that TCP/IP provides a reliable in order delivery. When reliable in order delivery is not possible the application layer is notified and the connection closed. I assumed you knew that, and instead pointed out that TCP/IP is robust in the face of packet loss (short of sufficient loss to render the connection unusable), not just because the designers knew that some packet loss was inevitable, but because they used loss of, and indeed discarding of, packets as a signal in the protocol. At which point you scoffed. And that's fine, scoff if you will. But, if you can contrive a scenario where packet loss will interfere with the application layer protocol enough to cause a fall back from IMAP/IDLE to polling, but not enough cause the application layer to be notified of data loss, or the connection to drop. And in addition to this leave a connection capable of supporting polled IMAP mailbox access but not capable of re-establishing IMAP/IDLE, I would really like to see a description of it. A man-in-the-middle attack could do this. Are you suggesting the OP is loosing push email because of a MITM attack?

[Omnitech]

Yes, specifically you suggested that packet loss could result in the application layer falling back from providing IMAP/IDLE service to providing polling service.

Can I ask whose posts you are reading? Because it doesn't sound like they are mine. You may want to re-read some things.

All of this is barely tangential to the topic anyway.

[Richard Buckley]

Can I ask whose posts you are reading? Because it doesn't sound like they are mine. You may want to re-read some things.

All of this is barely tangential to the topic anyway.

Let's start with this one, since it is the start.

I should write a FAQ on this, I've answered it enough times already..

IMAP DOES require polling, even with "IDLE" active, because IDLE only notifies of new messages. The regular polling cycle is responsible for reconciling other changes, such as messages moves and deletions, to the various endpoints.

IDLE is also dependent on network conditions. In order for this to work, it has to establish a long-term open TCP socket so it can receive immediate notification of any changes occurring on the server side. (Normally with the usual stateful firewalls and NAT devices in the network path, a server cannot easily initiate an inbound connection to a host on a private network. So the host on the private network (your BlackBerry in this case) initiates an outgoing connection to the server, and keeps that connection "alive" for a long time period. (Typically this lasts around 30 minutes or so, whereupon the remote device lets the connection close and then re-establishes another one)

Now here's the rub: the thing about establishing a long-term open TCP connection is that certain network elements are unfriendly to this. Common example: inexpensive home network equipment. There are 2 challenges to overcome when supporting this sort of traffic:

1) It's more costly to build routers and firewalls with enough buffer memory to maintain long-term open sessions without starving other packet streams for buffer memory. This is why low-end home network equipment oftentimes will not allow a 30-minute open TCP session - I've seen some that want to close them in 60 seconds.

2) Allowing long-term open sessions increases the difficulty of protecting from network attacks, and requires more sophisticated measures in the equipment.

So if you're stuck with something in your network path that is "unfriendly" to these long sessions that IMAP and other "push" internet protocols require, it will prevent them from "pushing" because it will keep closing the link. In such cases, email notifications will revert to the "polling interval".

Lastly - IMAP is rather "dumb" about this. If something won't allow it to keep a session open for its preferred time (typically 30 min), it will just stop pushing entirely. Microsoft's Exchange ActiveSync (EAS) protocol, on the other hand, has a mechanism where it will start trying to reduce the session time progressively until it finds something that works with the network path in use. (Until it gets down to a few minutes, whereupon it reverts to polling like IMAP.)

So in a nutshell, if you're having issues like that, it's more likely it will work if you use EAS than if you use IMAP.

From what you describe it sounds like there may be something in your case that is not re-establishing a long-term open session after a nighttime break, but I haven't seen many complaints about that kind of issue with BB10 so far.

I have never experience IMAP being dumb in the way you suggest. Every time I leave Wi-Fi coverage the session terminates un-gracefully as I'm sure you know. The next time a session is established all my IMAP/IDLE capable servers start pushing again. It has been a long time since I've seen a session terminated in less than the length of time the client wanted it open except for mobile wireless and Wi-Fi examples. My IMAP/IDLE session from my desktop stay open from when I login in the morning till when I log out at night. Over 8 hours. There may be some old devices around but it would be easy to find out if that was the cause of the problem.

Why do long-term open sessions increases the difficulty of protecting from network attacks, and require more sophisticated measures in the equipment?

Devices need buffer storage for handling packets. They don't need storage for handling packets on a per session basis. You're saying a device could reach its maximum simultaneous session limit because all packet buffer storage has been allocated to previously open sessions. But many, or even most of those sessions could be IMAP/IDLE or other long lived but low through put sessions that won't be using their buffers for seconds or even minutes. A device that operates like that would be broken.

It is not tangential because all of this distracts from the very real and reasonable mail server management issues which very easily explain why the OP isn't getting push service in the morning.

[Omnitech]

My IMAP/IDLE session from my desktop stay open from when I login in the morning till when I log out at night. Over 8 hours. There may be some old devices around but it would be easy to find out if that was the cause of the problem.

It's a well-known issue, and I'm losing my patience with this endless tail chasing. (As I assume anyone still left reading here is)

Why do long-term open sessions increases the difficulty of protecting from network attacks, and require more sophisticated measures in the equipment?

Because one of the oldest tricks in the book for doing a denial of service attack is doing what is called "SYN Flood". There are variants of this. Basically the intention is to exhaust all available packet buffers by initiating a massive number of sessions that you have no intention of using until you exhaust all available buffers on the device such that it can no longer pass any legitimate traffic.

If you do not have a mechanism to time-out unused sessions, you massively increase your device's vulnerability to this sort of attack.

[Richard Buckley]

It's a well-known issue,

There are some configurations that exhibit that behavior but they seem to be mainly linux systems using versions of OpenSSL that defeat SO-KEEPALIVE. Well know perhaps but not applicable to BB10.

and I'm losing my patience with this endless tail chasing. (As I assume anyone still left reading here is)

And I lost my patience with the snide replys some time ago.

Because one of the oldest tricks in the book for doing a denial of service attack is doing what is called "SYN Flood". There are variants of this. Basically the intention is to exhaust all available packet buffers by initiating a massive number of sessions that you have no intention of using until you exhaust all available buffers on the device such that it can no longer pass any legitimate traffic.

If you do not have a mechanism to time-out unused sessions, you massively increase your device's vulnerability to this sort of attack.

Yes I'm familiar with DOS using SYN Flooding. I don't know of anyone else who would seriously consider aggressively closing valid active, but idle, connections to defend against too many partially open connections. You, and anyone else, may do what ever you want to protect against what ever malicious actions you consider a threat. There are better strategies.

Certainly there are network nodes around that do agressively discard connection state information breaking valid connections.

A more significant situation, at least in my experience, is server managers that become concerned with the volume of connections to their servers. Some disable IMAP/IDLE and reduce the idle timeout on their servers either at the application or system level. Some allow IMAP/IDLE until some metric they devise is surpassed, then they disable IDLE untl the metric falls below the threshold. This is a natural outcome of the proliferation of mobile devices. At one time a family may have had one desktop computer and a small number of email accounts. Now it is common for one person to have a computer at home, at work, a laptop, a smartphone (or several) and a tablet (or several) all connected to the same email accounts. Email providers who were not prepared for this explosion, especially the smaller ones, are in reaction mode. Even Google stopped offering EAS on free accounts. Google is able to monetize email without necessarily displaying advertising with the mail. Companies that don't have the reach of a Google, Apple or Microsoft and whos' business plan depends on add revinue from web email access are seeing that access deminish in favour of non-revinue generating access. Some are begining to try strategies to recapture lost income. You don't have to go back very far in time to find email administrators talking to each other about how to shed all thos pesky smartphone connections that aren't passing data. More recently users have learned of and are demanding "push" email.

When I see a service level reduction that occurs regularly and roughly synchronized to a diurnal usage pattern, well let's just say that I've seen telecom companies do that many times over the decades.

[Omnitech]

And I lost my patience with the snide replys some time ago.

"snide replys" eh?

Whatever, I'm done with this meta-discussion, if you can call it a discussion. (More like I make one point, and you redirect until the cows come home and bring up one after the other irrelevant point or red herring.) Feel free to find someone else to 'debate' this with.

[Richard Buckley]

"snide replys" eh?

Whatever, I'm done with this meta-discussion, if you can call it a discussion. (More like I make one point, and you redirect until the cows come home and bring up one after the other irrelevant point or red herring.) Feel free to find someone else to 'debate' this with.

Call them redirections, irrelevant or red herrings if you want. The fact is you can't talk about TCP/IP without considering reaction to congestion (loss events) because the slow start and congestion avoidance procedures will continue to increase the amout of data introduced to the network until a loss event occurs. If not during the slow start exponential growth phase, then during the congestion avoidance linear growth phase. For the same reason you can't discuss TCP/IP buffering without considering the impact the buffering strategy will have on congestion avoidance.

Am I supposed to just accept what ever you say because if I don't you'll roll on the floor laughing at me? Even when you contradict the protocol specification and what the designers of the protocol have to say? Please. If that's the case then yielding is probably your best bet.

On the other hand if you would care to provide an authoritative link to this "well known" IMAP/IDLE issue, or to a white paper describing a router implementation where packet buffers are assigned to each connection on establishment I would be happy to read them.