Send Pipe Full Error message

[stuartgr]

Hi Guys,

We have recently installed a BES 5.0 server on WinServer 2k3 (as a VM on vSphere 4 cluster) and have migrated around half our users (100) to this new server.

Everything appears to be going well, but on examination of the BES Router Log we are getting the following errors;

[30000] (11/12 18:20:54.283):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 18:20:54.408):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 18:20:54.751):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 18:20:54.986):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 18:21:28.237):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 18:30:48.497):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 18:47:58.074):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 18:47:58.324):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 18:48:15.450):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 18:48:39.670):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 20:39:01.274):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 22:03:20.969):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 22:03:21.157):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 22:03:21.391):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 22:03:21.625):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 22:32:05.437):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 22:32:23.079):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 22:44:58.128):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 23:04:19.446):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full
[30000] (11/12 23:04:19.883):{0x126C} [RELAY_SRP:XXXXXXXX:0x00B3B280] Send: Pipe full


As you can see, they happen fairly frequently. We have tried to raise support from our carrier+RIM and they have referred us to the following KB:01854


Unfortunatley this does not actually give any steps to resolve the problem. We have monitored network performance on the server and from the vSphere console and the link usage (1gbit) is negligible.

Can anyone provide any assistance with this problem?

Cheers!
Stuart

[GlenB]

Thouhg I am not familiar with vSphere, my gut feel is the that the VM and its virtual NIC may be the route cause here, struggling to handle the data load. The monitoring you have done may only be showing what is happening on the physical NIC and on the network, VM environments must present a hardware as virtual device and then map that to a physical device, often using bridged networking.

Is this the only VM on the server? Is there a way from within VSpehere to monitor the load on the virtual NIC? Can you try using perfmon within the VM to monitor things like queues of data to the NIC?

[stuartgr]

Hi Glen,

We have a whole lot of other guests on this host, and none of them report any problems... We have just looked into it and there does not seem to be any sustained network activity. It seems to be sitting on around 100Kb/s (this is on a 1 GB network interface.

Cheers

[GlenB]

Stuart - I would go back to your carrier and ask what the next steps are. I still think it is a network issue - the SRP connection is what carries all data between your BES and RIM. For the BES to report "send pipe full" on a regular basis suggests that the BES is unable to push information to RIM quickly enough. Given the network path, it could be your VM, physical server, the route through switches and firewall, or your internet connection.

[Grease73]

Stuart - Did you ever find a fix for this. We have the same issue, and RIM keeps pointing to KB01854 which is useless.....

[stuartgr]

No unfortunatly - We had a carrier log a call with the technical team in Singapore (carrier rep knew one of the high ups) and they could not help us. In saying that, we have been running the two servers since then, and there has been no problems.

The error still occurs in the log but appears to have no negative effects. I have come to the conclusion that in our particular situation (ESX4i + BES 5.0) it is just a bug.....

Are you experiencing any problems other than just the log entries?

[Grease73]

Yes when we get this error in the logs, we have problems activating users, and some complain of getting there mail late. We ran 250 users in the setup for 2 months with no issues, we then migrated 250 more and and started to see this in the logs a few times a day for short periods. We migrated to a total of 620 users and it was constant with mail routing issues. We moved back to 550 mark and we are ok but the logs still seem to spit this error out on a consistent basis.