KPN in / GBLX out

The GBLX saga ends. With plenty off issues over the last year, we've shut the BGP down now and replaced it with IP transit from KPN.

Bringing the session with KPN (both v4/v6) online was comparable painless and so far, it's been flawless.

Out for GBLX

I've finally had it with Global Crossing. At least 4 issues in the last 6 months, first the incompetence to rate-limit ND/NS ICMPv6 traffic and I had to stick their nose at it, then several times with lost routing, where the routes weren't dropped.

Tonight again, routing lost, BGP routes are still there, but I had to clear the session manually get all back to normal.

All of that I might be able to live with, if the support was decent, but they are a complete nightmare. Also on April 27th the latency on both our IPv4 and IPv6 circuits went to pot. A consistant increase by 40-60 ms is not something i call Tier1 carrier grade Internet.

We've already advised our Layer2 provider to kill the session with GBLX and we're going to replace it with KPN instead. Hopefully that'll prove to be a lot better.

GBLX packet loss, flapping sessions

Turns out, GBLX has seen their filters breaking the NDP protocol before, however it didn't bother them enough to fix that globally.

At least they've fixed that for us now, until somebody decides, that it's not conforming with the standards of their filtering. The connection works again, but we'll definatly be switching for something else, because I can't use a YoYo like that for anything.

GBLX packet loss, flapping sessions

It looks like I've cracked the case on where Global Crossing just is plain wrong, with the help of Bernard, who pointed me in the right direction.

I did some tcpdumps on the IPv6 session for Neighbor Discovery and Solicitation, which uses ICMPv6 and is crucial in the interaction of IPv6 hosts. And guess what ? They are even rate-limiting that !!! No wonder, we're constantly loosing the connection to their router.

Anyhow, I've dumped more data including the tcpdump to them and we'll see, what they say to that.

INEX Meeting, GBLX, various.

The INEX meeting in Dublin was on. I gave a talk on IPv6 deployment. The video for that can be found here.

It also looks like in the likes of the issues with GBLX, we're probably going to kill that session and get IPv6 from KPN instead. Global Crossing engineers have no understanding for, that rate-limiting ICMP in IPv6 simply is a no no. Even though they are saying, that they only are limiting the ICMPv6 traffic targeted for their router. It clearly affects us badly. How bad ? Check this out:



The gap that looks ok from the 23rd to the 24th is, when they shut the filter down after much arguing. However on the 24th, after we confirmed it is the filter, that is causing the issue, they just bloody turned it on again. Disregarding that the service is useless to us that way.

GlobalCrossing, IPv6 and Packet Loss. AGAIN !!

Remember December 16th ?

Well, here we are again. As of the March 18th, the packet loss, flapping BGP sessions, etc is back. I guess the whole game starts over again.

Global Crossings loss of routing

It's been a while, since I last posted updates.

A lot has happened in the network and we're nearly at the point, where we're ready for automatic IPv6 provisioning. It'll actually mean, that all our customer get IPv6 by default.

One of the biggest issues holding us back is constant issues with Global Crossing (GBLX). Last week, the connection to their router started to have packet loss and it took them 4 days to fix that. Now, 6 days after, with good service, the whole thing starts over again :( The graph below is the trend from the last 10 days:



What this means to us is quite simple. It's not only packet loss to their router, but also complete loss of routing. However never for long enough to purge the routes and fall over to our other transits.