e-Net site survey for new fiber trunk (1/2)

e-Net was around today to do the full site survey for the fiber install.

Three options are available, but we ended up with exactly, what Keith from Smart and myself had been thinking all along in the first place: the cable-riser in the neighbor building.

No civils are needed for that job, which keeps the cost down and it's going to be guaranteed seperate feeding from the BT fiber.

Unfortunatly I didn't have the key for the neighbor building today, but we'll look at it again tomorrow, when I have it.

GBLX packet loss, flapping sessions

Turns out, GBLX has seen their filters breaking the NDP protocol before, however it didn't bother them enough to fix that globally.

At least they've fixed that for us now, until somebody decides, that it's not conforming with the standards of their filtering. The connection works again, but we'll definatly be switching for something else, because I can't use a YoYo like that for anything.

Change of core network to measure IPv6 traffic universally

I wanted to know, how much IPv6 traffic we're actually passing to our gateways, to the INEX, between the 6to4 and miredo gateway etc.

This was something I wanted to do especially, after I found out, that our approx 16-20 mbit/s traffic on the 6to4 gateway actually mainly is traffic between teredo and 6to4 hosts and doesn't end up anywhere, that is native IPv6.

Now, somebody might now come around and tell us to use netflow for that, but that doesn't work everywhere, I'm afraid.

So I came up with a better strategy. Any of our routers in the core network is Vlan capable. So I've stuck the IPv6 traffic in the core in a seperate Vlan from the IPv4, instead of running dual-stack. That way I can just monitor the individual interfaces.

That works pretty much on anything, as long as it's SNMP capable, and gives quite a good overview already. We'll see over the next days and weeks, how that tunes out.

GBLX packet loss, flapping sessions

It looks like I've cracked the case on where Global Crossing just is plain wrong, with the help of Bernard, who pointed me in the right direction.

I did some tcpdumps on the IPv6 session for Neighbor Discovery and Solicitation, which uses ICMPv6 and is crucial in the interaction of IPv6 hosts. And guess what ? They are even rate-limiting that !!! No wonder, we're constantly loosing the connection to their router.

Anyhow, I've dumped more data including the tcpdump to them and we'll see, what they say to that.

Mikrotik Queues broken AGAIN !! (1/3)

I discovered, that the queues in RouterOS can be escaped again. The particular setup, where I'm seeing this is a VPLS circuit, bridged to a lan-port. Queuing is established in the manner, as they suggest, by enabling firewalling for bridging and create the queue on the bridge.

So I send them a mail on the 22nd (4 days ago) about that and raised a ticket. Answer today is, they can't reproduce the problem. Guess I've got to figure that one out myself.

INEX Meeting, GBLX, various.

The INEX meeting in Dublin was on. I gave a talk on IPv6 deployment. The video for that can be found here.

It also looks like in the likes of the issues with GBLX, we're probably going to kill that session and get IPv6 from KPN instead. Global Crossing engineers have no understanding for, that rate-limiting ICMP in IPv6 simply is a no no. Even though they are saying, that they only are limiting the ICMPv6 traffic targeted for their router. It clearly affects us badly. How bad ? Check this out:



The gap that looks ok from the 23rd to the 24th is, when they shut the filter down after much arguing. However on the 24th, after we confirmed it is the filter, that is causing the issue, they just bloody turned it on again. Disregarding that the service is useless to us that way.

Adding 200 mbit/s from Galway to Dublin

It looks like the next couple of months will be very exciting.

We've recently peaked 86 mbit/s on our BT circuit to TeleCity in Dublin and the Smart circuit on the other end of town is running at around 23 mbit/s. That's an aggregated traffic volume of approx. 110 mbit/s at peak.

Now, we're going to shut the Dangan circuit down, maybe keep it as backup as it doesn't cost us much, but all the traffic on that circuit is going to be moved into our datacenter in Mervue. Also with only 14 mbit/s leeway left, more bandwidth was needed.

Smart is going to supply us with 100 mbit/s L2 (in addition to the BT circuit), which we're going to combine on the Cisco switches I've just bought for our network.

On top of that, we're going to get another 100 mbit/s L2 circuit into DEG in Dublin (another datacenter), where we are moving our INEX Lan#1 connection. That way, we'll be connected to INEX in two different datacenters and will be getting the optimum redundancy out of that.

That's a total of 400 mbit/s from Galway to Dublin now, where 300 mbit/s of that are in our datacenter in Mervue.

There were a lot of reasons for not increasing the circuit with BT:
- we wanted fiber into the building on a physical different path. Smart was not even allowed to bring the fiber in through the same ducting or elevator shaft. So matter of fact, we're bringing the fiber in through a cable-riser in the neighboring building and then through a complete different wall.
- we wanted the fiber on an entirely different path from Galway to Dublin. BT's fiber is along the irish railroads, while Smart uses ESB networks fiber along the pylons and high voltage lines.
- BT only installed a STM1 capable CPE, when they supplied us with the fiber initially. If we had opted just to increase that circuit, the CPE gear would have to be replaced resulting in downtime or us hauling the traffic across town to Dangan, which isn't really an option. The reason for establishing the DC in Mervue was, that we couldn't get decent wireless links to Dangan.

These are just a few of the reasons, why.

GlobalCrossing, IPv6 and Packet Loss. AGAIN !!

Remember December 16th ?

Well, here we are again. As of the March 18th, the packet loss, flapping BGP sessions, etc is back. I guess the whole game starts over again.

Server Racks for Mervue

eBay is good for many things. We bought 7 full height racks on eBay. With 5 KVM's, tons of power and network cabling and a a lot of shelfes fit in the racks, this definatly was a bargain.

We made a day trip out of it, rented two Ford Transit vans and collected the racks in Dun Laoghaire ourselfes today.

CeBIT

After many years, I managed to get to CeBIT again.

With 25% less companies showing their products and 20% less visitors it actually turned out nicely.
It was bearable, no rush, no stress and yes, your feet still hurt at the end of the day.

The important tasks for myself was to find some WDM fiber gear, some gear for licensed microwave links and maybe somebody who manufactured CPEs in a quality acceptable for us.

It looks like I've succeeded in all those tasks. Happy days.

A few notes:
- AVM launched their lab firmware for the Fritz!Box Fon WLan 7270 with IPv6 (Yay !)
- TP-Link doesn't even know what IPv6 is, or practically denies the fact of their knowledge.