Quagga BGP crash

Our Dublin BGP router just died this morning. It looks like, that there is a bug, that causes Quagga (0.99.10 and as it seems 0.99.11) to crash, when it receives a 32-bit ASN in the as-path on certain peering sessions.

A post to the users list confirmed very quickly, that it was a bug and it wasn't an isolated incident, a work-around was found, implemented and we got things quickly back up and running.

Luckily, our BGP feeds are split over the Galway and Dublin BGP servers, so nobody really noticed.

Netflow analysis

It gets more and more interesting for us to determine, where our users pull their traffic from etc., also to determine what Internet Exchanges we should connect to next, like LINX, AMS-IX or DE-CIX ?

For that purpose, we've got a trial for IBM Aurora, the Quagga BGP gateways have been installed with pmacct netflow probes a while ago. I had tested various open source solution for analysis, but wasn't quite happy with the result. ntop was working quite ok, but Aurora beats all of them. Obviously it's a commercial solution.

Company goes Twitter

I've added a Twitter account (apart from a personal one) for network updates and outages, so that people can follow, what's going on.

I know, there are a few out there, that will appreciate it.

Routers for NAT-PT and other services

I've ordered two Cisco routers.

One is thought as NAT-PT gateway, so that we'll be able to provide a IPv6-only service. The Linux implementation of NAT-PT isn't quite the best and the older Cisco boxes go dead cheap on eBay currently.

The other routers purpose is not fully determined yet. I might use it as 6to4 gateway, depending on what throughput it delivers or for testing.

Cisco Catalyst

We've recently bought a couple of older Cisco Catalyst switches and replaced the Dell switches in the network.

First of all, the Cisco switches are more flexible and some of them even do Layer3 services, like BGP. So on top of the Catalyst 4006 for ExWest, we've got one of them in TeleCity and one in Mervue.

The changeover was done last week and things seem to be running pretty good. I'm just left with replacing the switches in remote sites like Abbeyknockmoy etc.

The reason for replacing the old switches is, that once in a while, when the VLan configuration is committed, the darn things just go daft and drop everything. All that is left then is to drive on site and powercycle the switch. Not something you really want to do, when the switch is 60-70 km's away.

Another switch that had to go is the Linksys Enterprise switch, because it's just a PAIN to having to find a Windows box, just to configure the Vlans. The switch's webinterface only works in IE and you can't configure Vlan's via SSH or telnet. How daft is that ?

Also our INEX Lan#2 peering are handed by the Catalyst in Dublin now. No major box needed there currently and we can upgrade as we go.

Mikrotik Queues broken AGAIN !! (3/3)

Just received an email, that of the next version, use-ip-firewall will also have an effect on IPv6 packets.

Hopefully that solves that issues.

Mikrotik Queues broken AGAIN !! (2/3)

Enlightenment.

I cracked the issue. Finally i found some time to sit down and test, when this is happening, because I only saw it when testing against our own speedtest.net server, but the queues were properly working against other speedtest servers.

So what is different between our network and others ? *PLING* IPv6 !!!!!!

And yes, correctly. Once I disabled IPv6 on my laptop and did the speedtest, the queues will work. Once I enabled IPv6, do the speedtest, the queues won't work.

Basically it means, that Mikrotik is not shaping the whole interface, but only the IPv4 traffic inside it. Should a customer get the brilliant idea to run IPX, Appletalk or whatnot .. well .. or IPv6, like we do, he's got full throttle no limit and the license to bust our network. Fortunatly this only applies to enterprise customers on our layer2 MPLS network right now, but what a bummer.

155 mbit/s link from Ballybaan to Abbeyknockmoy

The traffic volumes from Abbeyknockmoy are getting every day more difficult to cope with, without killing the latency. We're backhauling on 3 links into town now, to cope with the volume.

The fiber that is being installed by Smart will enable us with access to our own leased dark fiber from e-net, which will link Mervue to Ballybaan.

From there a licensed 7 GHz link at 30+ m on the old tower should enable us to get to Abbeyknockmoy and solve all of the bandwidth issues for a good while.

I've also got to figure out, what kind of kit we're going to use for the dark fiber. Initially our idea was CWDM gear, but that's way out of the budget currently. I've got a few other things that I'm looking at for the interim.

Cisco Catalyst Switch for ExWest

I've finally aquired a proper switch for Exchange West.

A Cisco Catalyst 4006 with a bunch of FE ports and a couple of GBIC ports will be serving as Internet Exchange in Galway until the demand for something bigger or better arises. Yes, this beast is EoL, next year even EoS, but honestly, the service is going to be free on 100 mbit/s ports and will come with no SLA and no option to buy transit across the exchange.

I've also sourced a SuperVisor III engine for the beast, so that it's running IOS, has a 64 Gbps switching fabric and will forward 48-Mpps in hardware for both Layer 2 and Layer 3/4 traffic.

I reckon' it'll be a good start platform.

e-Net site survey for new fiber trunk (2/2)

The key was organised and the building surveyed.

All we have to do is to drill holes in two walls. The outside wall of our building, because when you're on the roof of our neighbor building, you are looking at exactly that wall :) Our building is one story higher. And then, once you've run the cable through the half of the building you've got to drill through the plaster-board wall, that was errected, when we moved in. That's it.

We'll just wait for them to do the job now. Might take a couple of weeks, but there's no immidiate rush.