Ubuntu Karmic

After having looked at Ubuntu 2 years ago and finding it wasn't flexible enough back then (old habbits don't die) recent problems with the Debian kernel made me have a look at it again.

The Debian problems I'm facing:
* fingerprint scanner on my Thinkpad X60 tablet didn't work anymore.
* laptop wouldn't boot randomly, but getting stuck during boot process.
* fluxbox randomly hangs in combination with wine.
* upgrade of X.org screwed the tablet screen rotation.
* Neither NetworkManager or wicd in Debian support 3G dial-up without configuring it manually and then it's half arsed.
* New atheros drivers are acting up very badly.

and many many more.

So I installed Ubuntu Karmic (9.10) and I'm certainly impressed.

* Touchscreen and tablet-pen work out of the box (had to be configured manually in Debian).
* Fingerprint scanner works flawlessly after installing the applications. After a few manual changes, even for gnome-screensaver
* NetworkManager works right out of the box, including internal 3G datacard.
* Atheros card works right out of the box.
* Suspend works flawlessly right out of the box.
* The boot time is amazing. It only takes a few seconds to boot the laptop.

All in all, I'm impressed. We'll see how long it'll survive on the laptop.

Cogent on IPv6

Cogent finally set our IPv6 BGP session up today. This brings us to the point where all our 4 of our carriers supply IPv6 to us.

Cogent only has 781 routes at the moment, but I'd say that'll improve over time, seeing that they only get full-feeds from GBLX and NTT, some feeds over Swipnet.

KPN in / GBLX out

The GBLX saga ends. With plenty off issues over the last year, we've shut the BGP down now and replaced it with IP transit from KPN.

Bringing the session with KPN (both v4/v6) online was comparable painless and so far, it's been flawless.

Google over IPv6

Google has whitelisted our nameservers for IPv6, which means that Google will answer with AAAA records, if requested, and sites like Google, GMail, Google Maps etc. will be served over native IPv6 to our customers.

Blogger.com doesn't seem to be IPv6 enabled by Google yet.

Callcenter trial

We're trialing a callcenter in Mayo as of today. Initially they are going to take our sales calls for the next month and we'll see how we get on.

With the growth of the business over the last 3 years, we're simply not able to take the calls at times anymore. At times there are just too many and at other times ... none.

The biggest issue is, that people have the attitude, that they either are impatient or don't want to leave a voicemail. So instead, they start calling like 20, 30, 40 times without leaving a voicemail, when they don't get anybody.

Or they leave a voicemail every time, still call 20, 30, 40 times, not waiting for the callback.

The upside is, if this works out for us, we'll be bringing the call volume down, because we hope to take every call. We should also be able to extend the support hours then.

e-Net Fiber being pulled in

We decided to move the fiber from Dangan across town to Mervue and also, we're going to light our own dark fiber in Galway. It'll also more than double our bandwidth to Dublin.

The drop connection for that is finally being pulled today, pictures will follow.

The beauty of it all is, that no civils are needed, because we can pull it through existing ducting in the cable-riser of the neighbor building (owned by the same landlord). That way we're ensured, that we've got entirely different paths.

Obviously, this fiber trunk is also on a completely different fiber ring and connection to Dublin from the BT fiber, that already is in Mervue.

Out for GBLX

I've finally had it with Global Crossing. At least 4 issues in the last 6 months, first the incompetence to rate-limit ND/NS ICMPv6 traffic and I had to stick their nose at it, then several times with lost routing, where the routes weren't dropped.

Tonight again, routing lost, BGP routes are still there, but I had to clear the session manually get all back to normal.

All of that I might be able to live with, if the support was decent, but they are a complete nightmare. Also on April 27th the latency on both our IPv4 and IPv6 circuits went to pot. A consistant increase by 40-60 ms is not something i call Tier1 carrier grade Internet.

We've already advised our Layer2 provider to kill the session with GBLX and we're going to replace it with KPN instead. Hopefully that'll prove to be a lot better.

PIv6 again

Looks like, I was the first in the RIPE region to request PIv6 space and get it assigned.

The changes in the way PI IP space is handled make it slightly more hassleful: beyond the business registration, a LIR contract is needed now, because PI is charged for as of recently.

It's good, that this gap (the lack of PI for IPv6) finally is closed. Some companies made this a requirement to even consider to move to IPv6.

PIv6 (Provider Independant IPv6 adressing space)

Finally, RIPE announced, that provider independant IPv6 now is available for request. I've been waiting for this a long time.

Quagga BGP crash

Our Dublin BGP router just died this morning. It looks like, that there is a bug, that causes Quagga (0.99.10 and as it seems 0.99.11) to crash, when it receives a 32-bit ASN in the as-path on certain peering sessions.

A post to the users list confirmed very quickly, that it was a bug and it wasn't an isolated incident, a work-around was found, implemented and we got things quickly back up and running.

Luckily, our BGP feeds are split over the Galway and Dublin BGP servers, so nobody really noticed.

Netflow analysis

It gets more and more interesting for us to determine, where our users pull their traffic from etc., also to determine what Internet Exchanges we should connect to next, like LINX, AMS-IX or DE-CIX ?

For that purpose, we've got a trial for IBM Aurora, the Quagga BGP gateways have been installed with pmacct netflow probes a while ago. I had tested various open source solution for analysis, but wasn't quite happy with the result. ntop was working quite ok, but Aurora beats all of them. Obviously it's a commercial solution.

Company goes Twitter

I've added a Twitter account (apart from a personal one) for network updates and outages, so that people can follow, what's going on.

I know, there are a few out there, that will appreciate it.

Routers for NAT-PT and other services

I've ordered two Cisco routers.

One is thought as NAT-PT gateway, so that we'll be able to provide a IPv6-only service. The Linux implementation of NAT-PT isn't quite the best and the older Cisco boxes go dead cheap on eBay currently.

The other routers purpose is not fully determined yet. I might use it as 6to4 gateway, depending on what throughput it delivers or for testing.

Cisco Catalyst

We've recently bought a couple of older Cisco Catalyst switches and replaced the Dell switches in the network.

First of all, the Cisco switches are more flexible and some of them even do Layer3 services, like BGP. So on top of the Catalyst 4006 for ExWest, we've got one of them in TeleCity and one in Mervue.

The changeover was done last week and things seem to be running pretty good. I'm just left with replacing the switches in remote sites like Abbeyknockmoy etc.

The reason for replacing the old switches is, that once in a while, when the VLan configuration is committed, the darn things just go daft and drop everything. All that is left then is to drive on site and powercycle the switch. Not something you really want to do, when the switch is 60-70 km's away.

Another switch that had to go is the Linksys Enterprise switch, because it's just a PAIN to having to find a Windows box, just to configure the Vlans. The switch's webinterface only works in IE and you can't configure Vlan's via SSH or telnet. How daft is that ?

Also our INEX Lan#2 peering are handed by the Catalyst in Dublin now. No major box needed there currently and we can upgrade as we go.

Mikrotik Queues broken AGAIN !! (3/3)

Just received an email, that of the next version, use-ip-firewall will also have an effect on IPv6 packets.

Hopefully that solves that issues.

Mikrotik Queues broken AGAIN !! (2/3)

Enlightenment.

I cracked the issue. Finally i found some time to sit down and test, when this is happening, because I only saw it when testing against our own speedtest.net server, but the queues were properly working against other speedtest servers.

So what is different between our network and others ? *PLING* IPv6 !!!!!!

And yes, correctly. Once I disabled IPv6 on my laptop and did the speedtest, the queues will work. Once I enabled IPv6, do the speedtest, the queues won't work.

Basically it means, that Mikrotik is not shaping the whole interface, but only the IPv4 traffic inside it. Should a customer get the brilliant idea to run IPX, Appletalk or whatnot .. well .. or IPv6, like we do, he's got full throttle no limit and the license to bust our network. Fortunatly this only applies to enterprise customers on our layer2 MPLS network right now, but what a bummer.

155 mbit/s link from Ballybaan to Abbeyknockmoy

The traffic volumes from Abbeyknockmoy are getting every day more difficult to cope with, without killing the latency. We're backhauling on 3 links into town now, to cope with the volume.

The fiber that is being installed by Smart will enable us with access to our own leased dark fiber from e-net, which will link Mervue to Ballybaan.

From there a licensed 7 GHz link at 30+ m on the old tower should enable us to get to Abbeyknockmoy and solve all of the bandwidth issues for a good while.

I've also got to figure out, what kind of kit we're going to use for the dark fiber. Initially our idea was CWDM gear, but that's way out of the budget currently. I've got a few other things that I'm looking at for the interim.

Cisco Catalyst Switch for ExWest

I've finally aquired a proper switch for Exchange West.

A Cisco Catalyst 4006 with a bunch of FE ports and a couple of GBIC ports will be serving as Internet Exchange in Galway until the demand for something bigger or better arises. Yes, this beast is EoL, next year even EoS, but honestly, the service is going to be free on 100 mbit/s ports and will come with no SLA and no option to buy transit across the exchange.

I've also sourced a SuperVisor III engine for the beast, so that it's running IOS, has a 64 Gbps switching fabric and will forward 48-Mpps in hardware for both Layer 2 and Layer 3/4 traffic.

I reckon' it'll be a good start platform.

e-Net site survey for new fiber trunk (2/2)

The key was organised and the building surveyed.

All we have to do is to drill holes in two walls. The outside wall of our building, because when you're on the roof of our neighbor building, you are looking at exactly that wall :) Our building is one story higher. And then, once you've run the cable through the half of the building you've got to drill through the plaster-board wall, that was errected, when we moved in. That's it.

We'll just wait for them to do the job now. Might take a couple of weeks, but there's no immidiate rush.

e-Net site survey for new fiber trunk (1/2)

e-Net was around today to do the full site survey for the fiber install.

Three options are available, but we ended up with exactly, what Keith from Smart and myself had been thinking all along in the first place: the cable-riser in the neighbor building.

No civils are needed for that job, which keeps the cost down and it's going to be guaranteed seperate feeding from the BT fiber.

Unfortunatly I didn't have the key for the neighbor building today, but we'll look at it again tomorrow, when I have it.

GBLX packet loss, flapping sessions

Turns out, GBLX has seen their filters breaking the NDP protocol before, however it didn't bother them enough to fix that globally.

At least they've fixed that for us now, until somebody decides, that it's not conforming with the standards of their filtering. The connection works again, but we'll definatly be switching for something else, because I can't use a YoYo like that for anything.

Change of core network to measure IPv6 traffic universally

I wanted to know, how much IPv6 traffic we're actually passing to our gateways, to the INEX, between the 6to4 and miredo gateway etc.

This was something I wanted to do especially, after I found out, that our approx 16-20 mbit/s traffic on the 6to4 gateway actually mainly is traffic between teredo and 6to4 hosts and doesn't end up anywhere, that is native IPv6.

Now, somebody might now come around and tell us to use netflow for that, but that doesn't work everywhere, I'm afraid.

So I came up with a better strategy. Any of our routers in the core network is Vlan capable. So I've stuck the IPv6 traffic in the core in a seperate Vlan from the IPv4, instead of running dual-stack. That way I can just monitor the individual interfaces.

That works pretty much on anything, as long as it's SNMP capable, and gives quite a good overview already. We'll see over the next days and weeks, how that tunes out.

GBLX packet loss, flapping sessions

It looks like I've cracked the case on where Global Crossing just is plain wrong, with the help of Bernard, who pointed me in the right direction.

I did some tcpdumps on the IPv6 session for Neighbor Discovery and Solicitation, which uses ICMPv6 and is crucial in the interaction of IPv6 hosts. And guess what ? They are even rate-limiting that !!! No wonder, we're constantly loosing the connection to their router.

Anyhow, I've dumped more data including the tcpdump to them and we'll see, what they say to that.

Mikrotik Queues broken AGAIN !! (1/3)

I discovered, that the queues in RouterOS can be escaped again. The particular setup, where I'm seeing this is a VPLS circuit, bridged to a lan-port. Queuing is established in the manner, as they suggest, by enabling firewalling for bridging and create the queue on the bridge.

So I send them a mail on the 22nd (4 days ago) about that and raised a ticket. Answer today is, they can't reproduce the problem. Guess I've got to figure that one out myself.

INEX Meeting, GBLX, various.

The INEX meeting in Dublin was on. I gave a talk on IPv6 deployment. The video for that can be found here.

It also looks like in the likes of the issues with GBLX, we're probably going to kill that session and get IPv6 from KPN instead. Global Crossing engineers have no understanding for, that rate-limiting ICMP in IPv6 simply is a no no. Even though they are saying, that they only are limiting the ICMPv6 traffic targeted for their router. It clearly affects us badly. How bad ? Check this out:



The gap that looks ok from the 23rd to the 24th is, when they shut the filter down after much arguing. However on the 24th, after we confirmed it is the filter, that is causing the issue, they just bloody turned it on again. Disregarding that the service is useless to us that way.

Adding 200 mbit/s from Galway to Dublin

It looks like the next couple of months will be very exciting.

We've recently peaked 86 mbit/s on our BT circuit to TeleCity in Dublin and the Smart circuit on the other end of town is running at around 23 mbit/s. That's an aggregated traffic volume of approx. 110 mbit/s at peak.

Now, we're going to shut the Dangan circuit down, maybe keep it as backup as it doesn't cost us much, but all the traffic on that circuit is going to be moved into our datacenter in Mervue. Also with only 14 mbit/s leeway left, more bandwidth was needed.

Smart is going to supply us with 100 mbit/s L2 (in addition to the BT circuit), which we're going to combine on the Cisco switches I've just bought for our network.

On top of that, we're going to get another 100 mbit/s L2 circuit into DEG in Dublin (another datacenter), where we are moving our INEX Lan#1 connection. That way, we'll be connected to INEX in two different datacenters and will be getting the optimum redundancy out of that.

That's a total of 400 mbit/s from Galway to Dublin now, where 300 mbit/s of that are in our datacenter in Mervue.

There were a lot of reasons for not increasing the circuit with BT:
- we wanted fiber into the building on a physical different path. Smart was not even allowed to bring the fiber in through the same ducting or elevator shaft. So matter of fact, we're bringing the fiber in through a cable-riser in the neighboring building and then through a complete different wall.
- we wanted the fiber on an entirely different path from Galway to Dublin. BT's fiber is along the irish railroads, while Smart uses ESB networks fiber along the pylons and high voltage lines.
- BT only installed a STM1 capable CPE, when they supplied us with the fiber initially. If we had opted just to increase that circuit, the CPE gear would have to be replaced resulting in downtime or us hauling the traffic across town to Dangan, which isn't really an option. The reason for establishing the DC in Mervue was, that we couldn't get decent wireless links to Dangan.

These are just a few of the reasons, why.

GlobalCrossing, IPv6 and Packet Loss. AGAIN !!

Remember December 16th ?

Well, here we are again. As of the March 18th, the packet loss, flapping BGP sessions, etc is back. I guess the whole game starts over again.

Server Racks for Mervue

eBay is good for many things. We bought 7 full height racks on eBay. With 5 KVM's, tons of power and network cabling and a a lot of shelfes fit in the racks, this definatly was a bargain.

We made a day trip out of it, rented two Ford Transit vans and collected the racks in Dun Laoghaire ourselfes today.

CeBIT

After many years, I managed to get to CeBIT again.

With 25% less companies showing their products and 20% less visitors it actually turned out nicely.
It was bearable, no rush, no stress and yes, your feet still hurt at the end of the day.

The important tasks for myself was to find some WDM fiber gear, some gear for licensed microwave links and maybe somebody who manufactured CPEs in a quality acceptable for us.

It looks like I've succeeded in all those tasks. Happy days.

A few notes:
- AVM launched their lab firmware for the Fritz!Box Fon WLan 7270 with IPv6 (Yay !)
- TP-Link doesn't even know what IPv6 is, or practically denies the fact of their knowledge.

Power issues continue

Most of the power got restored last night, but it seems that today, the south of the county is on. I've seen brief power cuts north of the bay, but anything near of Clare is gone for a while now.

Galway International Rally, wireless and gale force storms

The Rally is on again. They haven't been fortunate with the weather.

A shed-roof was blown onto the street and delayed one of the stages. In Moylough a tree fell into the powerlines and onto the street covering half of it. Around Monivea a couple of trees fell into the telephone wires and onto the street. The weather is quite inconsitant with dry spells, lashing rain and hail taking each their turn.

On top of all of that from mid day and on most of Moycullen and Menlo had no power. Roscommon seems to have random power issues, too. Our network has been keeping up so far, with only one base known to be damaged for sure, but looking at what's to come over the weekend, we might be unfortunate to see more down the road.

MPLS and RouterOS 3.19

With the release of RouterOS 3.19 and the last bugs fixed that affected us in the MPLS code, we've actually completed most of the change in the core network.

The changes to VPLS circuits on the core links has added another 15% bandwidth and cut latency into half or better.

Mikrotik RouterOS 3.18 release

The long awaited RouterOS 3.18 release came out today.

Part of the features, I was looking for was number one on the changelog list "*) IPv6 address auto-configuration: added recursive DNS server option;", but also various fixes in mpls-test and routing-test.

So, we'll start upgrading the lab-routers and DOH !!!! .. Instead of adding rdns advertisements, RA doesn't work at all. No IPv6 autoconfiguration. Let's see how the tests progress, but this is very bad already.