Global Crossings loss of routing

It's been a while, since I last posted updates.

A lot has happened in the network and we're nearly at the point, where we're ready for automatic IPv6 provisioning. It'll actually mean, that all our customer get IPv6 by default.

One of the biggest issues holding us back is constant issues with Global Crossing (GBLX). Last week, the connection to their router started to have packet loss and it took them 4 days to fix that. Now, 6 days after, with good service, the whole thing starts over again :( The graph below is the trend from the last 10 days:



What this means to us is quite simple. It's not only packet loss to their router, but also complete loss of routing. However never for long enough to purge the routes and fall over to our other transits.

RIPng broken

Well, it looks like RIPng is broken in RouterOS, when using Vlan's on the interfaces. It has the dynamic routes within the RIPng section, but doesn't redistribute them to the kernel-routing space.

Yuk. No IGP for a while until that is fixed. I will look at the OSPFv3 stuff in the next days, but that needs extensive testing on both Quagga and Mikrotik side, before I'll deploy that.

Got a mail back, that they'll fix it in one of of the next versions.

Mikrotik RouterOS 3.15 release

The release of ROS 3.15 yesterday fixed quite a few bugs, that affected us.

IPv6 and IPv4 address families can now co-exist on the same BGP session (routing-test). That means, if our testing goes well, we'll be announcing IPv6 routes on INEX Vlan#2, too, very soon.

BIOS 2.18 fixes the issue, where 2.17 was killing ether3 on RB600 and ether2 and 3 on RB500 series.

The issue, that one couldn't disable dns resolving on traceroutes was really annoying for diagnostics as well.

They however broke adding static IPv4 routes via WinBox instead :( and my own RB600 router went into a non-recoverable state during upgrade. Something I haven't seen in a very long time. Netinstall and reload of backup fixed that though.

Ballindereen basestation rebuild

On wednesday evening the omni in our Ballindereen basestation failed, cause yet unknown, but the board did just not respond anymore. Unfortunatly it's been lashing down since then and until today. The roads on the Galway - Roscommon road were flooded this morning.

You'd say, what has that to do with it? Well, this is a 100ft castle tower with a 20ft pole on the roof and the board we need to replace is at the top of the pole (see left, plastic box on top of the pole). It isn't an easy job to do so and the roof is impossible to climb in wet conditions, let alone dangerous.

The majority of the customers were moved to other basestations in the meanwhile.



Some time after midnight today it stopped raining. We used the nice day to basically rip the whole base apart. It's 2 1/2 years old and needed a overhaul. A upgrade was also planned for some time, so that's what we've been at today.

The (not final) result is the picture on the right. Obviously missing the panels for the inter-base links, yet.






Also the boards aren't mounted on the pole anymore, but reside in the box on the wall with coax to the antennas. That way, we'll be able to maintain them in bad weather. This is something we changed about a year ago, but some of the older bases haven't had the rework done yet. Once finished, a battery and trickle charger move into the empty slot and that'll be it then. This base consists of 1 legacy omni, 3 sectors, and 4 point-to-point links, 4 embedded routers in total. And with the upgrades, it'll be fully MPLS capable.

Mikrotik IPv6 BGP

Ok,

it looks as it's working anyhow, but i need to create different instances with different router-id's to have two sessions between two routers. Doesn't matter that they are different address-families. What a pain.

The as-path filters that i filed a bug report in August about are still not fixed and it looks like, that I can't filter on IPv6 prefixes, either.

Looks like I'm going to kick this box to h*** soon and replace it with something decent. Mikrotik might be good at wireless, but in regards to routing, it doesn't look pretty.

GBLX IPv4 session live

Finally the last bit for our IPv4 transit falls into place: Our IPv4 session with Global Crossing got turned up today, immidiatly eating up traffic.

Hurricane IPv6 BGP tunnel TCY <-> FFM

As IPv6 over Level3 fell flat (PacketExchange can't deliver) and I still have no second IPv6 peer for TeleCity, I did set a second tunnel up with Hurricane Electric. That finally got set up today, lads must be a bit busy at HE.net.

So for IPv6 we've now Smart Telecom (native) and Hurricane London (tunnel) in Dangan, Galway. In TeleCity we've got GlobalCrossing (native) and Hurricane Frankfurk/Main (tunnel), Germany on top of the INEX peerings on Vlan#1.

The INEX Vlan#2 is a Mikrotik router and they still can't do IPv6 BGP without mix/match :(. I've been chasing them on that since 3.10 now, it was supposed to be fixed in 3.12 and as of 3.14 routing-test it still doesn't work, if I got the configuration right.

IPv6 and null-route

After re-organising the whole network lately, I wanted to get the null-route in place again for our IPv6 range. It's just nicer, than being looped around between two routers.

First barrier: It's not possible in Mikrotik. DOH !!! .. Well .. I raised a ticket with them on that.

Second barrier: I've added it like usual in the zebra daemon of quagga, by using "ipv6 route 2a02:278::/32 Null0". Well, so I thought. It actually does not work on Quagga 0.99.5, which reminds me, that I need to upgrade that box and then test it again.

Anyhow, we'll fix it via "ip" then. The command to archieve this is "ip route add 2a02:278::/32 dev lo"

Job done, all happy.

Blog update

People that have subscribed to my blog will notice, that suddenly things have happened. Yes, I was idle updating it for a good while and yes, I've catched up on the last 2-3 months, sort of.

Well, hopefully I'll have more time in the future to keep it up to date a bit more often.

GBLX IPv6 and eXpress peering live

Finally, it took ages to get the details for the Globalcrossing IPv6 and the eXpress peering sessions. But they are live now and we'll just have to wait for the eXpress route-servers to pick up our prefixes. That should happen in the next 24 hours.

All we are waiting on our circuit to PacketExchange is the session for IPv4 from Globalcrossing.

Anyhow, this means, that we have 3 IPv4 and 3 IPV6 transit carriers now.

INEX meeting, Dublin

Today was INEX meeting. As we're one of the newest members at the INEX, a presentation of the company was required, sort of. The talk can be found here. Unfortunately I didn't have the time to prepare it properly, it got hacked together the same morning, but well, very few people will notice the real hick-ups :).

Level3 session live

The session for Level3 is live, which means, we've got 3 IPv4 transit carriers for our upstream.

Wall construction

The construction of the wall is now nearly finished.

Hurricane Electric moved

I've changed our transit with Hurricane Electric from their Ashbourne BGP router to London. That way we should have a lot better latency on IPv6 routes through them. Twice the atlantic isn't really great for ping times.

Wall construction

The wall that splits our half of the top floor from the neighbor is being build. There's been a lot of delays for various reasons, but it's



finally happening.
Until we can start building our own walls, get the computer floors in etc., I've stacked the servers in the corner. Ah well, they'll survive.

Cogent transit online

Our BGP session with Cogent went live today in TeleCity.

More X-connects live

The cross-connects to Cogent and PacketExchange went live. We'll just have to wait for them to supply us with the session details and test the connections.

BT handover

BT has finally, 6 weeks late in our schedule and 4 weeks late according to their maximum delivery time, handed the circuit over to us. Anyhow, this finally completes our second circuit to TeleCity and increases our bandwidth into the network drastically.

Uplink in Kilcash

We stuck a pole up temporarily last night to facilitate for the downloads, that are needed to bring the Midlans game server up to scratch just before the event. Part of the organiser team has "invaded" my home, as I'm near to the event and we can't get into the hall before this evening. The 3G connection was entirely unusable and this gives me a 4 mbit symmetric feed into Newbridge base. Mind ya, it'll blow over on the first windblow. It's going to move to a bracket on the garage later today.

Move to Kilcash, Rahara

I'm moving to Roscommon. I wasn't looking forward to another winter in the old house, as it's drafty and moist. Usually I would have moved into the living room and that would have been the only place used during the winter. The other incentive for moving is, that our coverage has grown into Co. Roscommon and it'll be better to have somebody locally initially to look after things. Gives a better spread between Ciaran and myself.

Fiber pull in

Finally, KN has arrived to pull the fiber into the ducts. Notice the difference on the vans ? Whoever screwed the signage up on the Focus should be shot.





And obviously, the manhole is full of water. That needs to be pumped out first.




Anyhow, until now the order is only 2 weeks late, so we'll see how much more it'll take before we go live.

Back on the road

Well, I've been on rental cars for nearly 2 weeks now and it's way too expensive, even though we're sending the bill on to the insurance.

Since it's gone 2 years, I last had a decent car, Robert and myself drove to Mallusk and imported this lovely BMW 525tds. Horrible drive home though. The M50 got flooded right when we came into Dublin in all the rain. We made it out of there over the N2 just in time.

BT has finally installed the gear for the fiber to TeleCity in Mervue. We're just waiting for the physical fiber from the road and into the datacenter now.

For the time being, the fiber will reside here and after completion of the datacenter moved into the facility itself.

INEX live

Both INEX connection are out of quarantine now and we're live. Just made it in July :)

Cherrypicker hit :(

Somebody drove into the back of our Iveco in the heavy rain, while I was turning off the road. Very annoying. Looks like we'll have to do without for a few days.

INEX Quarantine fail

Yikes. Turns out, that the switches I've used are leaking their own MAC-addresses into all VLan's. That kicks us another 24 hours back on the schedule.

INEX Quarantine

Our X-connect has been setup and the quarantine process to be enabled on the two INEX ports has been initiated. We'll see how things go in the next 24 hours.

TeleCity PoP

Our rack in TeleCity is ready to move in. The first routers are gone in and we're only waiting for our fiber link from Galway and the crossconnects to the carriers and the INEX.

INEX membership

We got an email from the INEX team today, that our membership has been approved, so it's all downhill from here :)

Mervue basestation

The first aerials have gone up on the roof and the birds love it. It's the perfect landing spot.

Mervue datacenter

We've signed a 21 year lease for office space in Mervue Business Park. The consideration was simple, we needed a place to get our

backhaul from, that was of our own, instead of co-hosting in other facilities. Beyond that, the servers had to move from Scalp to a proper facility.
Over the next months we'll be


developing a small office here for 4 or 5 employees and a datacenter for 40 racks.

We left on Friday for a road trip to Germany, moving Bo's furniture to Ireland. Well, 70 miles into France the transmission in the Transporter bust, Saturday night at 10pm in the middle of nowhere.

We had to abandon the car in Caen, no rental car was to be found on the easter holiday weekend and make our way back to Ireland by any means possibe. 2 days later we arrived back in Galway.

Left to be said, that I still need to figure out, what the repairs will cost, which can be challenging due to the lack of my french skills and obviously I will have to go back to France to collect the car, once repaired.

I've been in Galway to collect the van from the garage, that we bought in Northern Ireland 3 weeks ago. This van should ease some of our work with the basestations and surveys of new areas drastically. The hoist has a max. height of 14.5m and 8.5m reach.

The picture is taken in the days just after we bought it, when we tested, that all gear was in working order. The van was purchased on auction up there and is a ex-Road Service vehicle. The alternator was damaged and the van needed service, as it's been sitting on the lot for a while.

All that is left now, is to get the import to the Republic done and VAT in the north claimed back.

The van for our new installer is also back on the road, so we've got a fleet of 4 vehicles now.

14 installations have been done this week and I reckon, we'll increase the numbers in the next days, when we let Christy, our new installer loose on his own.

Bo, our new supporter, has been taking calls most of the week, but there's still a lot to learn, both in regards to geography, the structure of the network and the router software, we use.

New office established

After a long time with absolutely no time on my hands, I thought, I'd start this blog.

The company is after 2 years finally profitable, pays decent salaries and with the recent hirings, we're at a headcount of 5.

Yesterday the new office in Scalp got established, as Bo is going to start with us tomorrow. So we've been pulling wires, moved desks etc. over the weekend.