Global Crossings loss of routing

It's been a while, since I last posted updates.

A lot has happened in the network and we're nearly at the point, where we're ready for automatic IPv6 provisioning. It'll actually mean, that all our customer get IPv6 by default.

One of the biggest issues holding us back is constant issues with Global Crossing (GBLX). Last week, the connection to their router started to have packet loss and it took them 4 days to fix that. Now, 6 days after, with good service, the whole thing starts over again :( The graph below is the trend from the last 10 days:



What this means to us is quite simple. It's not only packet loss to their router, but also complete loss of routing. However never for long enough to purge the routes and fall over to our other transits.

RIPng broken

Well, it looks like RIPng is broken in RouterOS, when using Vlan's on the interfaces. It has the dynamic routes within the RIPng section, but doesn't redistribute them to the kernel-routing space.

Yuk. No IGP for a while until that is fixed. I will look at the OSPFv3 stuff in the next days, but that needs extensive testing on both Quagga and Mikrotik side, before I'll deploy that.

Got a mail back, that they'll fix it in one of of the next versions.

Mikrotik RouterOS 3.15 release

The release of ROS 3.15 yesterday fixed quite a few bugs, that affected us.

IPv6 and IPv4 address families can now co-exist on the same BGP session (routing-test). That means, if our testing goes well, we'll be announcing IPv6 routes on INEX Vlan#2, too, very soon.

BIOS 2.18 fixes the issue, where 2.17 was killing ether3 on RB600 and ether2 and 3 on RB500 series.

The issue, that one couldn't disable dns resolving on traceroutes was really annoying for diagnostics as well.

They however broke adding static IPv4 routes via WinBox instead :( and my own RB600 router went into a non-recoverable state during upgrade. Something I haven't seen in a very long time. Netinstall and reload of backup fixed that though.

Ballindereen basestation rebuild

On wednesday evening the omni in our Ballindereen basestation failed, cause yet unknown, but the board did just not respond anymore. Unfortunatly it's been lashing down since then and until today. The roads on the Galway - Roscommon road were flooded this morning.

You'd say, what has that to do with it? Well, this is a 100ft castle tower with a 20ft pole on the roof and the board we need to replace is at the top of the pole (see left, plastic box on top of the pole). It isn't an easy job to do so and the roof is impossible to climb in wet conditions, let alone dangerous.

The majority of the customers were moved to other basestations in the meanwhile.



Some time after midnight today it stopped raining. We used the nice day to basically rip the whole base apart. It's 2 1/2 years old and needed a overhaul. A upgrade was also planned for some time, so that's what we've been at today.

The (not final) result is the picture on the right. Obviously missing the panels for the inter-base links, yet.






Also the boards aren't mounted on the pole anymore, but reside in the box on the wall with coax to the antennas. That way, we'll be able to maintain them in bad weather. This is something we changed about a year ago, but some of the older bases haven't had the rework done yet. Once finished, a battery and trickle charger move into the empty slot and that'll be it then. This base consists of 1 legacy omni, 3 sectors, and 4 point-to-point links, 4 embedded routers in total. And with the upgrades, it'll be fully MPLS capable.

Mikrotik IPv6 BGP

Ok,

it looks as it's working anyhow, but i need to create different instances with different router-id's to have two sessions between two routers. Doesn't matter that they are different address-families. What a pain.

The as-path filters that i filed a bug report in August about are still not fixed and it looks like, that I can't filter on IPv6 prefixes, either.

Looks like I'm going to kick this box to h*** soon and replace it with something decent. Mikrotik might be good at wireless, but in regards to routing, it doesn't look pretty.

GBLX IPv4 session live

Finally the last bit for our IPv4 transit falls into place: Our IPv4 session with Global Crossing got turned up today, immidiatly eating up traffic.

Hurricane IPv6 BGP tunnel TCY <-> FFM

As IPv6 over Level3 fell flat (PacketExchange can't deliver) and I still have no second IPv6 peer for TeleCity, I did set a second tunnel up with Hurricane Electric. That finally got set up today, lads must be a bit busy at HE.net.

So for IPv6 we've now Smart Telecom (native) and Hurricane London (tunnel) in Dangan, Galway. In TeleCity we've got GlobalCrossing (native) and Hurricane Frankfurk/Main (tunnel), Germany on top of the INEX peerings on Vlan#1.

The INEX Vlan#2 is a Mikrotik router and they still can't do IPv6 BGP without mix/match :(. I've been chasing them on that since 3.10 now, it was supposed to be fixed in 3.12 and as of 3.14 routing-test it still doesn't work, if I got the configuration right.