Galway International Rally, wireless and gale force storms

The Rally is on again. They haven't been fortunate with the weather.

A shed-roof was blown onto the street and delayed one of the stages. In Moylough a tree fell into the powerlines and onto the street covering half of it. Around Monivea a couple of trees fell into the telephone wires and onto the street. The weather is quite inconsitant with dry spells, lashing rain and hail taking each their turn.

On top of all of that from mid day and on most of Moycullen and Menlo had no power. Roscommon seems to have random power issues, too. Our network has been keeping up so far, with only one base known to be damaged for sure, but looking at what's to come over the weekend, we might be unfortunate to see more down the road.

MPLS and RouterOS 3.19

With the release of RouterOS 3.19 and the last bugs fixed that affected us in the MPLS code, we've actually completed most of the change in the core network.

The changes to VPLS circuits on the core links has added another 15% bandwidth and cut latency into half or better.

Mikrotik RouterOS 3.18 release

The long awaited RouterOS 3.18 release came out today.

Part of the features, I was looking for was number one on the changelog list "*) IPv6 address auto-configuration: added recursive DNS server option;", but also various fixes in mpls-test and routing-test.

So, we'll start upgrading the lab-routers and DOH !!!! .. Instead of adding rdns advertisements, RA doesn't work at all. No IPv6 autoconfiguration. Let's see how the tests progress, but this is very bad already.

Global Crossings loss of routing

It's been a while, since I last posted updates.

A lot has happened in the network and we're nearly at the point, where we're ready for automatic IPv6 provisioning. It'll actually mean, that all our customer get IPv6 by default.

One of the biggest issues holding us back is constant issues with Global Crossing (GBLX). Last week, the connection to their router started to have packet loss and it took them 4 days to fix that. Now, 6 days after, with good service, the whole thing starts over again :( The graph below is the trend from the last 10 days:



What this means to us is quite simple. It's not only packet loss to their router, but also complete loss of routing. However never for long enough to purge the routes and fall over to our other transits.

RIPng broken

Well, it looks like RIPng is broken in RouterOS, when using Vlan's on the interfaces. It has the dynamic routes within the RIPng section, but doesn't redistribute them to the kernel-routing space.

Yuk. No IGP for a while until that is fixed. I will look at the OSPFv3 stuff in the next days, but that needs extensive testing on both Quagga and Mikrotik side, before I'll deploy that.

Got a mail back, that they'll fix it in one of of the next versions.

Mikrotik RouterOS 3.15 release

The release of ROS 3.15 yesterday fixed quite a few bugs, that affected us.

IPv6 and IPv4 address families can now co-exist on the same BGP session (routing-test). That means, if our testing goes well, we'll be announcing IPv6 routes on INEX Vlan#2, too, very soon.

BIOS 2.18 fixes the issue, where 2.17 was killing ether3 on RB600 and ether2 and 3 on RB500 series.

The issue, that one couldn't disable dns resolving on traceroutes was really annoying for diagnostics as well.

They however broke adding static IPv4 routes via WinBox instead :( and my own RB600 router went into a non-recoverable state during upgrade. Something I haven't seen in a very long time. Netinstall and reload of backup fixed that though.

Ballindereen basestation rebuild

On wednesday evening the omni in our Ballindereen basestation failed, cause yet unknown, but the board did just not respond anymore. Unfortunatly it's been lashing down since then and until today. The roads on the Galway - Roscommon road were flooded this morning.

You'd say, what has that to do with it? Well, this is a 100ft castle tower with a 20ft pole on the roof and the board we need to replace is at the top of the pole (see left, plastic box on top of the pole). It isn't an easy job to do so and the roof is impossible to climb in wet conditions, let alone dangerous.

The majority of the customers were moved to other basestations in the meanwhile.



Some time after midnight today it stopped raining. We used the nice day to basically rip the whole base apart. It's 2 1/2 years old and needed a overhaul. A upgrade was also planned for some time, so that's what we've been at today.

The (not final) result is the picture on the right. Obviously missing the panels for the inter-base links, yet.






Also the boards aren't mounted on the pole anymore, but reside in the box on the wall with coax to the antennas. That way, we'll be able to maintain them in bad weather. This is something we changed about a year ago, but some of the older bases haven't had the rework done yet. Once finished, a battery and trickle charger move into the empty slot and that'll be it then. This base consists of 1 legacy omni, 3 sectors, and 4 point-to-point links, 4 embedded routers in total. And with the upgrades, it'll be fully MPLS capable.