Information age meets rural Ireland

The irish Governments plan on getting ISPs to monitor, filter and block subscribers from downloading illegal content

2011-12-21T13:41:00.000-08:00

I rarely get political. First of all, I'm a non-national in Ireland, which means I can't vote on anything bar local stuff anyhow. But when it comes to the point, where they interfere with my bread and butter, to the point, where they actually could bankrupt me, I get furious. Especially, when they clearly don't have a clue of the industry they are affecting nor do proper consulting as usual.

The latest item on the agenda is getting ISPs to block internet users/subscribers from downloading illegal content / piracy.

What they don't realize though, is that if they force ISPs to filter/log content to the extend that they are looking into, they will most likely ruin most of the smaller regional ISPs financially in a split second on the cost for doing this alone. Especially because every good ISP doesn't have one single point where he terminates his internet traffic, nor one single point within the network where all traffic would be processed. Just for the sake of fail over. That means filters will have to be placed in several locations or a single point of failure will have to be created. Never mind the cost for putting the technology in place, another problem is the man power needed to update these filters on an ongoing basis as the ways the internet is used changes on a daily basis. Even for the ISPs that actually want to limit things like P2P, it's very hard to do so today. That's without having to try to log the data.

The impact on privacy is another problem, which has been given no thought at all. Implementing this type of filtering will literally give the Copyright agencies the right to tap your phone line. You think not ? Are you aware, how many phone calls are processed and routed across the Internet nowadays ? Even when you don't know it, IP data networks are used to route your phone call.

Also, no thought has been given to the problems, that already exists. Just look what happened to the way Eircom (and the Copyright holders) handled the three strikes scenario: http://businessetc.thejournal.ie/massive-blow-to-music-industry-as-eircom-anti-piracy-measures-rejected-307584-Dec2011/

Or do we want a scenario where the Music Industry tries to abuse law to earn more money: http://torrentfreak.com/music-rights-group-bills-internet-providers-for-piracy-licence-11110/

Better yet, the Music Industry doesn't even ensure, that they pay the copyright holders nor keep to their contracts: http://torrentfreak.com/copyright-corruption-scandal-surrounds-anti-piracy-campaign-111201/

In the US, it's going as far, that the Music Industry now collects royalties for the event, that you might be playing copyrighted music: http://seattletimes.nwsource.com/html/businesstechnology/2003815486_royalty01.html Before you ever violated any copyright law in the first place. Just in case, you know !

Another recent show, how the Industry just hits before even asking is the MegaUpload case: http://www.techdirt.com/articles/20111214/18263517094/umg-megaupload-case-gets-even-stranger-william-says-he-didnt-authorize-takedown.shtml

So, if or when the government makes a law, that should limit pirating copyrighted material, they should be VERY careful, about where they tackle the issue and on what grounds. Placing the burden at the ISPs is shooting pigeons with a heat-seaking rocket-launcher. The collateral damage will be fairly insane and kick our broadband infrastructure a good decade back to where it came from.

Get a grip. Act responsible and solve the issue at the source. There is plenty of ways to tackle this. When you spill a bowl of rice, you don't go and get somebody else to pick up each and every one of them by hand. You get a brush and brush them up in one go. Or get a vacuum cleaner and fix the mess. You especially don't rip the carpet up at the same time to pin it back down again afterwards.

Matter of fact, this is as bad as the law about knifes in Denmark, that made every builder and even every person that owned a Volkswagen Tuareg a criminal, just because a Stanley knife is part of their toolkit. It basically handed such a person an automatic jail term. At the least the recent government change there meant, that the law got adjusted so that it doesn't hit innocent people anymore ... hopefully.

No shit Sherlock !!!

Ubuntu Karmic

2009-11-29T17:30:00.000-08:00

After having looked at Ubuntu 2 years ago and finding it wasn't flexible enough back then (old habbits don't die) recent problems with the Debian kernel made me have a look at it again.

The Debian problems I'm facing:
* fingerprint scanner on my Thinkpad X60 tablet didn't work anymore.
* laptop wouldn't boot randomly, but getting stuck during boot process.
* fluxbox randomly hangs in combination with wine.
* upgrade of X.org screwed the tablet screen rotation.
* Neither NetworkManager or wicd in Debian support 3G dial-up without configuring it manually and then it's half arsed.
* New atheros drivers are acting up very badly.

and many many more.

So I installed Ubuntu Karmic (9.10) and I'm certainly impressed.

* Touchscreen and tablet-pen work out of the box (had to be configured manually in Debian).
* Fingerprint scanner works flawlessly after installing the applications. After a few manual changes, even for gnome-screensaver
* NetworkManager works right out of the box, including internal 3G datacard.
* Atheros card works right out of the box.
* Suspend works flawlessly right out of the box.
* The boot time is amazing. It only takes a few seconds to boot the laptop.

All in all, I'm impressed. We'll see how long it'll survive on the laptop.

Cogent on IPv6

2009-08-14T14:58:00.000-07:00

Cogent finally set our IPv6 BGP session up today. This brings us to the point where all our 4 of our carriers supply IPv6 to us.

Cogent only has 781 routes at the moment, but I'd say that'll improve over time, seeing that they only get full-feeds from GBLX and NTT, some feeds over Swipnet.

KPN in / GBLX out

2009-06-26T17:20:00.000-07:00

The GBLX saga ends. With plenty off issues over the last year, we've shut the BGP down now and replaced it with IP transit from KPN.

Bringing the session with KPN (both v4/v6) online was comparable painless and so far, it's been flawless.

Google over IPv6

2009-06-12T20:30:00.000-07:00

Google has whitelisted our nameservers for IPv6, which means that Google will answer with AAAA records, if requested, and sites like Google, GMail, Google Maps etc. will be served over native IPv6 to our customers.

Blogger.com doesn't seem to be IPv6 enabled by Google yet.

Callcenter trial

2009-06-11T11:30:00.000-07:00

We're trialing a callcenter in Mayo as of today. Initially they are going to take our sales calls for the next month and we'll see how we get on.

With the growth of the business over the last 3 years, we're simply not able to take the calls at times anymore. At times there are just too many and at other times ... none.

The biggest issue is, that people have the attitude, that they either are impatient or don't want to leave a voicemail. So instead, they start calling like 20, 30, 40 times without leaving a voicemail, when they don't get anybody.

Or they leave a voicemail every time, still call 20, 30, 40 times, not waiting for the callback.

The upside is, if this works out for us, we'll be bringing the call volume down, because we hope to take every call. We should also be able to extend the support hours then.

e-Net Fiber being pulled in

2009-05-18T16:42:00.000-07:00

We decided to move the fiber from Dangan across town to Mervue and also, we're going to light our own dark fiber in Galway. It'll also more than double our bandwidth to Dublin.

The drop connection for that is finally being pulled today, pictures will follow.

The beauty of it all is, that no civils are needed, because we can pull it through existing ducting in the cable-riser of the neighbor building (owned by the same landlord). That way we're ensured, that we've got entirely different paths.

Obviously, this fiber trunk is also on a completely different fiber ring and connection to Dublin from the BT fiber, that already is in Mervue.

Out for GBLX

2009-05-13T22:26:00.000-07:00

I've finally had it with Global Crossing. At least 4 issues in the last 6 months, first the incompetence to rate-limit ND/NS ICMPv6 traffic and I had to stick their nose at it, then several times with lost routing, where the routes weren't dropped.

Tonight again, routing lost, BGP routes are still there, but I had to clear the session manually get all back to normal.

All of that I might be able to live with, if the support was decent, but they are a complete nightmare. Also on April 27th the latency on both our IPv4 and IPv6 circuits went to pot. A consistant increase by 40-60 ms is not something i call Tier1 carrier grade Internet.

We've already advised our Layer2 provider to kill the session with GBLX and we're going to replace it with KPN instead. Hopefully that'll prove to be a lot better.

PIv6 again

2009-05-07T11:39:00.000-07:00

Looks like, I was the first in the RIPE region to request PIv6 space and get it assigned.

The changes in the way PI IP space is handled make it slightly more hassleful: beyond the business registration, a LIR contract is needed now, because PI is charged for as of recently.

It's good, that this gap (the lack of PI for IPv6) finally is closed. Some companies made this a requirement to even consider to move to IPv6.

PIv6 (Provider Independant IPv6 adressing space)

2009-05-06T10:21:00.000-07:00

Finally, RIPE announced, that provider independant IPv6 now is available for request. I've been waiting for this a long time.

Quagga BGP crash

2009-04-30T10:00:00.000-07:00

Our Dublin BGP router just died this morning. It looks like, that there is a bug, that causes Quagga (0.99.10 and as it seems 0.99.11) to crash, when it receives a 32-bit ASN in the as-path on certain peering sessions.

A post to the users list confirmed very quickly, that it was a bug and it wasn't an isolated incident, a work-around was found, implemented and we got things quickly back up and running.

Luckily, our BGP feeds are split over the Galway and Dublin BGP servers, so nobody really noticed.

Netflow analysis

2009-04-27T18:00:00.000-07:00

It gets more and more interesting for us to determine, where our users pull their traffic from etc., also to determine what Internet Exchanges we should connect to next, like LINX, AMS-IX or DE-CIX ?

For that purpose, we've got a trial for IBM Aurora, the Quagga BGP gateways have been installed with pmacct netflow probes a while ago. I had tested various open source solution for analysis, but wasn't quite happy with the result. ntop was working quite ok, but Aurora beats all of them. Obviously it's a commercial solution.

Company goes Twitter

2009-04-26T14:26:00.000-07:00

I've added a Twitter account (apart from a personal one) for network updates and outages, so that people can follow, what's going on.

I know, there are a few out there, that will appreciate it.

Routers for NAT-PT and other services

2009-04-25T23:00:00.000-07:00

I've ordered two Cisco routers.

One is thought as NAT-PT gateway, so that we'll be able to provide a IPv6-only service. The Linux implementation of NAT-PT isn't quite the best and the older Cisco boxes go dead cheap on eBay currently.

The other routers purpose is not fully determined yet. I might use it as 6to4 gateway, depending on what throughput it delivers or for testing.

Cisco Catalyst

2009-04-21T06:28:00.000-07:00

We've recently bought a couple of older Cisco Catalyst switches and replaced the Dell switches in the network.

First of all, the Cisco switches are more flexible and some of them even do Layer3 services, like BGP. So on top of the Catalyst 4006 for ExWest, we've got one of them in TeleCity and one in Mervue.

The changeover was done last week and things seem to be running pretty good. I'm just left with replacing the switches in remote sites like Abbeyknockmoy etc.

The reason for replacing the old switches is, that once in a while, when the VLan configuration is committed, the darn things just go daft and drop everything. All that is left then is to drive on site and powercycle the switch. Not something you really want to do, when the switch is 60-70 km's away.

Another switch that had to go is the Linksys Enterprise switch, because it's just a PAIN to having to find a Windows box, just to configure the Vlans. The switch's webinterface only works in IE and you can't configure Vlan's via SSH or telnet. How daft is that ?

Also our INEX Lan#2 peering are handed by the Catalyst in Dublin now. No major box needed there currently and we can upgrade as we go.

Mikrotik Queues broken AGAIN !! (3/3)

2009-04-08T15:14:00.000-07:00

Just received an email, that of the next version, use-ip-firewall will also have an effect on IPv6 packets.

Hopefully that solves that issues.

Mikrotik Queues broken AGAIN !! (2/3)

2009-04-08T02:19:00.000-07:00

Enlightenment.

I cracked the issue. Finally i found some time to sit down and test, when this is happening, because I only saw it when testing against our own speedtest.net server, but the queues were properly working against other speedtest servers.

So what is different between our network and others ? *PLING* IPv6 !!!!!!

And yes, correctly. Once I disabled IPv6 on my laptop and did the speedtest, the queues will work. Once I enabled IPv6, do the speedtest, the queues won't work.

Basically it means, that Mikrotik is not shaping the whole interface, but only the IPv4 traffic inside it. Should a customer get the brilliant idea to run IPX, Appletalk or whatnot .. well .. or IPv6, like we do, he's got full throttle no limit and the license to bust our network. Fortunatly this only applies to enterprise customers on our layer2 MPLS network right now, but what a bummer.

155 mbit/s link from Ballybaan to Abbeyknockmoy

2009-04-07T19:27:00.000-07:00

The traffic volumes from Abbeyknockmoy are getting every day more difficult to cope with, without killing the latency. We're backhauling on 3 links into town now, to cope with the volume.

The fiber that is being installed by Smart will enable us with access to our own leased dark fiber from e-net, which will link Mervue to Ballybaan.

From there a licensed 7 GHz link at 30+ m on the old tower should enable us to get to Abbeyknockmoy and solve all of the bandwidth issues for a good while.

I've also got to figure out, what kind of kit we're going to use for the dark fiber. Initially our idea was CWDM gear, but that's way out of the budget currently. I've got a few other things that I'm looking at for the interim.

Cisco Catalyst Switch for ExWest

2009-04-02T21:07:00.000-07:00

I've finally aquired a proper switch for Exchange West.

A Cisco Catalyst 4006 with a bunch of FE ports and a couple of GBIC ports will be serving as Internet Exchange in Galway until the demand for something bigger or better arises. Yes, this beast is EoL, next year even EoS, but honestly, the service is going to be free on 100 mbit/s ports and will come with no SLA and no option to buy transit across the exchange.

I've also sourced a SuperVisor III engine for the beast, so that it's running IOS, has a 64 Gbps switching fabric and will forward 48-Mpps in hardware for both Layer 2 and Layer 3/4 traffic.

I reckon' it'll be a good start platform.

e-Net site survey for new fiber trunk (2/2)

2009-04-01T14:00:00.000-07:00

The key was organised and the building surveyed.

All we have to do is to drill holes in two walls. The outside wall of our building, because when you're on the roof of our neighbor building, you are looking at exactly that wall :) Our building is one story higher. And then, once you've run the cable through the half of the building you've got to drill through the plaster-board wall, that was errected, when we moved in. That's it.

We'll just wait for them to do the job now. Might take a couple of weeks, but there's no immidiate rush.

e-Net site survey for new fiber trunk (1/2)

2009-03-31T14:00:00.000-07:00

e-Net was around today to do the full site survey for the fiber install.

Three options are available, but we ended up with exactly, what Keith from Smart and myself had been thinking all along in the first place: the cable-riser in the neighbor building.

No civils are needed for that job, which keeps the cost down and it's going to be guaranteed seperate feeding from the BT fiber.

Unfortunatly I didn't have the key for the neighbor building today, but we'll look at it again tomorrow, when I have it.

GBLX packet loss, flapping sessions

2009-03-30T15:17:00.000-07:00

Turns out, GBLX has seen their filters breaking the NDP protocol before, however it didn't bother them enough to fix that globally.

At least they've fixed that for us now, until somebody decides, that it's not conforming with the standards of their filtering. The connection works again, but we'll definatly be switching for something else, because I can't use a YoYo like that for anything.

Change of core network to measure IPv6 traffic universally

2009-03-29T19:46:00.000-07:00

I wanted to know, how much IPv6 traffic we're actually passing to our gateways, to the INEX, between the 6to4 and miredo gateway etc.

This was something I wanted to do especially, after I found out, that our approx 16-20 mbit/s traffic on the 6to4 gateway actually mainly is traffic between teredo and 6to4 hosts and doesn't end up anywhere, that is native IPv6.

Now, somebody might now come around and tell us to use netflow for that, but that doesn't work everywhere, I'm afraid.

So I came up with a better strategy. Any of our routers in the core network is Vlan capable. So I've stuck the IPv6 traffic in the core in a seperate Vlan from the IPv4, instead of running dual-stack. That way I can just monitor the individual interfaces.

That works pretty much on anything, as long as it's SNMP capable, and gives quite a good overview already. We'll see over the next days and weeks, how that tunes out.

GBLX packet loss, flapping sessions

2009-03-28T18:51:00.000-07:00

It looks like I've cracked the case on where Global Crossing just is plain wrong, with the help of Bernard, who pointed me in the right direction.

I did some tcpdumps on the IPv6 session for Neighbor Discovery and Solicitation, which uses ICMPv6 and is crucial in the interaction of IPv6 hosts. And guess what ? They are even rate-limiting that !!! No wonder, we're constantly loosing the connection to their router.

Anyhow, I've dumped more data including the tcpdump to them and we'll see, what they say to that.

Mikrotik Queues broken AGAIN !! (1/3)

2009-03-26T14:10:00.000-07:00

I discovered, that the queues in RouterOS can be escaped again. The particular setup, where I'm seeing this is a VPLS circuit, bridged to a lan-port. Queuing is established in the manner, as they suggest, by enabling firewalling for bridging and create the queue on the bridge.

So I send them a mail on the 22nd (4 days ago) about that and raised a ticket. Answer today is, they can't reproduce the problem. Guess I've got to figure that one out myself.

INEX Meeting, GBLX, various.

2009-03-26T11:00:00.000-07:00

The INEX meeting in Dublin was on. I gave a talk on IPv6 deployment. The video for that can be found here.

It also looks like in the likes of the issues with GBLX, we're probably going to kill that session and get IPv6 from KPN instead. Global Crossing engineers have no understanding for, that rate-limiting ICMP in IPv6 simply is a no no. Even though they are saying, that they only are limiting the ICMPv6 traffic targeted for their router. It clearly affects us badly. How bad ? Check this out:

The gap that looks ok from the 23rd to the 24th is, when they shut the filter down after much arguing. However on the 24th, after we confirmed it is the filter, that is causing the issue, they just bloody turned it on again. Disregarding that the service is useless to us that way.

Adding 200 mbit/s from Galway to Dublin

2009-03-25T13:00:00.000-07:00

It looks like the next couple of months will be very exciting.

We've recently peaked 86 mbit/s on our BT circuit to TeleCity in Dublin and the Smart circuit on the other end of town is running at around 23 mbit/s. That's an aggregated traffic volume of approx. 110 mbit/s at peak.

Now, we're going to shut the Dangan circuit down, maybe keep it as backup as it doesn't cost us much, but all the traffic on that circuit is going to be moved into our datacenter in Mervue. Also with only 14 mbit/s leeway left, more bandwidth was needed.

Smart is going to supply us with 100 mbit/s L2 (in addition to the BT circuit), which we're going to combine on the Cisco switches I've just bought for our network.

On top of that, we're going to get another 100 mbit/s L2 circuit into DEG in Dublin (another datacenter), where we are moving our INEX Lan#1 connection. That way, we'll be connected to INEX in two different datacenters and will be getting the optimum redundancy out of that.

That's a total of 400 mbit/s from Galway to Dublin now, where 300 mbit/s of that are in our datacenter in Mervue.

There were a lot of reasons for not increasing the circuit with BT:
- we wanted fiber into the building on a physical different path. Smart was not even allowed to bring the fiber in through the same ducting or elevator shaft. So matter of fact, we're bringing the fiber in through a cable-riser in the neighboring building and then through a complete different wall.
- we wanted the fiber on an entirely different path from Galway to Dublin. BT's fiber is along the irish railroads, while Smart uses ESB networks fiber along the pylons and high voltage lines.
- BT only installed a STM1 capable CPE, when they supplied us with the fiber initially. If we had opted just to increase that circuit, the CPE gear would have to be replaced resulting in downtime or us hauling the traffic across town to Dangan, which isn't really an option. The reason for establishing the DC in Mervue was, that we couldn't get decent wireless links to Dangan.

These are just a few of the reasons, why.

GlobalCrossing, IPv6 and Packet Loss. AGAIN !!

2009-03-22T23:11:00.000-07:00

Remember December 16th ?

Well, here we are again. As of the March 18th, the packet loss, flapping BGP sessions, etc is back. I guess the whole game starts over again.

Server Racks for Mervue

2009-03-18T10:00:00.000-07:00

eBay is good for many things. We bought 7 full height racks on eBay. With 5 KVM's, tons of power and network cabling and a a lot of shelfes fit in the racks, this definatly was a bargain.

We made a day trip out of it, rented two Ford Transit vans and collected the racks in Dun Laoghaire ourselfes today.

CeBIT

2009-03-04T17:00:00.000-08:00

After many years, I managed to get to CeBIT again.

With 25% less companies showing their products and 20% less visitors it actually turned out nicely.
It was bearable, no rush, no stress and yes, your feet still hurt at the end of the day.

The important tasks for myself was to find some WDM fiber gear, some gear for licensed microwave links and maybe somebody who manufactured CPEs in a quality acceptable for us.

It looks like I've succeeded in all those tasks. Happy days.

A few notes:
- AVM launched their lab firmware for the Fritz!Box Fon WLan 7270 with IPv6 (Yay !)
- TP-Link doesn't even know what IPv6 is, or practically denies the fact of their knowledge.

Power issues continue

2009-01-18T11:37:00.000-08:00

Most of the power got restored last night, but it seems that today, the south of the county is on. I've seen brief power cuts north of the bay, but anything near of Clare is gone for a while now.

Galway International Rally, wireless and gale force storms

2009-01-17T21:00:00.000-08:00

The Rally is on again. They haven't been fortunate with the weather.

A shed-roof was blown onto the street and delayed one of the stages. In Moylough a tree fell into the powerlines and onto the street covering half of it. Around Monivea a couple of trees fell into the telephone wires and onto the street. The weather is quite inconsitant with dry spells, lashing rain and hail taking each their turn.

On top of all of that from mid day and on most of Moycullen and Menlo had no power. Roscommon seems to have random power issues, too. Our network has been keeping up so far, with only one base known to be damaged for sure, but looking at what's to come over the weekend, we might be unfortunate to see more down the road.

MPLS and RouterOS 3.19

2009-01-16T23:00:00.000-08:00

With the release of RouterOS 3.19 and the last bugs fixed that affected us in the MPLS code, we've actually completed most of the change in the core network.

The changes to VPLS circuits on the core links has added another 15% bandwidth and cut latency into half or better.

Mikrotik RouterOS 3.18 release

2009-01-09T16:42:00.000-08:00

The long awaited RouterOS 3.18 release came out today.

Part of the features, I was looking for was number one on the changelog list "*) IPv6 address auto-configuration: added recursive DNS server option;", but also various fixes in mpls-test and routing-test.

So, we'll start upgrading the lab-routers and DOH !!!! .. Instead of adding rdns advertisements, RA doesn't work at all. No IPv6 autoconfiguration. Let's see how the tests progress, but this is very bad already.

Global Crossings loss of routing

2008-12-16T21:07:00.000-08:00

It's been a while, since I last posted updates.

A lot has happened in the network and we're nearly at the point, where we're ready for automatic IPv6 provisioning. It'll actually mean, that all our customer get IPv6 by default.

One of the biggest issues holding us back is constant issues with Global Crossing (GBLX). Last week, the connection to their router started to have packet loss and it took them 4 days to fix that. Now, 6 days after, with good service, the whole thing starts over again :( The graph below is the trend from the last 10 days:

What this means to us is quite simple. It's not only packet loss to their router, but also complete loss of routing. However never for long enough to purge the routes and fall over to our other transits.

RIPng broken

2008-10-21T11:27:00.000-07:00

Well, it looks like RIPng is broken in RouterOS, when using Vlan's on the interfaces. It has the dynamic routes within the RIPng section, but doesn't redistribute them to the kernel-routing space.

Yuk. No IGP for a while until that is fixed. I will look at the OSPFv3 stuff in the next days, but that needs extensive testing on both Quagga and Mikrotik side, before I'll deploy that.

Got a mail back, that they'll fix it in one of of the next versions.

Mikrotik RouterOS 3.15 release

2008-10-17T04:09:00.000-07:00

The release of ROS 3.15 yesterday fixed quite a few bugs, that affected us.

IPv6 and IPv4 address families can now co-exist on the same BGP session (routing-test). That means, if our testing goes well, we'll be announcing IPv6 routes on INEX Vlan#2, too, very soon.

BIOS 2.18 fixes the issue, where 2.17 was killing ether3 on RB600 and ether2 and 3 on RB500 series.

The issue, that one couldn't disable dns resolving on traceroutes was really annoying for diagnostics as well.

They however broke adding static IPv4 routes via WinBox instead :( and my own RB600 router went into a non-recoverable state during upgrade. Something I haven't seen in a very long time. Netinstall and reload of backup fixed that though.

Ballindereen basestation rebuild

2008-10-11T21:57:00.000-07:00

On wednesday evening the omni in our Ballindereen basestation failed, cause yet unknown, but the board did just not respond anymore. Unfortunatly it's been lashing down since then and until today. The roads on the Galway - Roscommon road were flooded this morning.

You'd say, what has that to do with it? Well, this is a 100ft castle tower with a 20ft pole on the roof and the board we need to replace is at the top of the pole (see left, plastic box on top of the pole). It isn't an easy job to do so and the roof is impossible to climb in wet conditions, let alone dangerous.

The majority of the customers were moved to other basestations in the meanwhile.

Some time after midnight today it stopped raining. We used the nice day to basically rip the whole base apart. It's 2 1/2 years old and needed a overhaul. A upgrade was also planned for some time, so that's what we've been at today.

The (not final) result is the picture on the right. Obviously missing the panels for the inter-base links, yet.

Also the boards aren't mounted on the pole anymore, but reside in the box on the wall with coax to the antennas. That way, we'll be able to maintain them in bad weather. This is something we changed about a year ago, but some of the older bases haven't had the rework done yet. Once finished, a battery and trickle charger move into the empty slot and that'll be it then. This base consists of 1 legacy omni, 3 sectors, and 4 point-to-point links, 4 embedded routers in total. And with the upgrades, it'll be fully MPLS capable.

Mikrotik IPv6 BGP

2008-10-10T23:02:00.000-07:00

Ok,

it looks as it's working anyhow, but i need to create different instances with different router-id's to have two sessions between two routers. Doesn't matter that they are different address-families. What a pain.

The as-path filters that i filed a bug report in August about are still not fixed and it looks like, that I can't filter on IPv6 prefixes, either.

Looks like I'm going to kick this box to h*** soon and replace it with something decent. Mikrotik might be good at wireless, but in regards to routing, it doesn't look pretty.

GBLX IPv4 session live

2008-10-10T16:56:00.000-07:00

Finally the last bit for our IPv4 transit falls into place: Our IPv4 session with Global Crossing got turned up today, immidiatly eating up traffic.

Hurricane IPv6 BGP tunnel TCY <-> FFM

2008-10-09T23:10:00.000-07:00

As IPv6 over Level3 fell flat (PacketExchange can't deliver) and I still have no second IPv6 peer for TeleCity, I did set a second tunnel up with Hurricane Electric. That finally got set up today, lads must be a bit busy at HE.net.

So for IPv6 we've now Smart Telecom (native) and Hurricane London (tunnel) in Dangan, Galway. In TeleCity we've got GlobalCrossing (native) and Hurricane Frankfurk/Main (tunnel), Germany on top of the INEX peerings on Vlan#1.

The INEX Vlan#2 is a Mikrotik router and they still can't do IPv6 BGP without mix/match :(. I've been chasing them on that since 3.10 now, it was supposed to be fixed in 3.12 and as of 3.14 routing-test it still doesn't work, if I got the configuration right.

IPv6 and null-route

2008-10-07T13:56:00.000-07:00

After re-organising the whole network lately, I wanted to get the null-route in place again for our IPv6 range. It's just nicer, than being looped around between two routers.

First barrier: It's not possible in Mikrotik. DOH !!! .. Well .. I raised a ticket with them on that.

Second barrier: I've added it like usual in the zebra daemon of quagga, by using "ipv6 route 2a02:278::/32 Null0". Well, so I thought. It actually does not work on Quagga 0.99.5, which reminds me, that I need to upgrade that box and then test it again.

Anyhow, we'll fix it via "ip" then. The command to archieve this is "ip route add 2a02:278::/32 dev lo"

Job done, all happy.

Blog update

2008-10-05T14:05:00.000-07:00

People that have subscribed to my blog will notice, that suddenly things have happened. Yes, I was idle updating it for a good while and yes, I've catched up on the last 2-3 months, sort of.

Well, hopefully I'll have more time in the future to keep it up to date a bit more often.

GBLX IPv6 and eXpress peering live

2008-10-03T19:14:00.000-07:00

Finally, it took ages to get the details for the Globalcrossing IPv6 and the eXpress peering sessions. But they are live now and we'll just have to wait for the eXpress route-servers to pick up our prefixes. That should happen in the next 24 hours.

All we are waiting on our circuit to PacketExchange is the session for IPv4 from Globalcrossing.

Anyhow, this means, that we have 3 IPv4 and 3 IPV6 transit carriers now.

INEX meeting, Dublin

2008-09-17T14:00:00.000-07:00

Today was INEX meeting. As we're one of the newest members at the INEX, a presentation of the company was required, sort of. The talk can be found here. Unfortunately I didn't have the time to prepare it properly, it got hacked together the same morning, but well, very few people will notice the real hick-ups :).

Level3 session live

2008-09-16T15:00:00.000-07:00

The session for Level3 is live, which means, we've got 3 IPv4 transit carriers for our upstream.

Wall construction

2008-09-16T12:14:00.000-07:00

The construction of the wall is now nearly finished.

Hurricane Electric moved

2008-09-14T00:07:00.000-07:00

I've changed our transit with Hurricane Electric from their Ashbourne BGP router to London. That way we should have a lot better latency on IPv6 routes through them. Twice the atlantic isn't really great for ping times.

Wall construction

2008-09-10T12:14:00.000-07:00

The wall that splits our half of the top floor from the neighbor is being build. There's been a lot of delays for various reasons, but it's

finally happening.
Until we can start building our own walls, get the computer floors in etc., I've stacked the servers in the corner. Ah well, they'll survive.

Cogent transit online

2008-09-08T23:30:00.000-07:00

Our BGP session with Cogent went live today in TeleCity.

More X-connects live

2008-09-05T12:00:00.000-07:00

The cross-connects to Cogent and PacketExchange went live. We'll just have to wait for them to supply us with the session details and test the connections.

BT handover

2008-09-05T11:00:00.000-07:00

BT has finally, 6 weeks late in our schedule and 4 weeks late according to their maximum delivery time, handed the circuit over to us. Anyhow, this finally completes our second circuit to TeleCity and increases our bandwidth into the network drastically.

Uplink in Kilcash

2008-08-22T12:14:00.000-07:00

We stuck a pole up temporarily last night to facilitate for the downloads, that are needed to bring the Midlans game server up to scratch just before the event. Part of the organiser team has "invaded" my home, as I'm near to the event and we can't get into the hall before this evening. The 3G connection was entirely unusable and this gives me a 4 mbit symmetric feed into Newbridge base. Mind ya, it'll blow over on the first windblow. It's going to move to a bracket on the garage later today.

Move to Kilcash, Rahara

2008-08-15T12:05:00.000-07:00

I'm moving to Roscommon. I wasn't looking forward to another winter in the old house, as it's drafty and moist. Usually I would have moved into the living room and that would have been the only place used during the winter. The other incentive for moving is, that our coverage has grown into Co. Roscommon and it'll be better to have somebody locally initially to look after things. Gives a better spread between Ciaran and myself.

Fiber pull in

2008-08-14T12:55:00.000-07:00

Finally, KN has arrived to pull the fiber into the ducts. Notice the difference on the vans ? Whoever screwed the signage up on the Focus should be shot.

And obviously, the manhole is full of water. That needs to be pumped out first.

Anyhow, until now the order is only 2 weeks late, so we'll see how much more it'll take before we go live.

Back on the road

2008-08-09T23:00:00.000-07:00

Well, I've been on rental cars for nearly 2 weeks now and it's way too expensive, even though we're sending the bill on to the insurance.

Since it's gone 2 years, I last had a decent car, Robert and myself drove to Mallusk and imported this lovely BMW 525tds. Horrible drive home though. The M50 got flooded right when we came into Dublin in all the rain. We made it out of there over the N2 just in time.

2008-08-07T12:49:00.000-07:00

BT has finally installed the gear for the fiber to TeleCity in Mervue. We're just waiting for the physical fiber from the road and into the datacenter now.

For the time being, the fiber will reside here and after completion of the datacenter moved into the facility itself.

INEX live

2008-07-31T09:31:00.000-07:00

Both INEX connection are out of quarantine now and we're live. Just made it in July :)

Cherrypicker hit :(

2008-07-29T20:00:00.000-07:00

Somebody drove into the back of our Iveco in the heavy rain, while I was turning off the road. Very annoying. Looks like we'll have to do without for a few days.

INEX Quarantine fail

2008-07-29T12:00:00.000-07:00

Yikes. Turns out, that the switches I've used are leaking their own MAC-addresses into all VLan's. That kicks us another 24 hours back on the schedule.

INEX Quarantine

2008-07-28T21:25:00.000-07:00

Our X-connect has been setup and the quarantine process to be enabled on the two INEX ports has been initiated. We'll see how things go in the next 24 hours.

TeleCity PoP

2008-07-23T13:50:00.000-07:00

Our rack in TeleCity is ready to move in. The first routers are gone in and we're only waiting for our fiber link from Galway and the crossconnects to the carriers and the INEX.

INEX membership

2008-07-10T10:54:00.000-07:00

We got an email from the INEX team today, that our membership has been approved, so it's all downhill from here :)

Mervue basestation

2008-07-07T20:16:00.000-07:00

The first aerials have gone up on the roof and the birds love it. It's the perfect landing spot.

Mervue datacenter

2008-07-03T08:39:00.000-07:00

We've signed a 21 year lease for office space in Mervue Business Park. The consideration was simple, we needed a place to get our

backhaul from, that was of our own, instead of co-hosting in other facilities. Beyond that, the servers had to move from Scalp to a proper facility.
Over the next months we'll be

developing a small office here for 4 or 5 employees and a datacenter for 40 racks.

2008-03-24T05:22:00.000-07:00

We left on Friday for a road trip to Germany, moving Bo's furniture to Ireland. Well, 70 miles into France the transmission in the Transporter bust, Saturday night at 10pm in the middle of nowhere.

We had to abandon the car in Caen, no rental car was to be found on the easter holiday weekend and make our way back to Ireland by any means possibe. 2 days later we arrived back in Galway.

Left to be said, that I still need to figure out, what the repairs will cost, which can be challenging due to the lack of my french skills and obviously I will have to go back to France to collect the car, once repaired.

2008-03-08T08:52:00.001-08:00

I've been in Galway to collect the van from the garage, that we bought in Northern Ireland 3 weeks ago. This van should ease some of our work with the basestations and surveys of new areas drastically. The hoist has a max. height of 14.5m and 8.5m reach.

The picture is taken in the days just after we bought it, when we tested, that all gear was in working order. The van was purchased on auction up there and is a ex-Road Service vehicle. The alternator was damaged and the van needed service, as it's been sitting on the lot for a while.

All that is left now, is to get the import to the Republic done and VAT in the north claimed back.

The van for our new installer is also back on the road, so we've got a fleet of 4 vehicles now.

14 installations have been done this week and I reckon, we'll increase the numbers in the next days, when we let Christy, our new installer loose on his own.

Bo, our new supporter, has been taking calls most of the week, but there's still a lot to learn, both in regards to geography, the structure of the network and the router software, we use.

New office established

2008-03-02T04:41:00.000-08:00

After a long time with absolutely no time on my hands, I thought, I'd start this blog.

The company is after 2 years finally profitable, pays decent salaries and with the recent hirings, we're at a headcount of 5.

Yesterday the new office in Scalp got established, as Bo is going to start with us tomorrow. So we've been pulling wires, moved desks etc. over the weekend.