Building My Own Network - Part 2

If you haven't already read the last part of this series, you should do that first, you can find that here.

Joining FCIX

Once we heard back from FCIX, we were able to join it. We got our IP allocation and a Letter of Authorization to run a cross connect from our rack to theirs. We sent the LOA to the datacenter and asked them to run the cross connect. It took less than 24 hours for that to be run, and we were in a quarantine VLAN on the internet exchange. This is done to make sure there's no issues with the configuration which can cause issues on the main peering LAN. After some verification, we were on the main peering lan and were able to exchange traffic with other peers. One of the peer is Hurricane Electric, and due to that, we got a free way to move 10gbps of traffic to some destinations. Hurricane Electric has a ton of customers and so carry a lot of traffic for their peers.

Monitoring

Now that we were a bit established, it was time to configure monitoring. While the servers had netdata for monitoring, I'm going to be focusing on how I was monitoring the network rather than the endpoints. I setup the ping monitoring tools to ping the router once every few minutes and send me an email if they aren't able to reach it. That was a basic one but is handy to tell when there might be issues. But I wanted more data, especially on seeing where our traffic was coming from, and where it was going to. The way to do that is by using sFlow/netFlow. There are some good tools for parsing this data like elastiflow but they can be expensive.

Enter Akvorado. It came out right at the perfect time, and was a free tool for flow monitoring. I set it up and boom, had some live data(though had some fun right before Christmas 2024 with our server hosting our Akvorado instance having run out of space). You can play with a demo instance of Akvorado here: https://demo.akvorado.net/.

_{Here's our akvorado, with actual data, you can see we get quite a bit of traffic from Meta, with actual traffic numbers blurred out.}

Akvorado is very useful for figuring out who to peer with and how we can further improve our network by adding peering links with the networks that we transfer a lot of data to and it gives data like what interface traffic from a particular ASN came from and where it went to. Without it, making decisions would be a lot harder.

There's also LibreNMS which is used for monitoring ports, devices and other handy things. It also has a nice weathermap which can be used for seeing which links are getting congested so we can decide to ramp that up.

_{Weathermap for Chicago region}

With monitoring in place, we are getting more and more legit.

Acquiring IP Space

We've been on the IP waitlist with ARIN for some time, and while we didn't get any IPs during the first distribution cycle that we were in, we requested some 4.10 IP Space which can be used alongside IPv6 deployment. That gave us some of our own IPv4 space to play with, and during the second cycle of waitlist distribution, we got our IPv4 space. It felt exhilarating, to have IPv4 space to call our own! It was just 256 IP addresses, a /24 block, but hey it's ours! They sell for $8,000 to $10,000 these days on the open market(don't let the crypto bros find out) so it seems quire nice to get it without paying that much. Now we are fully ready to be a network.

Story so far

We've gotten set up in a datacenter, connected to a internet exchange, setup monitoring and have our own IP space. That's what it takes to be a real network.