---
title: "IPv6-native networking: a project report"
date: 2021-08-23
...
If you have reached this post, chances are you already know my [AS211579](https://zotan.network){target="_blank"}
project.
This post serves as a summary of the things I learnt and the roadblocks
I had to overcome on the way to get the network to its current state.

### Goals and setup

For those who don't know the project, here's a quick recap. After having
dabbled a bit with [DN42](https://dn42.eu){target="_blank"} last year during the first lockdown, I wanted to do the real
thing, in
the same global routing system your ISP is using to reach this very web
server. While this sounds unnecessarily convoluted and complicated to
accomplish, it was actually pretty easy and not that expensive. The easy
part is almost completely due to my knowledge gained by interacting with
DN42 and the awesome people in #dn42 on the [hackint](https://hackint.org){target="_blank"} IRC, who have
helped me with debugging the stupidest of mistakes. If you're interested
in any of the things I'm about to talk about, be sure to check out the
community behind it, it's seriously amazing.

Alright, we've established the goals, where do we go from here? I
quickly found a sponsoring AS (something you need for the *cheap* part)
and had all the documents on the way to RIPE (the regional internet
registry responsible for, among others, Europe and Asia). Once all that
was processed, a nice person from one of the many network group chats
I'm in (Wim, if you are reading this, thank you so much), hooked me up
with a free /40 IPv6 subnet for all my routing needs. Next, I got BGP
VMs. Good places to get them are either
[Vultr](https://vultr.com){target="_blank"} (cloud provider)
or various smaller providers like
[iFog](https://ifog.ch){target="_blank"}. Peering mostly
happened on [LocIX](https://locix.online){target="_blank"}.
From there, I provided connectivity to home routers, laptops and other
computer-y devices via WireGuard.

### Going a step further

Once my projects (mostly RPKI) eclipsed the performance level provided
by the VMs, I contacted the nice people at
[Meerfarbig](https://meerfarbig.net){target="_blank"} for a
dedicated machine. That one is the primary server running the network to
this day. If you are trying to set up a similar thing and are looking
for specifics, feel free to [contact me](/#contact) and I will give you
appropriate resources.

### The trials and tribulations of networking without IPv4

Now for the fun part, and the likely reason you are here in the first
place: all the things that broke along the way. You see, as you might be
able to tell from the title, my project goal was to set up an
IPv6-native network. That implies NAT64, DNS64, 464XLAT and a whole bag
of other fun things. But let's start at the beginning.

When you have IPv6-only networks, especially when talking about eyeball
networks (the kind mostly used for content consumption, e.g. viewing
webpages and their content), you will want a way to reach IPv4-only
servers. Many popular websites still presently don't support IPv6. At
time of writing, this includes GitHub, which is fairly important for
developing things. A less important (but still relevant) example is
Reddit. For now, we can't reach those websites. What do we do from here?

Our (first) solution is called NAT64, which translates packets between
IPv6 and IPv4, hence the name. The software I chose for this task is
[Jool](https://jool.mx){target="_blank"}. Setting it up was
fairly trivial, and I quickly set up three redundant NAT64-gateways that
announce the NAT64-WKP (well-known prefix, <span
class="highlight-bg">64:ff9b::/96</span>). So far so good, but how do we
get our systems to actually *use* those gateways?

For that, we need to look at DNS, which is responsible for translating
our domain names (e.g. <span class="highlight-bg">github.com</span>) to
an address we can connect to (e.g. <span
class="highlight-bg">140.82.121.3</span>). If we configure the resolver
that does that lookup to synthesize an AAAA record for our IPv4-only
domain, we can connect to it! And that's what I'm doing. Using
<a href="https://www.nlnetlabs.nl/projects/unbound/about/"
target="_blank">unbound</a>, the address <span
class="highlight-bg">140.82.121.3</span> (from the real A record) is
translated to <span class="highlight-bg">64:ff9b::140.82.121.3</span>,
or <span class="highlight-bg">64:ff9b::8c52:7903</span> in encoded form,
and returned as a synthesized AAAA record. Once we configure our system
to use this resolver, *most* IPv4-only sites work perfectly! If you are
asking why I said most there, I hope you are in for a ride.

### 464XLAT and questioning whether connecting computers together was a good idea

The answer to that question is no, obviously. And it keeps chipping away
at [my sanity](https://xkcd.com/2259/){target="_blank"}. But
since we're here, I might as well roll with it. So, what does 464XLAT
even mean? It's a specific combination of systems in place to ensure
connectivity to IPv4 hosts from IPv6-only networks. Namely, in addition
to DNS64 and the NAT64 gateway (called PLAT in this setup, provider-side
address translator), we need a second piece of software, the CLAT
(client/customer-side address translator).

Why, you ask? Because lots of rather popular software (for example,
Skype and Spotify) not only use IPv4, but hardcoded IPv4 literals. That
means that instead of connecting to <span
class="highlight-bg">somedomain.tld</span>, the software tries to
connect to <span class="highlight-bg">192.0.2.255</span>, which will
fail without a CLAT, since we are only capturing (and synthesizing AAAA
records for) DNS queries. A CLAT will take that packet and translate it
to an IPv6 packet destined for the DNS64-synthesized address of the
target host, the same one the resolver would have synthesized, had we
not used literals.

Okay, sounds simple enough, how do we do this? Most mobile operating
systems (Android and iOS) and some desktop operating systems (Windows)
support this natively, though support outside of cellular connections is
limited to non-existent. Since we are working with WireGuard here, this
won't help us. The solution I used here is giving all clients a private
IPv4 address, and instead running the CLAT on the router the tunnel
terminates on. For compatibility reasons I use addresses from the prefix
<span class="highlight-bg">100.64.0.0/10</span> meant for CGNAT (which
is almost what we are doing here) for this purpose.

### Trying to get it all to work

After the configuration part, getting NAT64 to run was fairly easy,
despite some initial issues with routing the NAT64-WKP. Once I turned on
464XLAT however, everything broke. <span class="highlight-bg">::1</span>
(the IPv6 loopback address) was unreachable. Traceroutes that shouldn't
even have gone through the 464XLAT stopped working. I ran into bugs in
Jool. Two fairly major ones, to be exact. However, the developers were
very helpful in debugging the problems, and got both of them resolved
within about two months (shoutouts to ydahhrk!). If you intend on
deploying a similar setup, I recommend going for the -git version, since
those bugs are fixed there already.

### One more thing

One last roadblock was wireguard-quick for macOS. For those unfamiliar
with these issues, I'm glad you didn't go through that debugging rabbit
hole. To start with, if you are tunneling all of your traffic and your
WiFi or Ethernet connection doesn't support IPv6 while your tunnel does,
macOS will sometimes decide to be smart and not attempt to request AAAA
records at all, thereby making the IPv6 connectivity of the tunnel
redundant. To work around this issue I created a very dodgy-looking
script that is run post-up by wireguard-quick, which creates a custom
network service for the tunnel interface. If you have this problem and
want this script, please [contact me](/#contact) directly as I don't
feel comfortable publishing something that terrible on my website.

Back to wireguard-quick, though. It turns out that the macOS version is
particularly dodgy, since the bypass for the "we only have one routing
table to work with" problem the devs went for is adding a more specific
override route for the tunnel endpoint, which depends on parsing
unchecked output of commands that print the routing table to determine
the default gateway. For reasons I can't explain, this breaks when you
create a custom network service as mentioned above, and it tries to set
the default gateway to <span class="highlight-bg">link#32</span> or
similar. This fails, and therefore the tunnel breaks, as tunnel traffic
is then routed back through the tunnel in an infinite loop. After
contacting the devs in the #wireguard channel on
[libera.chat](https://libera.chat){target="_blank"}, my patch
was accepted and this specific problem shouldn't occur anymore, though
that doesn't change the bodge that is the route monitor code of
wg-quick.

### Epilogue

That concludes the network setup. Everything is running smoothly and
thus far, no further bugs were found. I provide IPv6-tunnels for a few
friends who haven't reported any problems either, so at least for now I
think that this project is complete.

I have already removed IPv4 addresses from a few services I run, and I
hope to do so for the entirety of my online presence by the end of 2021,
maybe with a few exceptions for critical services used by friends from
Austria where major telcos still don't support IPv6.

Maybe the end of IPv4 is actually near, at least in my small corner of
the internet. Thanks for reading, and have a wonderful day.
