Valid HTML 4.01 Transitional

A Server in the Clouds

James F. Carter <>, 2019-01-31

Since 2009 I have used IPv6 and IPv4 dual stack on my home net, and the same at work since 2014. My ISP, Verizon (now Frontier), does not support IPv6 natively (in 2019). Therefore I use the Hurricane Electric tunnel broker service, This has worked out well and reliably for a decade.

However, an extremely annoying trend is building up in the commercial world. Certain web services are geographically restricted, such as access to streaming video of sports events, and the providers detect if the client's IP address is within the region they are licensed to serve. Tech-savvy clients whose access is on unfavorable terms, e.g. not free, have learned to subscribe to a VPN or tunneling service whose egress server is more favorably located. Hurricane Electric has the advantage that it is free and has a variety of points of physical presence.

The providers are aware of the VPN and tunnel services and have taken to blocking traffic from them. Apparently this kind of blocking has become a common feature of content distribution networks even if the content is not actually geographically licensed, e.g. advertisements. I initially could not pay my bill on T-Mobile unless I suppressed IPv6 in my browser. The Raspberry Pi support site also is IPv4-only (for me). Now it has gotten so bad that I cannot use IPv6 to read CNN because of long delays loading the advertisements. The VPS Showdown article referred to below is available (to me) only on IPv4. So what am I going to do about this?

The goal is to be able to do normal Internet activities, particularly browsing the web, without degradataion of the user experience by sites which advertise an AAAA record but which block my connections to it or otherwise have botched their IPv6 configuration.

Happy Eyeballs (RFC 8305)

See RFC 8305 Happy Eyeballs Version 2: Better Connectivity Using Concurrency. The recommendation is that when the peer has both A and AAAA records, the client should initiate a connection to the preferred one, normally IPv6, but if it has not responded within a short configurable time, typically 100ms to 300ms, the client should try the other address. Whichever ultimately responds first is used and the other is abandoned. The successful choice should be cached.

Happy Eyeballs is on by default in Mozilla Firefox-64.0 and a lot earlier, and also in Google's Chromium, and likely in all well-maintained browsers. In Firefox it is governed by settings key , which is true by default. So why are my eyeballs unhappy?

At least for T-Mobile, which uses Incapsula as their content distribution and DDoS resistance service, the client makes a complete TLS connection to their reverse proxy site (eyeballs are happy so far), which then labels the connection as fraudulent, and closes it without a reset, so the client hangs until timing out. Hiss, boo, very unhappy eyeballs.

Plans and Requirements

Here is a list of mitigation plans:

  1. Induce Frontier to deploy real native IPv6 on their network. ROTFL.
  2. Induce Frontier to deploy a 6rd tunnel. ROTFL. (RFC 5569; RFC 5969; Wikipedia article)
  3. Induce the global business community to improve their handling of geographically restricted content distribution. ROTFL.
  4. Get onto the global Internet (IPv6) via a tunnel with a better reputation. ROTFL, they're all considered to be fraudulent.
  5. Create my own tunnel to a cloud server under my administrative control. This is the only plan that has a chance of working.

My cloud server will need these networking capabilities. Except as noted, everything is dual stack, IPv6 and IPv4 operating in parallel. The proposed design is described in the present tense even though not yet accomplished (as this paragraph is written).

These issues are seen about network names.

Picking a VPS Provider

What hosting provider has the services I want at a reasonable price?

VPS Showdown series by Josh Sherman. He sets up virtual machines on selected low cost providers and compares the results. This is re-done periodically. For the January 2019 showdown he compares these services:

Their cheap offerings at $5/month get you:

Customer or professional reviews of various hosting companies:

Here are some more details about the Linode VPS service, as of 2019-02-02.

Setting up the Linode

Initial Testing

Is the Linode going to solve my problem? I will test and . I'm using w3m (text only) as the browser.

Network Design

Here's an overview of how the network is going to go.

Work in Progress: Next Steps

Installing Tumbleweed on Surya

I side-graded Linode's SuSE Leap 15.0 image image into Tumbleweed, but I got tangled up with the disc format and the bootloader, so this failed. So I repartitioned the disc and did a bare metal install on the virtual disc, successfully.

Tunnel Between Jacinth and Surya

I have three VPN variants:

Access When the Net is Down

During this project it often happened that I made some improvement to the networking and was cut off from Surya, Jacinth or both. When all else fails I can connect to Surya via LISH (see above for the procedure), and to Jacinth's physical console (if I'm at home). However, for normal SSH access with X-Windows, I get good results using this procedure:


First connect to . Authorize your client's IPv4 address. Then ssh -4 in the normal way. You force IPv4 because there are more moving parts to your IPv6 connection, i.e. tunneling between the client and your tunnel broker and/or Surya or Jacinth itself, and the network screwups and repairs to it are more likely to mess up IPv6 than IPv4.


Use the same procedure except connect to, which is maintained with an 'A' record to Jacinth's current DHCP address from the carrier.

Names on the Wild Side

It turned out to be important, for sane network design, to be consistent about matching up the names and IP addresses of Surya, Jacinth, and Jacinth's various alter egos. In this table is given as Jacinth's wild IPv4 address, but this is from DHCP and changes at random intervals. It is updated on the DNS provider by ddclient.

Domain Name IPv4 Address IPv6 Address Description 2001:470:1f04:844::2 IP addresses on wild side 2600:3c01::f03c:91ff:fea9:875e IP addresses on wild side 2600:3c01::f03c:91ff:fea9:875e
A hybrid: Jacinth wild for IPv4 but Surya wild for IPv6. CNAME to 2600:3c01:e000:306::c1
A hybrid: Surya wild for IPv4 but Jacinth internal for IPv6. 2600:3c01:e000:306::c1 IP addresses on internal LAN 2600:3c01:e000:306::8:1 IP addresses on internal LAN (none) 2001:470:1f04:844::1 Hurricane Electric, their end 2001:470:1f05:844::3 Deprecated internal LAN IP, to be removed

The two hybrids ( and are intended to refer to the master site and its numerous services. IPv6 points to Surya because the IPv6 default route comes out there, and so replies to queries need to have that address; however, there is no benefit from adding complexity by running IPv4 in and out of Surya. is used by internal hosts, and since IPv6 is preferred (on my net), internal hosts will take the most reliable and most efficient route to Jacinth. But if roaming, the host (Xena) can fall back to IPv4 on Surya, which has the advantage of being fixed.

CouchNet Geometry

CouchNet is not a big enterprise net, but it has some details that add to the complication of setting up these tunnels. Most of the machines are on one internal LAN which uses IPv4 and IPv6 (dual stack). Each permanent machine has assigned fixed IP addresses (IPv4+6); a local convention is that the IPv6 address is $prefix::$octet where the last part is the last octet of its IPv4 address, converted to hex. For example, would become $prefix::14 (20 decimal = 14 hex).

The LAN address range for IPv4 could be from RFC 1918, but I'm actually using an equivalent non-routeable /24 range that has been forgotten by everyone but me. The main router (Jacinth or Surya) uses NAT to condense all the internal hosts to one wild-side IPv4 address: DHCP on Jacinth and fixed on Linode.

The IPv6 prefix(es) are assigned by my tunnel broker(s); they are /64 bits. As detailed at the beginning of this document, I have been using Hurricane Electric's service for over a decade, but I'm now switching to Mostly, NAT is not used; authorized wild-side hosts communicate directly with the internal hosts. This is impossible on IPv4.

Several of the machines are host to virtual machines.

The virtual machines all use bridge networking: the host's own IPs are on a bridge device, and the virtual machine's vnet0 is a member of the bridge. On Jacinth and Diamond the wired connection to the internal LAN (eth0) is also a bridge member. So both the host and the guest are directly on the internal LAN. Xena is similar except no eth0; it's a laptop.

Jacinth's wild side for IPv4 is via eth1, which does fairly conventional routing and NAT: traffic to the wild side, if sent via eth1 rather than Linode, gets the NAT treatment on Jacinth. In the old configuration Jacinth's IPv6 gets out via a SIT tunnel to Hurricane Electric; the bearer packets (IPv4) go in and out on eth1. In the new configuration IPv6 will be forwarded through an OpenVPN tunnel to Surya, which has native IPv6 service from Linode. For fallback, both of these need to be operating at the same time.

Xena is a laptop and its networking is on Wi-Fi. It particularly needs to function when roaming, i.e. when it's connected to an arbitrary foreign Wi-Fi net. I would very much like to put wlan0 into the bridge. But there are issues with Wi-Fi which preclude bridge membership, and I have tried and failed to implement various mitigations which I'm not going to discuss here. Instead, wlan0 is treated as a wild-side interface, which is no lie when Xena is roaming, and Xena routes between it and the bridge, very similar to what's done on Jacinth.

Surya routes IPv4+6 traffic to and from the internal LAN via an OpenVPN tunnel, called the segment tunnel. It has a dedicated interface, tun9, on both Surya and Jacinth. Our fixed IPv4 and IPv6 addresses are on eth0, and Linode routes the assigned subnet via the fixed IPv6 address. IPv4 traffic to the wild side gets NAT on Surya (not Jacinth). IPv6 traffic destined for the wild side is routed (without NAT) via Surya's eth0.

For NTP (time service), Jacinth is the master; it gets time from a carefully selected set of pool servers (rather than the generic [0123] The other machines, specifically Xena, use Jacinth and two generic pool servers. I'm running chrony and it doesn't take strata too seriously: it will prefer Jacinth over the pool servers because the quality of its time is substantially better, being on the same LAN (except Xena and Surya).

Three hosts provide directory services: Jacinth (master), Diamond (fallback), and Xena (so it can function autonomously when roaming). The directory services are DNS, LDAP and Kerberos authentication. Presently, Surya does DNAT from the wild side ( for these and several other services, redirecting them to Jacinth. The other DNATted services are IMAP (user mailboxes), XMPP (chat by text, voice, video), Icecast (streaming audio), RoundCube (webmail), OwnCloud (data storage and PIM), and NTP (time service).

Why am I going to the trouble and complication to do DNAT on Surya for these services? The original idea was, a roaming host (Xena, cellphones, and possibly others in the future) would specify as the server for these services, which formerly was Jacinth but now is Surya. The server daemons run on Jacinth, which formerly was exposed directly on the wild side, but now Surya acts as a bastion host and DNATs services through the DMZ to Jacinth. Roaming clients would use OOBA to get authorized to talk to these services on; actually the local LAN clients can talk to also, which formerly would land the connection on Jacinth, as desired.

First, the local LAN (non-roaming) clients need to be configured to talk to jacinth on the internal LAN. If their traffic flies 560km north to Surya, gets DNATted, and flies 560km back to Jacinth, this is awfully stupid and awfully vulnerable to failures, even though it does work. has been provided for this purpose.

Second, I'm wondering if it's worth the effort to make the services available on The alternative is to require the client to open a VPN tunnel, and then to access them directly on Jacinth. Comparing and contrasting:

I have tried it both ways. My final design is to provide both VPNs and DNAT plus OOBA on Surya, both of which work. And internal clients go direct to Jacinth on the internal LAN.

Web service is a little more complicated, since CouchNet has two webservers. Actually each host has a webserver, but Jacinth and Claude are the ones most frequently accessed. Claude (virtual machine on Jacinth) serves material intended for the general public, like this writeup, while various virtual web sites on Jacinth serve home automation, webmail, administrative reports, and so on. Port 80 on Jacinth and Surya, if the source address is on the wild side, is DNATted to Claude. Port 443 is dedicated to OpenVPN, but OpenVPN is now smart enough to DNAT HTTP to the webserver and port of your choice: Claude in my case. Port 80, if the source address is not wild, is served by the host's own webserver (Jacinth or Surya). Hosts that want HTTPS from Jacinth itself (not Claude) need to connect to a nonstandard port. Surya has no useful services over HTTPS, but if it did, it would use a nonstandard port.

CouchNet has several subnets packed within its /24 or /64 prefix, precluding RFC 4862 autoconfiguration for IPv6 (i.e. appending the EUI-64, the MAC address with leading bit 02 inverted and xxff:fexx squeezed in the middle). The subnets are:

IPv4 IPv6 Description
…128/29 ::1:0/112 OpenVPN 1194/udp on Jacinth
…136/29 ::2:0/112 OpenVPN 1194/udp on Surya
…144/29 ::3:0/112 OpenVPN 443/tcp on Jacinth
…152/29 ::4:0/112 OpenVPN 443/tcp on Surya
…160/29 ::6:0/112 StrongSwan (IPSec) on Jacinth
…176/29 ::7:0/112 StrongSwan (IPSec) on Surya
…168/29 ::5:0/112 Xena and its VM
…184/29 ::8:0/112 Segment tunnel: Surya and the other endpoint on Jacinth
…192/26 ::/112 Main CouchNet
…240/28 ::f000/124 DHCP addresses on main CouchNet
…0/25 (various) Future expansion
The relevant prefix is to be prepended to each of these address ranges.

As for routes, for both families except as noted: