Since 2009 I have used IPv6 and IPv4 dual stack on my home net, and the same at work since 2014. My ISP, Verizon (now Frontier), does not support IPv6 natively (in 2019). Therefore I use the Hurricane Electric tunnel broker service, This has worked out well and reliably for a decade.
However, an extremely annoying trend is building up in the commercial world. Certain web services are geographically restricted, such as access to streaming video of sports events, and the providers detect if the client's IP address is within the region they are licensed to serve. Tech-savvy clients whose access is on unfavorable terms, e.g. not free, have learned to subscribe to a VPN or tunneling service whose egress server is more favorably located. Hurricane Electric has the advantage that it is free and has a variety of points of physical presence.
The providers are aware of the VPN and tunnel services and have taken to
blocking traffic from them. Apparently this kind of blocking has become a
feature of content distribution networks even if the content
is not actually geographically licensed, e.g. advertisements. I initially
could not pay my bill on T-Mobile
unless I suppressed IPv6 in my browser. The
Raspberry Pi support site also
is IPv4-only (for me).
Now it has gotten so bad that I cannot use IPv6 to read
because of long delays loading the advertisements. The
article referred to below is available (to me) only on IPv4.
So what am I going to do about this?
The goal is to be able to do normal Internet activities, particularly browsing the web, without degradataion of the user experience by sites which advertise an AAAA record but which block my connections to it or otherwise have botched their IPv6 configuration.
See RFC 8305
Eyeballs Version 2: Better Connectivity Using Concurrency. The
recommendation is that when the peer has both A and AAAA records, the client
should initiate a connection to the preferred one, normally IPv6, but if it
has not responded within a short configurable time, typically 100ms to 300ms,
the client should try the other address. Whichever ultimately responds first
is used and the other is abandoned. The successful choice should be cached.
Happy Eyeballs is on by default in Mozilla Firefox-64.0 and a lot earlier, and also in Google's Chromium, and likely in all well-maintained browsers. In Firefox it is governed by settings key network.http.fast-fallback-to-IPv4 , which is true by default. So why are my eyeballs unhappy?
At least for T-Mobile, which uses Incapsula as their content distribution and DDoS resistance service, the client makes a complete TLS connection to their reverse proxy site (eyeballs are happy so far), which then labels the connection as fraudulent, and closes it without a reset, so the client hangs until timing out. Hiss, boo, very unhappy eyeballs.
Here is a list of mitigation plans:
My cloud server will need these networking capabilities. Except as noted, everything is dual stack, IPv6 and IPv4 operating in parallel. The proposed design is described in the present tense even though not yet accomplished.
These issues are seen about network names.
informaland is never seen on the global internet. DNS is provided by servers on the internal LAN, which are used by internal hosts travelling on the wild side, over their own tunnels.
What hosting provider has the services I want at a reasonable price?
VPS Showdown series by Josh Sherman. He sets up virtual machines on selected low cost providers and compares the results. This is re-done periodically. For the January 2019 showdown he compares these services:
Their cheap offerings at $5/month get you:
1Gb RAM, 1 core (specs below), 25Gb disc (SSD), and 1Tb/month data transfer, except Lightsail offers 40Gb disc and 2Tb/mo transfer.
CPU basic capabilites: 2.25 to 2.50 GHz; 16 to 30 Mb cache; 3059 to 3333 BogoMips. He says, when evaluating VMs you should create several instances and get a feel for how much they vary.
Time per event: jimc is not sure what an event is, but I assume it has to do with the internal resources used to serve a simple web page. Average speeds range from 1.05ms to 1.47ms, but the maximum ranges up to 14.78ms, attributed to possible neighbors hogging the CPU. For him, DigitalOcean has the fastest average speed.
Memory reading and writing: the benchmark did 8.1e5 to 3.8e6
operations per second.
File reading: 1094 to 2159 operations per second; writing: 729 to 1439 operations per second.
MySQL 670 to 3055 transactions per second.
Network speed tests: client was 2.09e6 to 2.45e6 meters from the server as the vulture flies. Latency was around 45ms; download 1.69e8 bit/sec to 1.28e9 bit/sec (big range there); upload 2.96e8 to 5.66e8 bit/sec.
Josh's conclusion: DigitalOcean had the lead in most metrics, and Lightsail gives you more storage and net data allowance. But ease of use and user experience are not reflected in the quantitative metrics. Please use the signup links in the article so he can get a credit.
Customer or professional reviews of various hosting companies:
DigitalOcean: On Digital.com: Inception 2011. They are currently the 3rd largest web host by count of servers (VM's?). Payment is hourly with a monthly maximum. Inbound data is free, outbound counts toward your limit. Their products include shared hosting (a virtual host on their webserver) and virtual private server (VPS). They have 8 datacenters, nearest is Silicon Valley.
Amazon Lightsail on TechRadar, review by Mike Williams (2018-08-28). Lightsail is based on AWS (Amazon Web Services). It really is a VPS, accessible by SSH. Up to 5 static IPs are free. Tech support costs a lot extra; you need to be able to handle the sysadmin issues. Datacenters are everywhere and uncountably many.
Linode on WhoIsHostingThis: Review by Claire Broadley, updated 2018-12-28. Inception 2003. They only (?) do VPS. 8 datacenters including California (Fremont, from Hurricane Electric). Root access to the VM. They will do backups at extra cost. Glowing customer reviews (selected?) They do have a reputation for good customer support, although you are expected to handle sysadmin stuff; they won't hold your hand unless you pay extra.
Vultr: Operating since 2014. On HostAdvice.com their reviewers are about equally balanced between wonderful and awful. For the latter, there are the usual trolls who have no idea how to interact with a business to make it work for them, but there are a lot more than usual complaints about Vultr's customer service. Vultr has 15 datacenters including Los Angeles. One fixed IP included.
Jimc's conclusion: The four vendors that I focused on have fairly similar product offerings at a relatively good price. As Josh Sherman pointed out, non-quantitative factors are going to drive the decision. Vultr customer reviews give me a not so comfortable feeling. DigitalOcean is a new entry in my experience. Amazon Lightsail: I have a lot of confidence in Amazon, and they are currently among the White Hats politically, but cozying up to the big dragon makes me rather nervous. I've known about Linode for a long time and I get a positive feeling about them from buzz on the web. I came into this project inclined to put my cloud server on Linode, and I'm sticking with this choice.
Here are some more details about the Linode VPS service, as of 2019-02-02.
$5/mo or $0.0075/hr gets you 1Gb RAM, 1 core, 25Gb disc, 1Tb data (presumed to be per month), 40Gb/s data rate incoming, 1Gb/s outgoing. You can multiply these in powers of 2, sort of, including the cost; see the pricing page for actual ratios.
You are billed at the end of each month. You pay by the hour up to the listed price per month. You pay for the existence of the machine; you still pay if it's powered off; when you're done with your Linode, delete it. You can increase or decrease resources at any time with no hassle, or add additional machines. Inbound traffic is free. If you go over your outbound quota, you pay $0.02 per Gb. ($5 for 1Tb = $0.005/Gb.) Payment by credit card, PayPal, or check (contact customer service first). You can prepay. Cancel in the first 7 days and get a full refund. If you read their Getting Started document they will give you a promo code for a small credit on a new account.
An additional fixed IP costs $1/mo but you need to give tech support a technical justification for it, due to exhaustion of IPv4 address space.
Create your account at Linode.com.
These items are needed for your profile:
getting startedarticle, they give you a promo code for a small credit toward your bill.
Key points from the Customer Agreement (IANAL):
Creating your Linode(s):
Add a Linode(right side below list of existing Linodes if any).
The next page is the Dashboard summary page. To get to it, in the
list of your Linodes, click on the name field or
of the line item for your Linode.
Brand New(refresh the page to see it)
Is the Linode going to solve my problem? I will test https://www.t-mobile.com/ and https://www.raspberrypi.org/ . I'm using w3m (text only) as the browser.
Positive control on my laptop on the internal net. With w3m, T-Mobile delivers the front page on both IPv4 and IPv6. With Firefox, T-Mobile delivers on IPv4 but hangs on IPv6 -- it must be that some page elements are blocked, like a style sheet or an image that w3m would not download, but the main page isn't blocked. RPi delivers on IPv4 but hangs on IPv6. I'm going to confine the testing to the RPi site.
Success! Executing on the Linode with w3m, and with both IPv4 and IPv6, https://www.raspberrypi.org/ delivers its front page. No delays, no hassle.
Running Firefox on a remote host is always a pain in the butt, but I'm going to install it on the Linode and try T-Mobile.
Conclusion: I'm on the right track, and I will continue to come up with a working solution.
Here's an overview of how the network is going to go.
My firewall relies on eth0 being on the
trusted net, and
rather than making this conditional per host, I'm going to use a udev
rule to rename eth0 to eth1, which the firewall knows is on the wild
For the tunnel I think that IPSec is more efficient of CPU time, politically correct and technologically cool. However, it has two disadvantages: First it isn't really a tunnel; the payloads are considered to have arrived from the interface that the ciphertext arrived on. This can be worked around, but it's going to be quicker and more reliable (designwise) to use OpenVPN, whose payloads emerge from a tun/tap device (tun0). Second, OpenVPN is either on or off, and if off, no data can be transferred. With IPSec, if the Security Association goes away the payloads can still be transferred without protection. I'm sure that the firewall can recognize tunnelled packets reliably and can toss any that lack a security policy. But when setting this up initially, I'm going to keep it simple and use OpenVPN, and I'll investigate IPSec later.
The gateway (Jacinth) will initiate a connection to Surya, and will reconnect automatically after disruptions such as a changed dynamic address.
The tunnel endpoint on Surya will have internal net IP addresses (IPv4 and IPv6) specific for Surya.
Jacinth now has OpenVPN responders on ports 1194/udp and 443/tcp. These will migrate to Surya. OpenVPN hands off HTTP on port 443 to Claude, the webserver (now and in future). With luck, the responder on 1194/udp can handle the tunnel from Jacinth.
www.jfcarter.net:80 (IPv4) will get DNAT to Claude. www.jfcarter.net's AAAA record will point directly to Claude.
Packets coming into Surya that are addressed (possibly after DNAT) to internal net addresses will be routed down the tunnel that Jacinth initiated.
The OOBA service opens a firewall hole for a specific IP address. (Authentication required.) This will migrate from Jacinth to Surya. I may or may not continue to run OOBA on Jacinth: it would be useful only if Surya became inoperative and if I were diagnosing or repairing it from the wild side and if I knew the current wild side address of Jacinth, each of which events are rare.
For the ssh daemon's nonstandard port, it will require OOBA authentication. It will not get DNAT (IPv4) on Surya. Jacinth will specially route this port direct to Surya's wild side, not through the tunnel. (Or can I rely on Lish access?)
Secure the machine. From Securing Your Server:
ssto list all sockets. It is like netstat but better. Command line: ss --listening --processes --tcp --udp) [sshd is the only service and I require it.]
Securing the manager account, from Linode Manager Security Controls:
scratch code(one-time password) in case of broken two-factor authentication.
Make Surya a part of my net: add its various addresses to /etc/hosts and directory server files (DNS etc). [Done]
It looks like they're using RFC 4862 addresses for everything in their datacenter, for example lish-fremont.linode.com has IPv6 address 2600:3c01::f03c:91ff:fe93:e32e. So I can't just pick arbitrary IPv6 addresses. I can ask tech support for allocations, size /116, /64 or /56. [Asked; they gave me a /64.]
Add Surya's private and public IPv6 addresses.
yast2 lan. Infrastructure for yast2 GUI is not installed, using ncurses. Add the addresses on the added subnet as well as the RFC 4862 address (with EUI-64) on the main subnet.
Add Surya to dyn.com DNS. DNS will get (temporarily) the same assignments as are in /etc/hosts, except replacing the realm with jfcarter.net. This will have to be done over when the tunnel is working and I want the addresses to point to e.g. www.jfcarter.net. [Done]
Activate Linode's reverse DNS (afterward). On Dashboard-Remote Access
tab, find the Reverse DNS link. Fill in the referent of the PTR record,
e.g. surya.jfcarter.net, and hit Look Up. In the resulting form, hit
Yes to make the address that it found point to that host. This will
have to be done over when the tunnel is working and I want the addresses to
point to e.g. www.jfcarter.net. [Done]
Running service daemons.
About haveged: This is required on some old AMD CPUs, but otherwise rng-tools has access to the hardware random number generator and so is better. I tested rng-tools (/usr/sbin/rngd), successfully. During installation, haveged will be replaced by rng-tools. A peculiarity: on the main net machines I hardly ever see FIPS test failures, but on 3 test runs (250 trials each) I got 4 failed trials, various patterns failed. On truly random data a few failures are inevitable, and this is well under the action level which I have set at 6 out of 250, but even so it's strange and I should keep an eye on it.
Either create a CouchNet configuration compatible with SuSE 15.0 and put it on Surya, or up-down-sideways-grade the existing Leap 15.0 into Tumbleweed and put my standard configuration on it.
I'm thinking of switching from Tumbleweed to a non-rolling distro, currently Leap 15.0, but I've decided that it's a jackass choice to entangle that project with the tunnel issue, so I'm going to sideways-grade Leap 15.0 to Tumbleweed. The key item is going to be installing the distro definitions: I should download direct from SuSE, not use CouchNet's enterprise mirror (a Squid proxy). Definitely not until the tunnel is operational, and maybe not even after that. This means jiggering audit-repos to know about Surya and omit the proxy parameter for it.
Installation of Tumbleweed on Surya:
Install local software needed to activate the firewall.
Firewall fixes to make it work on Surya. I've had this firewall since 2003 (with lots of modifications since then), and there were earlier incarnations.
Port scan again, are we locked up tight?
Backdoor SSH server on nonstandard port.
Learn to use Lish. On the remote access page, you can launch these items:
Lish can do more than console access. You can give it a subset
or superset of virsh commands such as shutdown or reboot;
is the equivalent of
Investigate package mtr, a hopped up traceroute.
Create the tunnel from Jacinth to Surya.