Valid HTML 4.01 Transitional

Network for Xena's Virtual Machine

James F. Carter <jimc@jfcarter.net>, 2018-07-05

I have a laptop (called Xena) with a Wi-Fi network connection, which hosts a virtual machine (called Petra) using libvirtd, qemu and KVM; a second VM is possible. The normal way (for me) to put a VM on the net is to create a bridge on the host, which becomes the host's main NIC, with the wired Ethernet NIC as a member, as well as a virtual network connection from each of the VMs. This does not work when the egress port is Wi-Fi.

Wi-Fi emulation of Ethernet is not complete, and in particular, there are corner cases which don't work. Therefore the Wi-Fi developers have flatly forbidden membership of a Wi-Fi NIC in a bridge or equivalent constructs. Specific problem areas are:

So I'm going to have to put together another network solution for the VM. Here are my requirements, in approximate order of importance.

Xena as a Router

I tried several networking schemes without success, detailed below. Now I'm reverting to an earlier concept: Xena as a router.

KVM creates on the VM host a network interface called vnet0 which bridges packets to or from the guest's eth0. In a simple setup the number 0 is fairly consistent, but it can vary randomly at boot time, and you need a setup procedure that works with any interface name.

KVM and libvirt have several styles to give network access to the guests. One is bridged networking; this is what I normally use on non-wireless VM hosts. It actually basically works without a bridge, but libvirt is going to be a lot happier if you create the bridge that your VM XML file says that vnet0 should be put into. Before starting the guest.

brctl is deprecated and its facilities have been added to the ip command from the iproute2 package.

NetworkManager has a connection type of bridge and in the GUI you can configure the bridge to be created at boot. Gotcha: if NetworkManager gets restarted, vnet0 will be evicted from the bridge and will not be reinstated automatically; you will need to add it by hand using one of the above commands.

Normal networking is designed to work with subnets, not individual hosts. I was able to get networking mostly working with an independent vnet0, not in a bridge, but the results are a lot cleaner if I create the bridge.

At present, CouchNet has these subnets. The default gateway for all of them is Jacinth, except as noted.

To this collection I'm going to add:

Do this to activate forwarding. The equivalents of these commands could go in /etc/sysctl.conf or /etc/sysctl.d/01-whatever.conf . When changed they reset most parameters to their defaults, so they should be executed early, and sysctl.d fragments are executed in lexical order.

The bridge on the VM host (or the naked vnet0 device) needs these addresses and routes. Although it's legal to use the same IP address for both the bridge and the egress interface (wlan0), I have found that fewer strange things happen and it's less confusing if the bridge has its own IP address (called xenavm). IPv4 is shown but these need to be duplicated for IPv6.

Xena, my VM host, uses NetworkManager, and I created a NM connection which creates the bridge automatically with these parameters as fixed IP addresses. One minor detail, I want to use a fixed MAC address for the bridge, because the guest's firewall requires that its peer's MAC be registered as trusted. Also I want to be able to recognize bridge traffic when seeing tcpdump output or error messages that contain the EUI-64. As a local convention, for my VMs I use 52:54:0: which is the assigned OUI (MAC range) for KVM, followed by the last 3 octets of the interface's fixed IPv4 address. I'm using the same convention for the bridge. The NetworkManager GUI for Edit Connections does not have a text box for overriding the MAC address. But see this blog post about making NetworkManager set a fixed or random MAC address by Thomas Haller (2016-08-26). He shows how to use nmcli to modify the connection file (/etc/NetworkManager/system-connections/br0, on Xena), as well as how to display the available parameters and the connection name. For example:

nmcli connection modify br0 bridge.mac-address 52:54:00:09:c8:a9

All other hosts on the local LAN need a route via the VM host to the bridge's subnet:

The guest needs theese addresses and routes. The goal here is to have this machine be as normal as possible. One component of being normal is to have the address and routes appear by DHCP and/or IPv6 Router Discovery.

At this point, bidirectional communication is achieved between the guest and these peers: VM host, LAN gateway, other LAN neighbor, offsite peer. This is on both IPv4 and IPv6. (The IPv6 commands are analogous but are not shown.)

Normal networking does not require a bunch of manual commands every time you start up your VM. I'm setting up dnsmasq and radvd to provide the required address and routes by DHCPv4, DHCPv6 and IPv6 Router Discovery. Global issues about dnsmasq:

Features for /etc/dnsmasq.d/*.conf:

The hosts on the local LAN need to be told to send traffic for the VM guest (Petra) via the VM host (Xena). Dnsmasq on the local LAN's gateway sends out this route for IPv4, and I'm using radvd on the VM host to send it for IPv6. Key features in /etc/radvd.conf:

Additional flies in the ointment that had to be fixed:

With those miscellaneous fixes, Petra can configure its network autonomously at boot, and can make and accept the connections listed in the requirements. So this project has come to a successful conclusion, if we're not too picky about various kludges.

Pseudo-Prerequisite: systemd-resolved

Avahi-daemon gives endless trouble, losing its addresses for no obvious reason. I've thought about just not having a mDNS service. But systemd-resolved is a new entry in this area and I'm going to try to get it to work before giving up the whole service. Here's what systemd-resolved does. Most of these sub-services can be turned on or off in the configuration file, /etc/systemd/resolved.conf .

In conclusion, I did not succeed in my main goal of replacing Avahi with systemd-resolved. However, I think I have made progress in cleaning up an area of my network infrastructure that was making a lot of trouble on my VM, that I didn't want to deal with while debugging VM networking.

Alternative Methods that Didn't Work

Proxy ARP

One possible solution involves proxy ARP. (2009-06-24, OP bodhi zazen.) Remember that ARP (specifically proxy ARP) is only defined for IPv4. Here's a summary of his howto:

It looks like the proxy ARP method won't fly because IPv6 is impossible.

Tunnels of Various Kinds

Another family of solutions involves tunnels. The basic design would go like this:

What kind of tunnel might I want to use? These are some key aspects of the tunnel:

I require authentication, so only the authorized laptop can connect to the gateway's endpoint and so it can be sure that the intended endpoint is being connected to. I'm providing for the guest the same privacy and integrity that the host gets, i.e. none. However, some protocols like SSH cannot turn off those features, and I won't reject them just for that reason.

Here is a promiscuous list of varieties of tunnel, with evaluations.

SSH

While normally it is used as a point to point link to a single client session, it can be switched to generic tunneling at layer 2 or 3 (link or network, i.e. IP packets).

I'm familiar with SSH tunneling, it is widely trusted including by me, and I already have the authentication infrastructure (public and private keys) in place. But you can't turn off encryption. SSH remains one of the front runners. But beware of TCP meltdown!

OpenSuSE Tumbleweed is using package openssh-7.7p1 at the time of writing.

SSTP (Secure Socket Tunneling Protocol)

(Not to be confused with Simple Symmetric Transport Protocol.) It runs PPP (point to point protocol) over TLS (Transport Layer Security). Authentication is required for both layers.

Assuming PPP can be made to defer to TLS for authentication, this looks like a possible winner. I already have the X.509 certificates needed for authentication. But beware of TCP meltdown!

There is a package NetworkManager-sstp for OpenSuSE Tumbleweed, community contributed (several instances). Except it has a missing dependency (libnm-gtk.so.0()(64bit)) that I can't find.

OpenVPN

On CouchNet, it is normally used at the network layer (3), but it can be switched to the link layer (2).

I'm familiar with OpenVPN and use it regularly on the host. That's both good and bad: it would have to continue to run at the network layer. I'm a little worried about committing to have OpenVPN running on the host at all times. A more serious complaint is that I still need a tunnel protocol for the guest: OpenVPN provides a tunnel from the host's whole network stack to the local LAN, but I need part of that stack to be a tunnel that carries the guest's traffic to the gateway, and OpenVPN can carry the tunnel, but would have trouble to be the tunnel itself at the same time.

Another possibility is an OpenVPN tunnel from the guest itself to the gateway, but this would mean that I could not test or develop generic networking on the guest.

IPSec

It does privacy and integrity, and to establish the Security Association the peers need to mutually authenticate. In IPv6 IPSec is implemented just as another packet header identifying the Security Association and giving the Message Integrity Code (HMAC); all that follows is encrypted, and as part of removing and obeying the header the kernel decrypts subsequent headers and the payload. But IPv4 headers aren't so flexible, and a separate IP protocol (ESP and/or AH) is used. In Linux the payload is considered to be received on the same interface as the bearer was, so for routing purposes IPSec isn't really a tunnel.

PPP: Point to Point Protocol (RFC 1661)

Wikipedia article about PPP. It's a link layer (2) protocol designed to work over alien links including ATM, in addition to IP. PPP includes (or could include) authentication, privacy and integrity (just a CRC, not cryptographic). Also compression. It is very modular and includes setup modules (Network Control Protocols) for most known protocol families.

Generally PPP is not an independent tunnel but is used as the stuffing for another tunnel protocol.

gre, ip6gre

Wikipedia article about GRE, q.v. for relevant RFCs. It can do encryption using RC4 which is deprecated. Can do integrity, but it's not too clear how cryptographically robust the checksums are. PPTP uses slightly modified GRE packet headers. GRE was developed by Cisco and they have appliances that use it. GRE is popular in Windows shops.

Given the questionable security and the hassle of PPP/PPTP authentication, I'm not going to waste time trying to set up GRE.

L2TP

Wikipedia article about L2TP. It's a hybrid of Cisco's L2F and Microsoft's PPTP. It encapsulates PPP (point to point protocol), but L2TPv3 can bear other link-level protocols. The outermost bearer packets use UDP. L2TP doesn't do authentication, privacy or integrity, but prepended IPSec headers can do so, and the interior protocols also can do so.

This protocol looks viable, but probably it has various frusrating issues which will turn up when I try to actually implement it. I will investigate it if none of the front runners pan out.

VXLAN

Wikipedia article about VXLAN (Virtual Extensible LAN), RFC 7348. Its goals are being scalable to large cloud nets. It has the equivalent of VLANs. A lot of vendors and software support it. Open vSwitch is one of these. Its main target is cloud isolation within a multi-tenant datacenter. Bearer packets use UDP. They contain the entire (almost) Ethernet frame that the guest would have sent on a wired connection.

It would be a big commitment to learn how to make this work. I don't see a whole lot of support for authentication. I doubt I will be using this one.

ipip, ip6ip6, ipip6

Encapsulates IP packets; does not handle non-IP packets.

The lack of authentication makes this protocol unsuitable.

isatap and SIT

Two IPv6 transition mechanisms that transmit IPv6 payload packets over a IPv4 network. No IPv6 bearer packets.

I need to handle mixed IPv4 and IPv6 traffic and I already have IPv6 payload capability; these transition solutions will not be helpful.

vti, vti6: Virtual Tunnel Interface

I wasn't able to find much information about this mode.

So the front runners are SSH and SSTP. I think I'm going to try SSH first,

Tunnel Details

One way to create the tunnel is by

ip tunnel add NAME mode any? remote ADDR local petraguest pmtudisc dev br0

However, this doesn't include SSTP and so this approach is useless.

There are a lot of companies that publish setup guides for SSTP. The idea apparently is, your client establishes a SSTP tunnel to their server, which is not free, and the result is a VPN. According to the ExpressVPN docs, SSTP is owned directly by Microsoft and is available for Windows only, which doesn't exactly match with jimc's experience.

On Github there exists sstp-server by sorz (Shell Chen). A package of it apparently is available on Arch Linux but not OpenSuSE. pppd is a prerequisite.

I think that I'm going to follow my original plan and try ssh first.

ssh -w any -o Tunnel=ethernet

It also needs in the server's /etc/ssh/sshd_config:

The infrastructure manager will want to create a tun/tap device and put it in the relevant bridge. Here's a tutorial on doing this by waldner (2010-03-26). He has discovered an undocumented feature of iproute2 (the ip command); do ip tuntap help for a usage summary.

ip tuntap [add|del|show|list|help] mode [tun|tap] [user U] [group G] [name itsname]
ip tuntap add mode tap name tap8

Cutting off the tunnel idea. I got a lot of it working, but one issue killed it: Xena communicates over Wi-Fi and the normal channel ends in Jacinth's br0. Xena also creates a tunnel, whose endpoints are in Xena br0 and Jacinth br0. This creates a loop. Various maneuvers were used to keep Petra traffic in the tunnel and Xena traffc on Wi-Fi (including tunnel bearer packets), including turning on the Spanning Tree Protocol on one or the other bridge, but they were not effective enough, or were too effective, killing transport between various endpoints.