It's time once again to make sure the VPNs are working. This is a log of what I did, but organized (I hope) so it can be re-played for the next audit.
Lollipop.
OpenVPN Connectversion 1.1.16 (net.openvpn.openvpn).
strongSwan VPN Clientversion 1.5.0 (org.strongswan.android).
Tethering & Portable Hotspotservice, believed to be the actual hostapd, version not known. This will be turned on (when needed) so the other machines can get access to the wild side.
KitKat.
Out of Band Authentication, but not IPSec (ISAKMP, ESP, AH) or OpenVPN.
The first step will be to verify that all hosts have the needed certificates. The ones for this year are:
Firefox functioning is not strictly a VPN issue, but it's an annoyance when Firefox messes up, and Firefox is often used over the VPN, and for efficient troubleshooting I don't want the VPN to be blamed for a Firefox screwup. Firefox does not use the host's trust store (called the Android Keychain); certs must be loaded into its collection separately; see the appendix for the very user-unfriendly procedure on Android.
On Android the host trust store is called the Android Keychain
.
All of these tests are done via the internal WiFi network.
| Selen | Mica | Xena | Description |
|---|---|---|---|
| S.OK | M.OK | X.OK | Can Firefox use the CFT root cert when doing HTTPS to an internal host whose cert is signed by that CA? Test by https://diamond.cft.ca.us/~certauth/ |
| S.OK | M.OK | X.OK | Can Firefox present Jimc's user certificate? Test by https://jfcarter.net:1446/roundcube/ |
| S.OK | M.OK | X.OK | Does the host trust store
have the CFT root CA certificate? On Xena, test by
openssl s_client -connect diamond.cft.ca.us:443 -CApath /etc/ssl/certs < /dev/null >& $j/errs On Android, use the Hurricane Electric app (SSL checker). |
| -- | -- | X.OK | Can OpenSSL present Jimc's user
cert (Xena only)?
openssl s_client -connect jacinth.jfcarter.net:1446 -CApath /etc/ssl/certs -cert ~jimc/certs/u-jimc-cft-R2016.usr -key ~jimc/certs/u-jimc-cft-R2016.key |& tee $j/errs Hurricane Electric app can't present the user cert. |
Jacinth has two OpenVPN services: the normal port 1194/udp, and also 443/tcp. Hotel WiFi services often block all ports except HTTP (80/tcp) and HTTPS (443/tcp), and some countries block VPN ports at the national level. By stealing 443/tcp I work around stupidity and/or hostility. Beware, if a national security service takes an interest in a connection on 443/tcp, they may notice that packet timings and sizes are not typical of HTTPS, and that the peer in the civilized world is not a webserver.
IPv6 has multiple issues and so I seriously tested only IPv4 over the VPNs. This is signified by an outcome of OK4. The servers are supposed to accept transport streams on IPv6, but making clients use IPv6 from the wild side is another story: IPv6 transport streams were not tested.
In the initial round of testing, DNS name resolution on Android was a mess, probably because the DNS Forwarder app was fixated on the internal or the external DNS server and hostnames in the opposite realm could not be resolved. However, after a reboot and with the DNS Forwarder app unused, OpenVPN could use the carrier's DNS servers to resolve the VPN server's address, and could then use the DNS server sent out by the VPN server; apps could resolve hostnames both on the wild side and on the internal net. IPSec was another story; see below.
| Selen | Mica | Xena | Description |
|---|---|---|---|
| S.OK | M.OK | X.OK | Can OpenVPN on port 1194 connect from the wild side? |
| S.OK | M.OK | X.OK | Does DNS resolve names, specifically diamond.cft.ca.us? Internal and external names are resolved and connected to, showing that Jacinth is being used as the DNS server, pushed by the server. But the Hurricane Electric DNS tester can't get records for the same names, hiss, boo! This is on Selen; I think it worked OK on Mica. |
| S.OK4 | M.OK4 | X.OK4 | Can the client ping diamond.cft.ca.us? The packet loss rate varies, sometimes 0% but sometimes near 100%, but all internal hosts can be pinged (IPv4). |
| S.OK4 | M.fail4 | X.OK4 | Can diamond ping the client? I can't tell why you couldn't ping4 Mica when you could connect by SSH4. Over-paranoia on KitKat? This happens on both 1194 and 443 and IPSec. |
| S.OK4 | M.OK4 | X.OK4 | Can the client connect with SSH to diamond? |
| S.OK4 | M.OK4 | X.OK4 | Can diamond connect with SSH to the client? |
| S.OK | M.OK | X.OK | Can OpenVPN on port 443 connect from the wild side? |
| S.OK4 | M.OK4 | X.OK4 | Repeated the above functional tests on OpenVPN port 443 with the same outcomes. IPv4 only. Jacinth one time could ping 2001:470:1f05:844::3:1000 which is the IPv6 address assigned to Mica by OpenVPN-443. Not to Selen. |
IPv6 outcomes: The Android clients do not send any packets even though the client and server configurations have IPv6 turned on. After an initial fixup, Diamond's ICMP6 ping packets are sent down the tunnel on Jacinth but there is no reply. TCP activities (TLS tester and SSH) similarly don't connect and time out. On Xena, the OpenVPN plugin for NetworkManager is incapable of specifying an IPv6 configuration to the daemon.
| Selen | Mica | Xena | Description |
|---|---|---|---|
| S.?? | M.OK | X.OK | Can StrongSwan connect from the wild side? On Selen: Adr lookup failed (for jfcarter.net). Turned on DNS forwarder to 8.8.4.4 (Google). Selen: Gateway unreachable (IKE[500] chattered, sends IKE_AUTH packet, peer does not respond, Jacinth does not log receipt of auth packet.) Selen connects over WiFi (??), switch to 4G, still works. I edited and saved the conf; now it can connect! Possibly a database schema issue after the upgrade to Lollipop and StrongSwan-1.5.0. |
| S.OK | M.OK | X.OK | Does DNS resolve names, specifically diamond.cft.ca.us?
OKwith a grain of salt. Initially only IP addresses can be used; it is still using the Verizon Wireless DNS servers, but the queries go down the tunnel and come out on Jacinth's wild side, and the VZW servers ignore non-VZW clients. If you turn on DNS Forwarder to Jacinth, then it works. However, the Hurricane Electric DNS tester can get records when nobody else can resolve names. |
| S.OK4 | M.OK4 | X.OK4 | Can the client ping diamond.cft.ca.us? |
| S.OK4 | M.fail4 | X.OK4 | Can diamond ping the client? Mica doesn't answer ping4 on any of the VPNs. |
| S.OK4 | M.OK4 | X.OK4 | Can the client connect with SSH to diamond? |
| S.OK4 | M.OK4 | X.OK4 | Can diamond connect with SSH to the client? |
IPv6 outcomes: When Selen pings Diamond, packets leave Selen, Diamond answers, routed to Jacinth, but the app on Selen does not get the packets. On Mica the packets are not transmitted. When Diamond pings Selen, the replies are produced but sometimes are delayed as long as 7 seconds; the delay varies from one try to the next and sometimes can be normal. When Diamond does SSH to Selen it gets connection refused: Selen's SSH server lacks IPv6 but transport happened.
Mica and Xena had no IPv6 transport at all. There is a vague forum posting
alleging that IPv6 dual stack is broken on Android-4.4 KitKat
(on Mica).
On Xena the issue appears to be that it needs the default next hop router to be
SelenAP (this is what's done for IPv4), and SelenAP does not offer IPv6
transport, nor does it announce its nonexistent IPv6 address that could be
put in the routing table. The same issue would preclude IPv6 on Mica.
For StrongSwan on Android, the server has the leftdns keyword and should make the DNS server addresses available, but the client has to request them by rightdns=%config4 and/or %config6. The client app likely is not doing this. The symptom is that public hostnames can be resolved but not internal hosts. You need to use the DNS Forwarder app to set the DNS server by hand to Jacinth.
These items got improved successfully:
Use the client key in the Android Keychain (OpenVPN and StrongSwan).
Master copy of the client configuration is available for: UDP/1194 or TCP/443
Several improvements to the OpenVPN server and client configuration files:
Turn on IPv6 on the server -- starting at or before OpenVPN-2.3.2 it's allowed at last. Involved keywords are: tun-ipv6 server-ipv6
If there are multiple clients, internally route between them: client-to-client keyword.
Peer authentication by tls-auth. Keywords: tls-auth with an inline key, and key-direction. This puts a special HMAC on every packet on the control channel; packets with the wrong HMAC are tossed without inspection. Otherwise an evil client could initiate lots of connections, using up server resources, which the server would have to provide until a late stage of the authentication process when the client would be required to send in a valid user cert (which it doesn't have). In other words, an easily deployed denial of service attack is repelled.
This is a little confusing in the docs: You need the same key
on the server and the client, but key-direction needs to be 0 on
one end and 1 on the other; informal standard is 0 on the server
and 1 on the client. The key-direction could be omitted or
declared to be bidirectional
, but that opens up an obscure
scenario where a man in the middle accomplishes something by
replaying back to the client packets that the client has sent.
For clients, the trust chain for the server cert is inserted in-line (keyword ca) rather than requiring a separate file to be installed on the client. Apparently it won't use the local trust store to verify the server cert.
The verify-x509-name keyword replaced tls-remote, which is
deprecated. Give the Common Name from the server's host cert, not
the alias from the remote
keyword. Syntax:
verify-x509-name jacinth.jfcarter.net name
Added the multihome keyword on the 1194/UDP server, and nobind on both clients. This makes the server send from the interface that the client is sending to; the client will drop the reply packets if from the wrong interface (having the wrong IP address).
IPv6 routes were redesigned:
| All | 2001:470:1f05:844::/108 | All the special subnets |
| Jacinth | 2001:470:1f05:844::0:0/112 | Jacinth's 2001:470:1f05:844::3 |
| IPSec | 2001:470:1f05:844::2:0/112 | |
| OpenVPN 443 | 2001:470:1f05:844::3:0/112 | |
| OpenVPN 1194 | 2001:470:1f05:844::4:0/112 |
In /etc/radvd.conf, a route was added to attract this traffic to Jacinth. But router advertisements are not sent to the special subnet because the EUI-64 is not well defined, and we want the VPN to manage routing in its own way. Unfortunately the clients ignore a variety of routes sent in this way. In a ServerFault.com forum post, OP Thomas Berger (2012-10-12) spotted in /usr/src/linux/Documentation/networking/ip-sysctl.txt that /proc/sys/net/ipv6/conf/all/accept_ra_rt_info_max_plen (sysctl path net.ipv6.conf.all.accept_ra_rt_info_max_plen) is 0 by default and needs to be raised to more than the number of bits in the prefix, 128 suggested. But for me, that was not enough to get it working.
So I added a static route to the VPN subnets on the clients, hiss, boo!
ip route add to 2001:470:1f05:844::/108 via 2001:470:1f05:844:201:c0ff:fe12:3044
Formerly, IPv6 routes on Jacinth were hosed: hosts on the local net including Jacinth thought they're going to reach VPN clients by transmitting on the local net.
In initial testing, Xena kpropd complained that it could not communicate with the master KDC. I'm not sure what I did, but later it connected as it should. Tested on IPSec which I think is where the original problem occurred.
Xena's firewall recognized IPSec packets correctly, but it was rejecting them if not on the internal net, which they never are. This was fixed.
On Android, get the unencrypted keys (no longer used) off the SD card, including out of the jail. [Done]
Items that aren't going to improve:
For StrongSwan on Android, the server has DNS server addresses available, but it looks like the app does not request them or, more likely, is not able to install them for Android to use.
In the Hurricane Electric network app, the DNS tester cannot get records when others (Firefox, H.E. pinger) can resolve and connect to the same hosts by name. This is for both internal and external hosts, showing that Jacinth was being used as the DNS server. Conversely, the DNS tester often can deliver records when the other apps cannot resolve the hostnames.
Cellular data switched frequently between EDGE, HSPA, HSPA+ and LTE. Whenever it did so, OpenVPN had to reconnect. Tests done while it was reconnecting of course failed. In at least one case the client was issued an IP address of 192.9.200.134, but had worked up to 138 by the end of the tests. Need to watch on StrongSwan to see if it resists this problem better. Yes it does; the Security Association persists even if the client's or server's IP address changes.
If you have OpenVPN running on Selen, nothing can connect to SelenAP. Not tested with StrongSwan.
Can we get the OpenVPN Access Server for free? Or reverse engineered? The idea is to use OpenVPN Connect's feature to import a conf file to Android. Price: $9.60 per client per year, 10 client minimum. This is for the complete package: server, management UI, client UI. Importing the conf file depends on a major generic scripting feature, which is not going to appear for free.
This works almost the same on Selen/Lollipop and Mica/KitKat.
download in progress. Wait 1 sec. for completion notification.
Don't mess with opening the PKCS#12 file in the Download app or with
ES File Explorer; the Certificate Installer app will maunder:
Couldn't install because the cert file couldn't be read.
Message is bogus; ES File Explorer can compute the checksum;
likely the problem is writing on the cert storage.
See jimc's writeup on this topic. Do this in Firefox:
add to Firefox.
On Selen (Lollipop), OpenVPN Settings by Friedrich Shauffelhut had trouble to connect. It did get SU permission from SuperSU, but no error messages in the log file. So I'm trying a different client. Which one?
Using OpenVPN on Android Lollipop by Francois Marier
(about 2015-04-05):
He's using OpenVPN for Android
(de.blinkt.openvpn) and it works for him.
He doesn't say if the key is encrypted.
Jeremy says, IPv6 dual stack works on Android-5.0; it's broken on 4.4.
Georg Sauthoff uses OpenVPN Connect
(net.openvpn.openvpn) (on Android-5)
and prefers it. His key is unencrypted.
From the product hype on Play Store for OpenVPN Connect:
More points about OpenVPN Connect (by jimc):
How to use OpenVPN Connect:
Importfrom the menu and select to-cft.ovpn .
When the connection fails,
ssl handshake failure.
These error messages are seen for the tested defects:
key values mismatch(doesn't even connect)
sslv3 alert certificate expired
tlsv1 alert unknown ca
In Lollipop (on Selen), Android has a built-in VPN client. (I didn't try this on KitKat (Mica).) I probably could have gotten it to work if I had changed the server's IPSec configuration, but I didn't want to go that far. Here's the procedure to set up a connection:
Edit VPN Profile. Set these values:
Typechoices rely on the user cert and not the loginID and password.
Tidbit: http://www.entropykey.co.uk/ sells a USB random number generator, GBP £36 on their store, not available on Amazon (US or UK).
Tidbit: 7 Security Measures to Protect Your Servers by Justin Ellingwood (2015-03-05). This tutorial, with a very cute title graphic, has a basic user-level or boss-level explanation of these security tactics: