| Selection | Setup (Jac / Iris) | Testing | Software (Jac / Iris) | Top |
Since this machine is our home server and hosts a lot of mission critical services, the emphasis was to get it checked out fast, and to move existing services unchanged as nearly as possible. New material will be deferred until the machine is fully operational.
The existing server, which is not broken, is a Koolu called Jacinth. The new machine is called Orion. When the new machine is ready to go into production, these identities will be swapped. I reserve the name Orion for new machines being set up.
SYSTEMD_NO_WRAP=1 /etc/init.d/firewall.J dump -- It appears to be complete, and it does let in legitimate other hosts. From off site only port 1443/tcp is open, 25, 587, 1723 (pptp) admit that they exist but are closed; SMTP is broken at the moment and pptp is transit for Ben's laptop. See also ooba, which works.
Copying over: cronj.jacinth.conf cron.d cron.daily . /var/lib/cronj is populated and /var/lib/cronj/activity shows the expected cron jobs.
Orion can send out print jobs, but the media size is wrong (looks like A4) with this command line: gpgview -p pws.gpg and the same with setenv LC_PAPER Letter. This magically fixed itself later.
It's running; hard to do a functional test.
The session gets a Kerberos ticket at login, which is available in all xterms.
Copied /etc/named.conf /var/lib/named .
It delivers (to dig
) the SOA for cft.ca.us
and outside domains. Running as Jacinth, it has to be using the off-site
forwarders from Verizon.
Being configured with /etc/ntp.master it is synced with Verizon's NTP server.
I have an ongoing project to set up authenticated NTP, but haven't made much progress.
Copied /etc/rsyslog.conf . Requires manual merging of /etc/sysconfig/syslog [done].
The SYSLOGD_ADDITIONAL_SOCKET_${SERVICE} parameters, for importing into various chroot jails, are no longer present. Make sure the chrooted services can write log messages. Services: named, ntp, dhcp. Confirmed that named, dhcpd and ntp are logging.
This is the management daemon for the APC (American Power Conversion) UPS (Uninterruptible Power Supply). Copied /etc/apcupsd /etc/sysconfig/apcupsd s1/upgrade/jacinth.etc/sysconfig/shutdown . Functional test has to wait until the machine is connected to the UPS. Functional test: apcaccess status ; it reports UPS info.
I did a complete test by unplugging power. The UPS beeps every 30 secs (can't shut it up). As configured, the system runs for 30 minutes on battery: CPU, powered USB hub and its clients including wild side network, 2 MOCA bridges. The only thing not powered is the monitor. After 30 mins, apcupsd sets the UPS for a delayed shutdown, and sends the OS into a voluntary power-off action. 2 mins later, the UPS shuts off power. When power returns, the system boots up.
In theory, with a new battery, this UPS can provide 50 watts for 30 mins. The daemon monitors the battery voltage and when it declines indicating about 5 minutes of runtime left, if this happens before the configured 30 mins, it will immediately start the shutdown sequence.
Need to set up the network bridge [done].
This script turns on bridge multicasting. It works: functional test is
that slpd also works (q.v.) But presently, Jacinth sees only its own
daemon, indicating a problem. [Fixed
]
Copied /etc/dhclient.conf Need to manually reconcile /etc/sysconfig/network/dhcp [done]. Can't test it until we're connected to the wild side. Once various network problems were straightened out, dhclient did its job.
Copied /etc/ddclient.conf /etc/init.d/dhclient-hooks . Can't test it until we're connected to the wild side. It did successfully register the new IP address with both dyn.com and Hurricane Electric.
Copied /etc/init.d/postgresql . This took some work to get running.
This does traffic shaping so when a big download is going on, other streams like audio or video get the access they need. Copied /etc/init.d/netpolice.J . Can't test it until we're connected to the wild side. It claims to be running; a functional test is a little bit hard. Later I took the time to re-do speed tests; netpolice.J is doing its job to limit the data rate.
Copied /etc/init.d/network6 ; requires manual editing of /etc/udev/rules.d/70-persistent-net.rules [done]. Can't test it until we're connected to the wild side. Once the bridge was unscrambled, it set up IPv6 successfully.
Wait for default route and DNS: The script times out. It does the same thing on old Jacinth. This needs to be worked on. [Fixed, see below.]
Copied /etc/init.d/radvd /etc/sysconfig/radvd . It starts even though IPv6 forwarding is turned off. It should be safe if it has no routes to anywhere. Can't really test it until we're connected to the wild side. Yes, it is passing out routes.
Useless, no hardware random number generator on this machine. The Koolu (AMD Geode) had one.
Copied /etc/apache2 /etc/sysconfig/apache2 . Functions served through Apache:
system error while authenticating, cannot issue certificate.[Fixed]
Apache problems:
Copied /etc/dhcpd.conf /etc/dhcp.README Need to manually merge /etc/sysconfig/dhcpd [done]. Did not start on boot (missing conf file?) [fixed]. Started manually, it's listening on port 67/udp. Confirmed that it is passing out IP addresses.
Oopsie! You're allowed to say "fixed-address FQDN;" but if it happens to resolve to the IPv6 address (as it now does on CouchNet) it will be silently ignored, and all the clients will get aleatory addresses. Use the IPv4 address explicitly. This behavior is new (since when?) /etc/dhcpd.conf has been fixed.
Copied /etc/dovecot . Running, but rejects authentication from SquirrelMail (Jacinth did this too). This is not Dovecot's fault; SquirrelMail is trying to use the master user with the real user's password.
Copied /etc/hostapd.conf /etc/init.d/hostapd . OK, client can connect and communicate. This is using the TP-Link TL-WN722N (Atheros chipset) normally on Jacinth. Later I want to try using the internal wireless, but that has to wait until I get a driver for it. Later: hostapd can use the internal wireless and associates with clients. But I reverted to the TP-Link TL-WN722N because it has a better external antenna.
Copied /etc/darkice.cfg /etc/icecast.xml . I tried a MP3 transcoded stream, but it did not materialize. [Fixed, see below.]
Copied /etc/openldap /var/lib/ldap . Did not start on boot. Did not start after boot either. The wrong hostname is going to really freak it out. I ended up copying Diamond's /var/lib/ldap lock stock and barrel, and that brought it back to life.
Copied /var/lib/kerberos . Can't test until LDAP is working. Yes, it works now.
Copied /etc/init.d/ooba (is this actually obsolete?) and /etc/sysconfig/ooba . It appears to be working, although the browser complains that the wrong host cert was presented. Need to re-test once the host is using the Jacinth host cert. Accessed as Jacinth the browser is happy and can do X.509 auth. An authorized wild-side host sees the expected ports open (tested TCP only): ftp ssh http imap imaps ldap ldapssl https ooba(1443); these are active but closed: smtp submission kerberos-sec(88) xmpp-client xmpp-server finger(2003, really Squid). Except for squid which is vaporware, these are all for services that ought to be available but are broken at the moment. An unauthorized host sees only ooba, submission (587 for SMTP), pptp, which are the expected open ports.
It accepted a message but doesn't know how to send it out. Copied /etc/postfix/main.cf and master.cf ; now it can send (and receive) mail.
This is the XMPP instant message server. Copied /etc/prosody/ /var/lib/prosody . Does not start. See ~jimc/misc/homeserv/prosody-1304.txt for hacks needed to make it work. These hacks were ineffective. After considerable work I replaced Prosody with DJabberd, which now works. See the separate writeup about Replacing My XMPP Server.
Copied /etc/samba/ . Working, you can mount //orion/jimc on Jacinth.
Is sound working? Yes, I got some speakers for Jacinth and can play media.
Copied /etc/sysconfig/fax . Did not start on boot. I think it also won't start on Jacinth. The FAX modem is connected to neither Jacinth nor Orion, and is going to be moved to Diamond (eventually).
Copied /etc/openvpn . Not going to test this until operational due to certificate problems. Partners do not seem to be able to connect, which they could do on the old machine. [Configuration problem, now fixed.]
Copied /etc/init.d/sogod /etc/sysconfig/SOGo . This badly needs an upgrade and some infrastructure (gdomap, memcached) is either not installed or not available on Orion. SOGo unfortunately has been fired and replaced with ownCloud.
Responds and gives a functioning X session.
I'm in the process of getting these working on Jacinth. This effort will be transferred to Orion, but they are not expected to be fully operational.
Running. Functional test: ping -w 3 orion.local, which works.
Running. It provides a d-bus protocol; I'm not sure how to do a functional test.
Running. Functional test, do this on another host: slptool findsrvs service:service-agent ; done from Iris, it gets answers from Iris, Diamond, Jacinth. Done on Jacinth it gets an answer only from Jacinth. Whatever the issue was, it's been fixed now.
Copied /etc/ipsec.conf /etc/strongswan.conf . It is not yet set to start on boot [now it is], but if started manually, it runs. On Jacinth, the Android client can connect to it and route packets.
Running. Functional test not done. There is a driver for the Bluetooth NIC, which is integrated with the wireless NIC, for which there is no driver. Update: a Bluetooth mouse is functional with Jacinth using the internal NIC.
There is no driver for the
internal wireless (in the standard kernel, 3.7.10-1.16-desktop).
This appears to be a Realtek Device 8723
. This is possibly the
Realtek RTL8723AE-BT and a lot of people have trouble with drivers for it.
askubuntu.com, OP ugopozo 2012-05-20. He tried ndiswrapper (failed). He says the driver for RTL8192CE/SE/DE mentions the 8723 in the sources, but he tried various maneuvers and could not get the driver to work with the card. PCI product id is 10ec:8723 and indeed this is the card I have.
izx replies that Realtek's 92-series driver version 0006.0514.2012 will handle this card. But it's being distributed through a dropbox link, see the post for MD5sum and build instructions. But it only compiles for kernels up to 3.3. There's another link for up to 3.5.
Keep an eye on
Realtek's web site for the real
driver. As of 2013-07-21
I can't find any reference to the RTL8723AE-BT (or variations) on the
website.
The required firmware is available already from kernel-firmware-20130114git-1.2.1.noarch . Filenames: /lib/firmware/rtlwifi/rtl8723fw.bin and rtl8723fw_B.bin .
As suggested in the post, I'm writing to Realtek and begging for the (latest) sources, rather than getting the file from an unknown source. It appears that the contact address for wireless NICs is wlanfae@realtek.com . Mail sent on 2013-07-21. No (or spammy?) reply.
Update: with SuSE 13.1 and kernel 3.11.10 there is a driver and hostapd can use the internal wireless. However, I reverted to the TP-Link TL-WN722N because it has a better external antenna.
These should be investigated and identified:
bluetooth, rfcomm, bnep, btusb -- Loaded, find out what's using them. This is 0bda:8723 Realtek Semiconductor Corp. Bluetooth radio on USB. This is the device for which I didn't have a WLAN driver.
sp5100_tco -- AMD chipset watchdog timer.
r8169 -- PCI bus 0000:02:00.0, Realtek RTL8111/8168 GigE 802.3 NIC.
Do we need to load on boot nf_conntrack_sane (in /etc/sysconfig/kernel )?
These are fixed:
Wait for default route and DNS: times out. Jacinth fails similarly. Fix it on Orion which isn't mission critical yet. Nothing wrong with the script; Orion really doesn't have a default IPv6 route since it expects to set it up through Hurricane Electric, which it can't do until operational. Problem on Jacinth may have been that named (DNS) doesn't start until after LDAP which depends on network-wait which waits for DNS to be ready. No, it was waiting for the IPv6 default route.
The problem was that LSB scripts, such as /etc/init.d/network6, that depend on network or $network are mapped by systemd to depend on network.target, not network.service. These scripts are affected: bridge.J named netpolice network6. I created a proxy "service" called network-J (since I think LSB scripts normally can't depend on a target) which depends on network.service, and changed the affected scripts to depend on that, so they can start immediately after network.service finishes. Now on reboot, network-wait blows right through with 0 secs of waiting.
Why is xinetd installed? Should /etc/xinetd be deleted completely?
If not, manual merging will be needed. It is required
by
tightvnc. Leave it just as it was installed, but don't enable it.
LDAP is totally horked. Need to track this down. Probably best to clear the database and propagate everything from Diamond.
The LDAP server can present one host certificate. Use the local LAN hostname or the wild side, according to which cert is used; don't try to use both. I'm using the local net's hostname and cert on all servers including Jacinth.
LDAP can do: ldapsearch -x -D cn=config -y /etc/openldap/root.secret -H ldaps://jacinth.cft.ca.us/ -b cn=config (lots of output).
LDAP can do: ldapsearch -x -H ldaps://jacinth.cft.ca.us/ -b uid=jimc,ou=People,dc=cft,dc=ca,dc=us -LLL gecos
So what's the problem? ldapsearch -H ldaps://jacinth.cft.ca.us -x -D uid=ldaproot,dc=cft,dc=ca,dc=us -y /etc/ldap.secret -LLL '(uid=jimc)' userPassword returns nothing. Stop after -LLL and everything is dumped including the targeted record. The command does work on Diamond. This suggests that the indices are horked, e.g. i586 vs. x86_64 issue.
Forum posts suggest to shut off slapd and use slapindex. Command line:
slapindex -n $N [-v]
See /var/lib/ldap/slapd.d/cn\=config olcDatabase entries for the numbers {$N} and do your HDB database(s) last since they're biggest. For me, N = 1. Database 0 (configuration) and 2 (monitor) cannot be indexed so skip them. Reindexing appeared to be successful but did not improve behavior.
Trying to just copy everything from Diamond: That seems to have done it!
A lot of services have authentication problems. A possible reason is that /usr/lib/sasl2/*.conf were all absent. One reason they are absent is that they belong in /etc/sasl2 (but are still missing). Looking in /usr/lib64/libsasl2.so.2, apparently /usr/lib64/sasl2/*.conf would also be honored for backward compatibility. Affected services:
SquirrelMail IMAP auth (Dovecot). The main config file was not getting backed up: /srv/www/htdocs/squirrelmail/config/config.php . If we ever lose the password for the SquirrelMail user, I know how to set a new one for Dovecot but not how to tell SquirrelMail what it is. Found it: ./plugins/login_auth/config.php (mode 600).
Now the issue is that IMAP (Dovecot) auth always fails. That is, you get to the SquirrelMail login page, give your loginID and password, and it is rejected. It also failed on the old machine. The farthest I've gotten so far is that it tries to log in as squirrelmail (maybe with the user's password). I think I'm going to punt on this one.
SquirrelMail LDAP auth. Now that LDAP is fixed and SquirrelMail is actually configured, content is delivered.
SquirrelMail Kerberos auth. Now that Firefox is configured to allow SPNEGO, this gets me on.
Certificate authority. Needed /etc/sasl2/cgi-helper.conf
Prosody. Still working on this.
How to set up Firefox for Kerberos authentication:
negotiate.
Need to test the Certificate Authority after transition to be Jacinth. Authentication problem [fixed].
Straighten out missing directories for SquirrelMail authentication by Kerberos and LDAP [done]. Straighten out authentication probs [fixed].
Postfix is not sending out mail. main.cf and master.cf didn't get propagated. Now it's fixed. There are still complaints about one or more unused parameters; check this out. Unused parameter was removed (on all hosts).
prosody.J (XMPP server) won't start. Fixed, now it won't authenticate. /usr/lib/sasl2/*.conf were all absent, should be in /etc/sasl2. Replaced with Tigase. Tigase had problems too. Replaced with DJabberd, which is working (so far). See the separate writeup about Replacing My XMPP Server.
SquirrelMail's login page cannot successfully authenticate to Dovecot. It may or may not be authenticating as the master user but with the wrong password (?) It failed on the old machine too. It would appear that this is a bug/feature of SquirrelMail. Not going to be fixed. [Update: jimc fixed the bug and submitted a report. Login page works now.]
icecast -- Can't produce a MP3 stream. The problem was that /usr/local/bin/ezstream is a 32bit program. It needs libshout3-32bit, and installing it brought streaming media back to life. It turns out that icecast and ezstream are now available on the SuSE Build Service. I upgraded to the current version for x86_64 and can now send out Icecast streams.
/etc/ethers is wrong for Orion vs. Jacinth. There's an entry for 88:b2 labelled Jacinth and for 9b:ce labelled Orion. In addition to entries earlier in the file labelled the right way. [Fixed.]
nmap says jacinth has a slew of open ports that the firewall has
not blocked. 3551 apcupsd, 7777 openfire, 8000 icecast (should be
open?), 9090/tcp openfire. They are blocked on IPv6 but you can telnet
to them on IPv4 and get connected. Both the local and wild IPv4
addresses. The reason is that $myip is being set to
jacinth-w.cft.ca.us = 192.168.1.48 . When I reload the firewall, it's
made right and you can no longer telnet to the unwanted ports and nmap
doesn't see them either. The reason may have been this: The firewall
has to start before the network, so does name to address translation
by parsing /etc/hosts by cowboy programming. For some unknown reason,
/etc/hosts had jacinth-w.cft.ca.us before jacinth.cft.ca.us, and grep
for the 1-component hostname found jacinth-w. Later I sorted /etc/hosts
with jacinth before jacinth-w, so when I reloaded the firewall the
correct address was used. I also changed the regexp to require
a dot after jacinth
so it would not have failed with the original
/etc/hosts.
bridge.J -- slptool findsrvs service:service-agent
on
Jacinth used to get answers only from Jacinth (wrong). I didn't do
anything for this; whatever the issue was, it is now fixed; Jacinth
gets responses from all active hosts.
OpenVPN -- Mica could not connect; I think Mica is properly configured. No it wasn't, but that was fixed and it can communicate with Jacinth's Apache through OpenVPN. Now the issue is, NAT is needed on Jacinth so replies to tunneled packets can be returned to the client. Also IPv6 needs to be tested. OpenVPN can't use IPv6 for the tunnel but it can transport IPv6 through the tunnel. But only in point-to-point mode, hiss, boo, so for me IPv6 is useless via OpenVPN.
OpenVPN is now able to accept connections from a (properly configured) Android client, and route them to the global Internet. Test configuration: Android tablet (Mica) with OpenVPN, to Android phone (Selen) with WiFi access point, to carrier's cellular data (wild side), to Jacinth over FIOS, to OpenVPN, payload to Jacinth packet bus, NAT on Jacinth, to FIOS, to remote server. The path has been verified by traceroute. This is IPv4 only -- OpenVPN in server mode can't handle IPv6 payloads. (It can in point-to-point mode.)
StrongSwan -- I haven't had an IPsec service for quite a while; now I've set this up. See the separate writeup about StrongSwan.
Cups paper size is wrong. A PostScript document produced by enscript looks like A4, tops and bottoms cut off, despite setting the size to Letter. Whatever the issue was, the paper size is now Letter, correct.
One last item: Jacinth's host certificate is expiring soon and needs to be renewed.
First is to renew my personal certificate on Startcom. The procedure is hidden in my writeup on installing Tigase. For the personal cert they did not ask for a CSR; they induce Firefox or MSIE to create the CSR itself. The cert ends up installed in Firefox.
To export it, go to: Edit-Preferences-Advanced-Encryption- Your Certs (select the cert) - Backup. It asks for a filename to write in (u-jimc-startcom-2014.p12) and for a password to encrypt the key.
I am going to get a XMPP
cert, which will have SANs
for both the domain and for Jacinth itself. This is the same
as I did for the XMPP server except there I dumbly specified a
host CNAME of XMPP
. [Update: inverse of dumb. If you use
a XMPP cert on a webserver, the client checks OCSP for the name
it requested and the CA has it filed under the Common Name, which
is the name of the domain. Oops.]
When creating the CSR specify the usual Distinguished Name, which Startcom will proceed to ignore, replacing with some kind of encoded description, CN = requested FQDN, and emailAddress = your address on file.
Fly in ointment: with the XMPP cert, when Firefox connects via HTTPS, Startcom's OCSP server says sec_error_ocsp_unknown_cert. Startcom revocation requested on 2013-08-24 for cert ID 1429877.
OK, now I have the correct renewed webserver cert. It's installed and outside clients (authorized by OOBA) can connect to it with no complaints about trust or expiration.
These are still being worked on.
The session's default xterm has a (real) GPG agent, but another xterm started from the XFCE main menu has GPG served from Gnome Keyring Agent, and of course they do not share keys.
| Selection | Setup (Jac / Iris) | Testing | Software (Jac / Iris) | Top |