Intel NUC11PAHi5
Setting Up Diamond

Contents

• In the Box
• Setup Outline
• BIOS Update
• BIOS Setup
• Partition the Disc
• Copy Diamond to Orion
• Boot New Disc, Checkout and Fix
• Subsystem Checkout and Testing
• Speed Measurements

In the Box

Check the box contents and record serial numbers. Try to get the Ethernet MAC address so it can be registered with the firewall. Recent (2017) NUCs have the MAC address and the serial number on a sticker on the bottom of the machine and also on the box.

What's in the box:

• The machine itself.
• VESA mounting plate with 75mm and 100mm squares.
• Package of screws: 4 to attach VESA plate to monitor, 2 to attach NUC to VESA plate, 4 smaller ones, M3x4mm pan head, for the disc drive.
• Small sticker saying Intel Core i5.
• Power supply; input 100-240V, up to 1.8A, 50-60Hz; output 19VDC, up to 6.32A 120W. Plug adapter on one end is Mickey Mouse (3 female round pins); on the other is NEMA 5-15P (2 parallel flat blades and grounding pin, for North America, Japan, Andean South America), length 59cm/23in. The output cable is 173cm/68in and ends in an inline barrel connector, about 5mm OD, 3mm ID. Length overall including the brick's body: 244cm/96in.
• Documentation: All/No language setup guide; regulatory statements; safety information sheet.

Setup Outline

• Make a special backup of Diamond, in case of mistakes. [Done]

• The new machine initially will be called Orion. Add it to hostdata.db and install relevant files everywhere (/etc/hosts, /etc/ethers, hostgroup.db, trusted-adr.fw). Reload the firewall, to accept the new MAC address. [Done.]

• The machine is pre-assembled: no assembly steps. But it needs a name sticker. [Done]

• Its temporary home will be atop Jacinth's cabinet. Steal Jacinth's monitor, keyboard and mouse. Connect Ethernet to the spare hub port. [Done]

• Unplug the Ethernet (for the next several steps) and let Orion boot into Windows. Once it tries to start setup, shut it down. If it were dead on arrival, it would fail in this step. [It successfully did limited Windows Setup. It did a Wi-Fi scan and found the CouchNet SSID. Not dead on arrival.]

• Update the BIOS using the flasher in the BIOS. [Done] Details here.

• Set up options in the BIOS. [Done] Details here.

• Download a recent Tumbleweed rescue disc, onto USB flash memory. Boot up Orion on this disc. [Done]

• Execute /etc/kea/mkstatic.pl which replaces Orion's MAC address, so it will get the correct address when I plug in Ethernet. [Done, and Orion got its correct address.]

• Partition the disc. [Done] Details here.

• Copy partition contents from the old Diamond to Orion. [Done] Details here.

• Boot up Orion on its own disc. Check out all services (checkout.sh). Fix any problems found. Details here.

• Subsystem checkout and testing. I went through subsystem.shtml and did the listed tests; q.v. for the results. Discrepancies and fixups are listed here.

• Do the power and speed measurements on the new machine. [Done] Details here.

• I need to make Diamond become Orion, and Orion become Diamond, without making errors, I created /s1/etc-orion and /s1/etc-diamond on both hosts, with a directory structure actually starting at the root, containing host-specific files. I can swap one or the other into place by:
  rsync -n -a -O /s1/etc-diamond/ /
  (-n for testing, and it will help to squeeze in --log-format="%o %f" to make it report what it's copying.)

  The files involved are:

  • etc/hostname
  • /etc/sysconfig/network/ifcfg-br0 : has static IPs and MAC
  • /etc/krb5/krb5.{conf,keytab} : Kerberos host key
  • /etc/ssh : all host keys (+everything excluding moduli)
  • /etc/ssl/{hostcerts,private} : SSL/TLS host key
  • etc/apache2/conf.d/$HOST.conf

  For each directory, once it's populated, install it on the other host. Test that everything in the host's own dir is right (doesn't get selected for non-copying with rsync -n) and that all files would get non-copied (with rsync -n) from the other host's dir.

• Burn bridges: interchange Diamond's and Orion's name and host keys; see the previous paragraph for which files those are and how to make the exchange. Details:

  • Starting with Diamond, then Orion, install the other host's keys from the staging directory, then shut it down. Minimize the time when there are two Orions on the net at once, and particularly avoid having two Diamonds.
  • Boot up the new machine (now Diamond), do audit-scripts, then restarter, and checkout.sh. Check that LDAP, Kerberos, SSH and Apache (TLS) function with the new name. Repeat with the old machine (now Orion); formerly working LDAP and Kerberos KDC should be out of action.
  • {All done,]

• hostdata.db: Put Orion in the down hostgroup. Interchange the MAC address for orionen0 and diamonden0. Don't swap the MACs on br0, which by local policy are derived from the IPv4 address. Rebuild /etc/ethers and /usr/diklo/lib/hostgroup.db and install on all hosts. [Done]

• Shut down the old machine (Orion). Print up a note about the old machine's status. Put the old machine in its box in the storage area. [Done]

• Move the new machine to Alice's desktop. Re-pair the Bluetooth mouse. Check that Cups can print. [Done]

BIOS Update

As delivered, the machine has BIOS version 0040 dated 2021-04-14. Not too old, but I should check for recent updates. How to obtain the latest BIOS version:

• Start at Intel Download Center.
• Search for the product, NUC11PAHi5. Narrowest category is NUC kits, 11th generation. Restrict the OS to OS Independent. Download type = BIOS.
• Pick the latest BIOS update; older versions are also on the list. As of 2021-11-20 the latest version for our NUC is 0041 dated 2021-09-07, title PATGL357.
• Now, which package do I want? Presently, the one with extension .CAP is for flashing with the BIOS' builtin flasher on F7.
• Agree to the license. This time it automatically started the download.
• Copy the .CAP file to a USB flash stick with a FAT filesystem. (VFAT OK; 12-16-32bit FATs OK.)
• With the USB stick in Orion, reboot and hit F7 when the splash screen appears. Give it some time, probably doing the long POST.
• You drop into BIOS Flash Update. This page is also accessible from the regular setup on F2. It shows the current BIOS version; we have version 0040 dated 2021-04-14.
• The third line says Unknown Device. Scroll down to this line and hit Enter.
• It lists all the files whose names end in .CAP (in my case just one). Scroll down to the one you want (highlighted white) and hit Enter.
• After a confirmation box, it reboots and starts flashing the power button light. It shows progress messages; the Main Block Firmware takes the longest time. Total time about 80 seconds. It then reboots normally. When the splash screen appears (Intel NUC), hold down F2 to get into BIOS setup. Verify that the new BIOS version got installed.

BIOS Setup

• This should be done after the BIOS update because that resets at least some options to factory defaults.

• Press F2 during booting to get into BIOS setup. It shows Intel NUC for about one second, and likely you have to hit F2 while that screen is showing. Alternatively, just hold down F2 when it starts the boot process.

• Main page: System Information.

  • Processor type: 11th Gen Intel® Core® i5-1135G7 @2.40GHz
  • Turbo: 4.20GHz
  • L2 cache: 1280Kb on each core
  • L3 cache: 8Mb
  • CPU ID: 0x806C1
  • Microcode update revision: 72
  • Memory installed: 16Gb (2x 8Gb)
  • Memory speed: 2667MHz
  • It has the Intel Management Engine, firmware 15.0.10.1447 . See the Wikipedia article about the Management Engine, its security issues, and partial mitigations (which I didn't do).
  • The time and date are shown in 2 places: the one in the list is 16:30 which is the local timezone, -0800; in the headline it's 04:30 PM (equal). I changed the system time to UTC right away, so files created on the rescue disc will have the correct time. It was shipped from Oregon; I'll bet it was assembled in the -0800 timezone (Oregon), though marked as made in China. Prior NUCs had the timezone for China.

• Advanced (Devices): I changed only one of these.

  • Storage: The M.2 slot has a NP830 256Gb (PCIe x4, 8.0GT/s) (giga transactions per second?), PCI ID 126f:2263 for the Silicon Motion SM2263EN controller. I haven't found the name of the disc manufacturer.
  • VMD controller is disabled by default (enable for RAID or Optane).
  • Onboard Devices: Thunderbolt support is enabled. I have no displays that use Thunderbolt.
  • Wireless charging is enabled. There is a module for this. In the lid?
  • CIR was disabled. I was tempted, but I left it off.
  • It has HDMI CEC enabled, and 4 behaviors preset. I turned this off. CEC (if supported by the display, ROTFL) turns on/off the display when the CPU is powered off, and vice versa.
  • Video: Video RAM minimum 64Mb, aperture 256Mb, max is controlled by the OS.
  • Add-In Config: we have an Intel Ethernet Controller (3) 1225-V. Here you can configure IPv4+6 host addresses for PXE boot. I didn't touch this.

• Cooling: By default the Fan Control Mode is Balanced but I changed it to Quiet. It has temp sensors on the CPU, the PCH (whatever that is), the memory, and the motherboard. But its policy is to leave the fan off until the CPU temperature reaches 80 C, then turn the fan on full blast, and it leaves a bit set so that at the next boot it shows a message that a thermal emergency was responded to; check for blocked airflow; press any key to continue. I reverted to Balanced.

• Performance: Processor: I turned off hyperthread.

• Security: Totally unlocked. I left these alone. In a public computer lab you will want to set a BIOS password. These Security Features are enabled: virtualization; virtualization with direct I/O; Platform Trust Technology.

• Power:

  • The main profile was set at Max Performance; I changed to Balanced. (Remaining possibility: Low Power.)
  • Dynamic Power Technology was set to Energy Efficient Performance; I left it.
  • In Secondary Power Settings, I changed:
  • Modern Standby Indicator Color: Amber. Default is blue.
  • S4/S5 Deep Sleep: Off (is the default). The item help says that if it's on, the only way to turn the machine on is the power button, because power is off to everything else. On Jacinth with deep sleep it would wake on LAN, USB or CIR (if these are turned on) but not wireless.
  • After Power Failure: Stay off (is the default). On Jacinth this is set to reboot.
  • Wake on LAN from S4/S5: Power On, normal boot (vs. PXE). This requires that the NIC have power at all times. The default is to stay off.
  • Wake from S5: Off! (Is the default.) This is to wake on a schedule.
  • USB S4/S5 Power: Off. This supplies power to the keyboard etc. at all times and makes it possible to wake on keyboard activity. Diamond has USB powered speakers so I left this off. Except I needed to turn this on to keep it from waking immediately after hibernating (??) Further research is needed.

• Boot: Both Legacy Boot and UEFI were enabled, so it says. Under Secure Boot, it's enabled, and this is supposed to disable Legacy Boot. But my kernel had an invalid signature so I disabled Secure Boot. (With a new kernel, is the signature once again valid?) Under Boot Priority, Windows Boot Manager came first. I switched to USB flash drive first, and I disabled PXE over IPv4+6. The USB drive persists even if not plugged in, unlike in some older BIOS versions.

• To save and reboot, hit F10. It booted Windows anyway (shut down). To boot the rescue system, when booting hold down F10 and it will give you a menu on which you can choose Windows or the USB drive. That worked (whew!)

Partition the Disc

At this point there are three ways to proceed:

• Install OpenSuSE Tumbleweed, then restore the special backup onto Orion. This is the most work but has fewest dicey steps. See Holly Hosed, Tested Backup for details of the procedure.

• Partition Orion's disc by hand including creating filesystems and labels. Mount each partition in the rescue system and use rsync, or the equivalent with tar and ssh, to copy Diamond's content onto it. This is the procedure I will try first.

• Partition Orion's disc by hand (no filesystems). For each partition, copy Diamond's raw device onto Orion's (across the net). This method may not work because the partitions probably won't be exactly the same size, even though close, the filesystems will appear to be corrupt, and repairs probably will work but this isn't assured.

• Should I take out Diamond's disc, connect it to Orion with my SATA to USB adapter, and not have to deal with and wait for the network? It's probably better to use the network connection even if slower — the USB adapter isn't that fast anyway.

Partition table on old Diamond:

Orion's disc: /dev/nvme0n1 Silicon Motion SM2263EN/SM2263XT SSD Controller, 238.47GiB, 250.0Gb, can't tell the actual vendor's name but it's in Guangdong, CN.

Partition table on Orion:

The Yast2 partitioner populates /etc/fstab (on the rescue system) with lines for the mountable partitions created. It persists across reboots; there's a copy-on-write overlay area. Mount /dev/nvme0n1p3 (use /dev/disk/by-label/ROOT-11) on /mnt. Copy /etc/fstab to /mnt/etc/fstab-orion . Unmount the partition.

Copy Diamond to Orion

The goal at this point is to copy stuff from Diamond to Orion's disc. A few preliminary steps were needed.

It got into a weird state. For reasons unknown, the live CD user linux (no password) could not log in; logs suggest that it did log in but exited immediately. I'm continuing as root on tty1.

It has a correct /etc/resolv.conf, generated by netconfig, probably from DHCP options that it was given. ssh jimc@diamond uname -a works (password required).

For easier copying I will give root@orion a key agent with jimc's secret key. It will get this key by:
rsync -a jimc@diamond:~/.ssh/ ~/ .ssh/ (which already exists).
eval $(ssh_agent)
ssh-add /root/.ssh/id_rsa (give password)
ssh diamond uname -a (works, no password needed)

Command lines to suck a filesystem: /boot/EFI, home and s1 are mounted on their proper mount points, but not root, of course, so put it on /mnt. But I'll first do EFI because it's very small Guess what, Diamond, Jacinth, Iris don't have EFI booting, only Xena does. Get it from Xena. Another gotcha: rsync will set the owner and group to the numeric values in the rescue disc's /etc/passwd and /etc/group for their names, which differ from what CouchNet is enforcing, leading to things not working. Head this off with the --numeric-ids option.
mount /dev/disk/by-label/ROOT-11 /mnt
rsync -a --one-file-system --numeric-ids --exclude lost+found xena:/boot/efi/ /boot/efi/
rsync -a --one-file-system --numeric-ids --exclude lost+found diamond:/ /mnt/
rsync -a --one-file-system --numeric-ids --exclude lost+found diamond:/home/ /home/

Seems to have arrived in good order, cross fingers. Root had close to 10.0Gb. If it saturates MOCA (100Mbit/sec) it would take 600sec (10min). Actual 19min, no error messages. /home has similar size and speed. I'm not copying /s1; it's all ancient special backups and ancient virtual machine images. But remember to re-create /s1/scr [done].

The daily report had a bunch of permission and ownership issues. This was already seen for /var/lib/lightdm/.Xauthority (preventing the display-manager from starting, see above). I'm beginning to suspect this scenario: rsync was used to import Diamond's files to Orion. For owners and groups it sends the alphabetic names from Diamond (and numeric), and Orion looks up the name in /etc/passwd or /etc/group, and uses the resulting numbers in the file's inode. This is /etc/passwd or /etc/group on the rescue disc, not the CouchNet values, producing these error reports. So I'm going to re-do the transfers.
rsync -a -O --one-file-system --numeric-ids --exclude lost+found diamond:/home/ /home/
And similarly for / (root) and /boot/efi . But I'm going to have to be cautious, to avoid overwriting files whose content is supposed to be different on Orion.

• /home: Not going to copy anything. 3800 files from Alice's browser cache, plus other cruft that I'll delete on both ends rather than copy.
• /boot/efi: no differences.
• / (root): 340 files non-identical. 9 needed to be re-copied. It was a lot of work to recognize them among the majority that were legitimately different.

Subsystem Checkout and Testing

These are discrepancies encountered when doing the tests listed in subsystem.shtml.

• I tried to boot the kernel and initrd copied from Diamond. They needed these steps before I could boot them:

  • Secure Boot Violation, invalid signature detected. I turned off Secure Boot in BIOS. Now that I've gotten it booting, I wonder if I can turn it back on?
  • /boot/grub2/grub.cfg had root=UUID=xxxx with the UUID of Diamond's root partition. Fixed by rebuilding the file: grub2-mkconfig -o /boot/grub2/grub.cfg .
  • The initrd is host specific. You have to run mkinitrd.

• Speed test output in Kb/sec: SHA512 1 core 345663 4 cores 1382652 reading 87859 overall 346384 Comparison to the old Diamond: CPU about 2.5x, disc reading 9x.

• Wake on LAN does not wake Orion from S5. It should be enabled in BIOS Power-Secondary. See BIOS Setup power section. [Done, now it wakes.]

• After hibernation, Orion does a full reboot, not resuming. This happened on Xena too, but other hosts resume with no special configuration. To fix, I created /etc/dracut.conf.d/99-resume.J.conf containing
  add_dracutmodules+=" resume "
  (a space is required before and after each module name). #Comments OK. Then execute mkinitrd --or-- dracut -vf . [Fixed, now it resumes from hibernation.]

  To see if an initrd has the resume module, do lsinitrd /boot/initrd-… | less and look for resume in the modules section.

• Now when it goes into S4, 2 secs later it wakes up, i.e. starts the boot process and diverts into restoring the saved image. There was no net traffic (pinger or ssh) to Orion but broadcast packets (ARP, IPv6 neighbor discovery, etc) are likely. Per /proc/acpi/wakeup these devices could wake the machine from S4: (and many more were disabled)

  • PEG0 S4 *enabled pci:0000:00:06.0 PCI bridge
  • RP05 S4 *enabled pci:0000:00:1c.0 PCI bridge
  • RP07 S4 *enabled pci:0000:00:1c.6 PCI bridge
  • PXSX S4 *enabled pci:0000:59:00.0 Ethernet NIC
  • XHCI S4 *enabled pci:0000:00:14.0 USB XHCI
  • TXHC S4 *enabled pci:0000:00:0d.0 Thunderbolt USB
  • TDM0 S4 *enabled pci:0000:00:0d.2 USB4 HCI on Thunderbolt
  • TDM1 S4 *enabled pci:0000:00:0d.3 USB4 HCI on Thunderbolt
  • TRP0 S4 *enabled pci:0000:00:07.0 Thunderbolt root port
  • TRP3 S4 *enabled pci:0000:00:07.3 Thunderbolt root port
  • AWAC S4 *enabled platform:ACPI000E:00

  There are forum discussions of Bluetooth causing this problem, supposedly fixed in kernel 5.3.x. I stopped bluetooth and hibernated: didn't help, it woke up again.

  Someone mentioned that if you echo PXSX > /proc/acpi/wakeup it will toggle the enabled status of that device. I tried disabling all of them, then hibernating: it disabled all but PXSX (Ethernet NIC). Success: it stayed off for 60 sec. Wake on LAN woke it up.

  Turning them on one at a time, most suspicious first: XHCI: wakes immediately. I put everything on except XHCI. Stayed asleep for 60sec. In setup, USB S4/S5 power was off; suppose I turn it on (and wake on USB from S5). After reboot, XHCI wakeup is enabled. Wonder of wonders, it stayed off for 60 sec. And it woke on USB. I'll want to keep an eye on power consumption, and maybe suppress waking on USB.

• Intel docs claim the i5-1135G7 chipset includes a watchdog timer, but the iTCO_wdt.ko driver was not loaded. It would appear that this happens in the initrd, if Dracut configuration includes the watchdog module. Most machines have this by default, but on Orion you have to configure it. Create /etc/dracut.conf.d/99-watchdog.J.conf containing
  add_dracutmodules+=" watchdog "
  (spaces required before and after the module name; #comments OK), and run mkinitrd or dracut -vf. [Done]

Speed Measurements

Speed test on Jimc's benchmark. Columns in the output:

• CPU speed, kbytes/sec, doing a sha512 sum on concatenated large files preloaded into the buffer cache, using 1 core.
• Same, times the number of cores (4), not counting hyperthread, which is disabled on these machines.
• Speed in kbytes/sec to read OS files, way beyond the buffer cache.
• Composite score, 0.375*(sha512) + 0.125*(sha512)*(number of cores) + 0.5*(I/O speed)

The test was run 3 times and the last one is reported. Actually the test is designed to be reasonably immune to buffer cache effects and scores vary only about 3% between repetitions. Numbers are in kbytes/sec.

SHA512	SHA512*cores	Disc read	Composite	Machine
120848	241696	12003	81531	NUC5i5RYH
200479	400958	4964	127781	NUC7i5BNH
346151	1384604	84795	345279	NUC11PAHi5

Selection

Testing

Setup

Top

Photo Credit