+# Xeon Phi Setup/Build Notes #
+
+These notes cover the creation of a Debian 10.9 (buster) server with ZFS root
+which serves as host to Knights Corner Xeon Phi coprocessor cards.
+
+Each of these coprocessor cards features a P54C-derived core extended to
+support the X86-64 instruction set, 4-way SMT, and a beefy 512-bit vector
+processor bolted alongside. Sixty of these cores are connected on a roughly 1
+terabit/s bi-directional ring bus. In addition to 8GB of GDDR5 RAM, each core
+has 512kB of local cache and, via the ring bus and a distributed tag store, all
+caches are coherent and quickly accessible from remote cores. This hardware is
+packaged up on a PCIe card which presents a virtual network interface to the
+host. The coprocessor card runs Linux+BusyBox, allowing SSH access to a
+traditional Linux environment on a familiar 60-core x86-64 architecture.
+
+The hostname `frostburg.subgeniuskitty.com` stems from the original
+[FROSTBURG](https://en.wikipedia.org/wiki/FROSTBURG), a CM-5 designed by
+Thinking Machines. Although the fundamental connection topology of a fat tree
+was different than the ring used in this Xeon Phi, the systems are somewhat
+similar. Both feature a NUMA cluster of repackaged and extended commercial
+processor cores operating on independent instruction streams in a MIMD fashion
+focused on small local data stores. By coincidence, both also feature similar
+core counts and total memory size.
+
+The information on this page includes:
+
+ - Hardware compatibility notes for Xeon Phi and Xeon host.
+
+ - Installation of Debian 10.9 (buster) root on encrypted ZFS mirror with
+ automated snapshots and scrubs.
+
+ - (TODO) Porting the Intel kernel module to Linux kernel version 4.19.0.
+
+ - (TODO) Installing MPSS toolkit on Debian (or CentOS VM).
+
+ - (TODO) Building GCC toolchain for Xeon Phi.
+
+ - (TODO) Installing Intel toolchain for Xeon Phi.
+
+These notes are a high-level checklist for my reference rather than a
+step-by-step installation guide for the public. That means they make no attempt
+to explain all options at each step, rather that they mention only the options
+I use on my servers. It also means they use my domains, my file system paths,
+etc in the examples. Don't blindly copy and paste.
+
+--------------------------------------------------------------------------------
+
+
+## Hardware ##
+
+The host system was kept low power both figuratively and literally. It will
+primarily serve as a host for the Phi coprocessors and bridge to the network.
+
+ - **Chassis:** Supermicro 2027GR-TR2
+
+ - **Motherboard:** Supermicro X9DRG-HF+II
+
+ - **CPU:** 2x Xeon E5-2637
+
+ - **RAM:** 8x 4GB DDR3 RDIMM
+
+ - **Storage:** 2x Intel 160GB X-25M SSD
+
+ - **Payload:** 4x Intel Xeon Phi 5110P
+
+To enter the BIOS, use the `DEL` key. Similarly, a boot device selection menu
+is obtained by pressing `F11`. System will display two-character status codes
+in the bottom right corner of display.
+
+Support files are stored under `hw_support/Intel Xeon Phi/supermicro/`.
+
+
+### Memory ###
+
+Using eight identical sticks of MT36JSZF51272PZ-1G4 RAM. These are ECC DDR3
+2Rx4 PC3-10600 RDIMMS operating at 1.5V. Per page 2-12 of the manual
+(`MNL_1502.pdf`), DIMMs are installed in all blue memory slots.
+
+
+### Processors & Heatsinks ###
+
+Xeon E5-2637 CPUs selected for lower power, high frequency, cheap price, and
+'full' PCIe lane count. They only need to be a host for the real show. Per page
+5-7 of the chassis manual (`MNL-1564.pdf`), CPU1 requires heatsink SNK-P0048PS
+and CPU2 requires heatsink SNK-P0047PS.
+
+
+## SAS Backplane & Motherboard SATA ###
+
+The SAS backplane is a little odd. The first eight drive bays connect via a
+pair of SFF-8087 connectors and the last two drive bays connect via standard
+7-pin SATA connectors.
+
+Since the motherboard provides ten 7-pin SATA connectors, two cables breaking
+out SFF-8087 to quad SATA will be required. I tried using just such a cable,
+but had no luck. There doesn't appear to be anything configurable on the
+backplane itself. The backplane manual is stored at `BPN-SAS-218A.pdf`. My
+cable was of unknown origin. Per photos on some eBay auctions, the proper
+Supermicro cable appears to be part number 672042095704. In addition to the
+four SATA connectors, this cable also bundles some sort of 4-pin header,
+presumably the SGPIO connection.
+
+In the meantime, since I only intend to use two small drives in a ZFS mirror
+for the OS and home directories, with all other storage on network shares,
+simply use the last two slots and connect with normal 30"+ SATA cables.
+
+These last two drive bay slots are connected to the two white SATA ports on the
+motherboard, with the lowest numbered drive slot connected to the rear-most
+white SATA port. When SFF-8087 connectors are eventually used to increase local
+storage, relocate the boot drives to drive slots 0 and 1, and connect these
+slots to the white SATA ports.
+
+On the motherboard, the white ports are SATA3 and the black ports are SATA2.
+The line of 2x white and 4x black SATA ports are part of the primary SATA
+controller or `I_SATA`. The other line of 4x black SATA ports is part of the
+secondary or `S_SATA` controller. Put any boot drives on the `I_SATA` ports.
+
+
+### Xeon Phi ###
+
+Section 5.1 of the Intel Xeon Phi Coprocessor Datasheet (DocID 328209-004EN)
+mentions that connecting the card via both 2x4 and 2x3 power connectors enables
+higher sustained power draw up to 245 watts versus 225 watts of other power
+cable configurations. This chassis will easily support the higher power draw
+and heat dissipation.
+
+The Xeon Phi coprocessor cards reserve PCIe MMIO address space sufficient to
+map the entire coprocessor card's RAM. Since this is >4GB, PCIe Base Address
+Registers (BAR) of greater than 32-bit size are required. This should be
+enabled in the BIOS of this particular motherboard under
+`PCIe/PCI/PnP Configuration` -> `Above 4G Decoding`.
+
+In general, motherboards with chipsets equal to or newer than the C602 should
+work. This includes most Supermicro motherboards from the X9xxx generation or
+later. None of the Supermicro X8xxx generation motherboards appear to be
+compatible.
+
+The Xeon Phi 5110P, per the suffix, is passively cooled. Section 3 of the Intel
+Xeon Phi Coprocessor Datasheet (DocID 328209-004EN) details the cooling and
+mounting requirements.
+
+
+### Optional Fans ###
+
+There are a number of optional fans for this chassis, all detailed in the
+chassis manual (`MNL-1564.pdf`). My machine includes the optional fan for
+another double-height, full-length PCIe card with backpanel IO slots, intended
+to support something like a GPU to drive monitors. Since the optional fan is
+installed and since the power budget easily supports it, this means the fifth
+Xeon Phi card could be installed, albeit with slower PCIe connection.
+
+Regardless, since this fan is installed, whenever fewer than four Xeon Phi
+cards are installed, preferentially locate them on the left hand side of
+chassis, near the lower numbered drive bays.
+
+
+### Power Supply ###
+
+The system contains dual redundant power supplies. Each is capable of supplying
+1600 watts, but only when connected to a 240 volt source. When connected to a
+120 volt source, maximum power output is 1000 watts.
+
+
+### Rackmount ###
+
+The chassis is over 30" long and protrudes from rear of rack by approximately
+1/2". To avoid the rear cable snagging passing carts and elbows, chassis was
+mounted at top of rack (after empty 1U). The Supermicro rails required cutting
+four notches in the vertical posts, so this is a semi-permanent home.
+
+Inserting or extracting the server from the rack at that height requires an
+extraordinary amount of free space in front of the rack and some advance
+planning. Where possible, try to do hardware modifications in-rack. The rails
+are extremely solid even when the server is fully extended. The grey
+OS-114/WQM-4 sonar test set chassis makes a solid step stool at the ideal
+height for working on the server while installed in the rack.
+
+
+### USB Ports ###
+
+There are only two USB ports, both located on the rear of the chassis. During
+OS installation, if a mouse is required in addition to the keyboard and USB
+install drive, then a USB hub is required.
+
+--------------------------------------------------------------------------------
+
+
+## Debian Buster Installation ##
+
+These installation instructions use the following XFCE Debian live image.
+
+ debian-live-10.9.0-amd64-xfce.iso
+
+Both the Gnome and XFCE live images were unusably slow in GUI mode. The text
+installer was fast and responsive, as were VTYs (`Ctrl`+`Alt`+`F2`) from within
+the live environment. Only the GUIs were slow, but they were slow to the point
+of being unusable, with single keypresses registering over a dozen times. Once
+Debian was installed on the SSD and booting normally, the GUI is perfectly
+usable. Since the local terminal is only used to install and start an OpenSSH
+daemon, and since this can be done from a VTY, the issue was not investigated
+further.
+
+The root on ZFS portion of this installation process is derived from the guide
+located here:
+
+<https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Buster%20Root%20on%20ZFS.html>
+
+
+### Remote Access ###
+
+From the `F11` BIOS boot menu, select the UEFI entry for the USB live image.
+Lacking a mouse, press `CTRL`+`ALT`+`F2` after X is running in order to access
+a text-only VTY, already logged in as the user `user`. Install an SSH server so
+the remaining install can be done over the network.
+
+ apt-get update
+ apt-get install openssh-server
+ systemctl enable ssh
+
+From wherever you intend to complete the install, SSH into the live Debian
+environment as user `user` with password `live`.
+
+
+### ZFS Configuration ###
+
+Edit `/etc/apt/sources.list` to include the following entries.
+
+ deb http://deb.debian.org/debian/ buster main contrib
+ deb http://deb.debian.org/debian/ buster-backports main contrib
+ deb-src http://deb.debian.org/debian/ buster main contrib
+
+Install the ZFS kernel module. Specify `--no-install-recommends` to avoid
+picking up `zfsutils-linux` since it will fail at this point. See
+<https://github.com/openzfs/zfs/issues/9599> for more details.
+
+ apt-get install -t buster-backports --no-install-recommends zfs-dkms
+ modprobe zfs
+
+With the kernel module successfully loaded, proceed to install ZFS.
+
+ apt-get install -t buster-backports zfsutils-linux
+
+After using `dd` to eliminate any existing partition tables, partition the
+disks for use with UEFI and ZFS.
+
+First, create a UEFI partition on each disk.
+
+ sgdisk -n2:1M:+512M -t2:EF00 /dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTPO1252011L160AGN
+
+Next, create a partition for the boot pool.
+
+ sgdisk -n3:0:+1G -t3:BF01 /dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTPO1252011L160AGN
+
+Finally, create a partition for the encrypted pool.
+
+ sgdisk -n4:0:0 -t4:BF00 /dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTPO1252011L160AGN
+
+Now that partitioning is complete, create the boot and root pools.
+
+The boot pool uses only ZFS options supported by GRUB.
+
+ zpool create \
+ -o cachefile=/etc/zfs/zpool.cache \
+ -o ashift=12 -d \
+ -o feature@async_destroy=enabled \
+ -o feature@bookmarks=enabled \
+ -o feature@embedded_data=enabled \
+ -o feature@empty_bpobj=enabled \
+ -o feature@enabled_txg=enabled \
+ -o feature@extensible_dataset=enabled \
+ -o feature@filesystem_limits=enabled \
+ -o feature@hole_birth=enabled \
+ -o feature@large_blocks=enabled \
+ -o feature@lz4_compress=enabled \
+ -o feature@spacemap_histogram=enabled \
+ -o feature@zpool_checkpoint=enabled \
+ -O acltype=posixacl -O canmount=off -O compression=lz4 \
+ -O devices=off -O normalization=formD -O relatime=on -O xattr=sa \
+ -O mountpoint=/boot -R /mnt \
+ bpool mirror \
+ /dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTPO1252011L160AGN-part3
+ /dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTHC72250AKD480MGN-part3
+
+Now create the root pool with ZFS encryption.
+
+ zpool create \
+ -o ashift=12 \
+ -O encryption=aes-256-gcm \
+ -O keylocation=prompt -O keyformat=passphrase \
+ -O acltype=posixacl -O canmount=off -O compression=lz4 \
+ -O dnodesize=auto -O normalization=formD -O relatime=on \
+ -O xattr=sa -O mountpoint=/ -R /mnt \
+ rpool mirror \
+ /dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTPO1252011L160AGN-part4
+ /dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTHC72250AKD480MGN-part4
+
+All the pools are created, so now it's time to setup filesystems. Start with
+some containers.
+
+ zfs create -o canmount=off -o mountpoint=none rpool/ROOT
+ zfs create -o canmount=off -o mountpoint=none bpool/BOOT
+
+Now add filesystems for boot and root.
+
+ zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/debian
+ zfs mount rpool/ROOT/debian
+ zfs create -o mountpoint=/boot bpool/BOOT/debian
+
+Create a filesystem to contain home directories and mount root's homedir in the
+correct location.
+
+ zfs create rpool/home
+ zfs create -o mountpoint=/root rpool/home/root
+ chmod 700 /mnt/root
+
+Create filesystems under `/var` and exclude temporary files from snapshots.
+
+ zfs create -o canmount=off rpool/var
+ zfs create -o canmount=off rpool/var/lib
+ zfs create rpool/var/log
+ zfs create rpool/var/spool
+ zfs create -o com.sun:auto-snapshot=false rpool/var/cache
+ zfs create -o com.sun:auto-snapshot=false rpool/var/tmp
+ chmod 1777 /mnt/var/tmp
+ zfs create rpool/var/mail
+
+Create a few other misc filesystems.
+
+ zfs create rpool/srv
+ zfs create -o canmount=off rpool/usr
+ zfs create rpool/usr/local
+
+Temporarily mount a `tmpfs` at `/run`.
+
+ mkdir /mnt/run
+ mount -t tmpfs tmpfs /mnt/run
+ mkdir /mnt/run/lock
+
+
+### Debian Configuration ###
+
+Install a minimal Debian system.
+
+ apt-get install debootstrap
+ debootstrap buster /mnt
+
+Copy the zpool cache into the new system.
+
+ mkdir /mnt/etc/zfs
+ cp /etc/zfs/zpool.cache /mnt/etc/zfs
+
+Set the hostname.
+
+ echo frostburg > /mnt/etc/hostname
+ echo "127.0.1.1 frostburg.subgeniuskitty.com frostburg" >> /mnt/etc/hosts
+
+Configure networking.
+
+ vi /mnt/etc/network/interfaces.d/enp129s0f0
+
+ auto enp129s0f0
+ iface enp129s0f0 inet static
+ address 192.168.1.7/24
+ gateway 192.168.1.1
+
+ vi /etc/resolv.conf
+
+ search subgeniuskitty.com
+ nameserver 192.168.1.1
+
+Configure packages sources.
+
+ vi /mnt/etc/apt/sources.list
+
+ deb http://deb.debian.org/debian buster main contrib
+ deb-src http://deb.debian.org/debian buster main contrib
+
+ deb http://security.debian.org/debian-security buster/updates main contrib
+ deb-src http://security.debian.org/debian-security buster/updates main contrib
+
+ deb http://deb.debian.org/debian buster-updates main contrib
+ deb-src http://deb.debian.org/debian buster-updates main contrib
+
+ vi /mnt/etc/apt/sources.list.d/buster-backports.list
+
+ deb http://deb.debian.org/debian buster-backports main contrib
+ deb-src http://deb.debian.org/debian buster-backports main contrib
+
+ vi /mnt/etc/apt/preferences.d/90_zfs
+
+ Package: libnvpair1linux libuutil1linux libzfs2linux libzfslinux-dev libzpool2linux python3-pyzfs pyzfs-doc spl spl-dkms zfs-dkms zfs-dracut zfs-initramfs zfs-test zfsutils-linux zfsutils-linux-dev zfs-zed
+ Pin: release n=buster-backports
+ Pin-Priority: 990
+
+ apt-get update
+
+Chroot into the new environment.
+
+ mount --rbind /dev /mnt/dev
+ mount --rbind /proc /mnt/proc
+ mount --rbind /sys /mnt/sys
+ chroot /mnt
+
+Configure the new environment as a basic system.
+
+ ln -s /proc/self/mounts /etc/mtab
+ apt-get update
+ export TERM=vt100
+ apt-get install console-setup locales
+ dpkg-reconfigure locales tzdata keyboard-configuration console-setup
+
+Install ZFS on the new system.
+
+ apt-get install dpkg-dev linux-headers-amd64 linux-image-amd64
+ apt-get install zfs-initramfs
+ echo REMAKE_INITRD=yes > /etc/dkms/zfs.conf
+
+Install GRUB and configure UEFI boot partition.
+
+ apt-get install dosfstools
+ mkdosfs -F 32 -s 1 -n EFI /dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTPO1252011L160AGN-part2
+ mkdir /boot/efi
+ echo "/dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTPO1252011L160AGN-part2 /boot/efi vfat defaults 0 0" >> /etc/fstab
+ mount /boot/efi
+ apt-get install grub-efi-amd64 shim-signed
+ apt-get remove --purge os-prober
+
+Ensure the bpool is always imported, even if `/etc/zfs/zpool.cache` doesn't
+exist or doesn't include a relevant entry.
+
+ vi /etc/systemd/system/zfs-import-bpool.service
+
+ [Unit]
+ DefaultDependencies=no
+ Before=zfs-import-scan.service
+ Before=zfs-import-cache.service
+
+ [Service]
+ Type=oneshot
+ RemainAfterExit=yes
+ ExecStart=/sbin/zpool import -N -o cachefile=none bpool
+ # Work-around to preserve zpool cache:
+ ExecStartPre=-/bin/mv /etc/zfs/zpool.cache /etc/zfs/preboot_zpool.cache
+ ExecStartPost=-/bin/mv /etc/zfs/preboot_zpool.cache /etc/zfs/zpool.cache
+
+ [Install]
+ WantedBy=zfs-import.target
+
+ systemctl enable zfs-import-bpool.service
+
+Create a `tmpfs` mounted at `/tmp`.
+
+ cp /usr/share/systemd/tmp.mount /etc/systemd/system/
+ systemctl enable tmp.mount
+
+
+### Bootloader Configuration ###
+
+Verify ZFS boot filesystem is recognized.
+
+ grub-probe /boot
+
+Refresh initrd.
+
+ update-initramfs -c -k all
+
+Configure GRUB by editing `/etc/default/grub`. Remove the `quiet` option from
+`GRUB_CMDLINE_LINUX_DEFAULT` and add the following two options to the
+appropriate entries.
+
+ GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/debian"
+ GRUB_TERMINAL=console
+
+Install GRUB to the UEFI boot partition.
+
+ grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=debian-1 --recheck --no-floppy
+
+Install GRUB on the other hard drives, incrementing `-2` to `-N` as necessary.
+
+ umount /boot/efi
+ dd if=/dev/disk/by-id/scsi-SATA_disk1-part2 \
+ of=/dev/disk/by-id/scsi-SATA_disk2-part2
+ efibootmgr -c -g -d /dev/disk/by-id/scsi-SATA_disk2 \
+ -p 2 -L "debian-2" -l '\EFI\debian\grubx64.efi'
+ mount /boot/efi
+
+Fix filesystem mount ordering. Quoting from the install reference, "We need to
+activate `zfs-mount-generator`. This makes systemd aware of the separate
+mountpoints, which is important for things like `/var/log` and `/var/tmp`. In
+turn, `rsyslog.service` depends on `var-log.mount` by way of `local-fs.target`
+and services using the `PrivateTmp` feature of systemd automatically use
+`After=var-tmp.mount`."
+
+ mkdir /etc/zfs/zfs-list.cache
+ touch /etc/zfs/zfs-list.cache/bpool
+ touch /etc/zfs/zfs-list.cache/rpool
+ zed -F
+
+From another SSH session, verify that zed updated the cache by making sure the
+previously created empty files are not empty.
+
+ cat /etc/zfs/zfs-list.cache/bpool
+ cat /etc/zfs/zfs-list.cache/rpool
+
+If all is well, return to the previous SSH session and terminate `zed` with
+`Ctrl`+`C`.
+
+Fix the paths to eliminate `/mnt`.
+
+ sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/*
+
+
+### Reboot ###
+
+The Debian install is almost ready for use without the live Debian host
+environment. Only a few steps remain.
+
+Do a final system update.
+
+ apt-get dist-upgrade
+
+Disable log compression since ZFS is already compressing at the block level.
+
+ for file in /etc/logrotate.d/* ; do
+ if grep -Eq "(^|[^#y])compress" "$file" ; then
+ sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file"
+ fi
+ done
+
+Install an SSH server so we can login again after rebooting.
+
+ apt-get install openssh-server
+
+Set a root password.
+
+ passwd
+
+Create a user account.
+
+ zfs create rpool/home/ataylor
+ adduser ataylor
+ mkdir /etc/skel/.ssh && chmod 700 /etc/skel/.ssh
+ cp -a /etc/skel/. /home/ataylor/
+ scp ataylor@lagavulin:/usr/home/ataylor/.ssh/id_rsa.pub /home/ataylor/.ssh/authorized_keys
+ chown -R ataylor:ataylor /home/ataylor
+ usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video ataylor
+
+Snapshot the install.
+
+ zfs snapshot bpool/BOOT/debian@install
+ zfs snapshot rpool/ROOT/debian@install
+
+Exit the chroot and unmount all filesystems.
+
+ exit
+ mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
+ zpool export -a
+
+Reboot the computer and remove the USB stick. Installation is complete.
+
+
+### UNIX Userland ###
+
+Install various no-config-required userland packages before continuing.
+
+ apt-get install net-tools bzip2 zip ntp htop xterm screen git \
+ build-essential pciutils smartmontools gdb valgrind
+
+
+#### X Window Manager ####
+
+Install X and dwm to ensure all dependencies are met for running my dwm-derived
+window manager.
+
+ apt-get install xorg dwm numlockx
+
+Install dependencies for building my window manager.
+
+ apt-get install libx11-dev libxft-dev libxinerama-dev
+
+Copy the Hophib Modern Desktop git repo to the new server. Make the following changes:
+
+ - `hhmd/src/mk.conf`: Change the installation prefix from `/hh` to
+ `/home/ataylor/bin`
+
+ - `hhmd/src/window_manager/Makefile`: Change library and include paths from
+ `/usr/local/...` to `/usr/...`
+
+ - `hhmd/src/window_manager/dwm-status.c`: Change `#include <sys/time.h>` to
+ `#include <time.h>` and add `#define _GNU_SOURCE` as well as
+ `#define _DEFAULT_SOURCE` to the top of the file
+
+ - `hhmd/src/window_manager/dwm.c`: Add `#define _POSIX_C_SOURCE 2` to the top
+ of the file.
+
+ - `hhmd/src/window_manager/dwm-watchdog.sh`: Change paths and executable
+ names from `/hh/...` to `/home/ataylor/bin/...` and from `wm` to `dwm`.
+
+Execute `make clean install`. Verify that `dwm`, `dwm-status` and
+`dwm-watchdog.sh` all ended up in `/home/ataylor/bin` with appropriate
+permissions. Delete the man pages that were installed in ataylor's homedir.
+
+Create `~/.xinitrc` with following contents.
+
+ /usr/bin/numlockx &
+ /home/ataylor/bin/dwm-status &
+ /home/ataylor/bin/dwm-watchdog.sh
+
+Verify X and my window manager start successfully and that `dwm-watchdog.sh`
+keeps X and X applications alive during a window manager live restart.
+
+
+#### VIM ####
+
+Install gvim.
+
+ apt-get install gvim
+
+Create `~/.vimrc` with the following contents.
+
+ set nocompatible
+ filetype off
+ set mouse=r
+ set number
+ syntax on
+ set tabstop=4
+ set expandtab
+
+ "Folding
+ "http://vim.wikia.com/wiki/Folding_for_plain_text_files_based_on_indentation
+ "set foldmethod=expr
+ "set foldexpr=(getline(v:lnum)=~'^$')?-1:((indent(v:lnum)<indent(v:lnum+1))?('>'.indent(v:lnum+1)):indent(v:lnum))
+ "set foldtext=getline(v:foldstart)
+ "set fillchars=fold:\ "(there's a space after that \)
+ "highlight Folded ctermfg=DarkGreen ctermbg=Black
+ "set foldcolumn=6
+
+ " Color the 100th column.
+ set colorcolumn=100
+ highlight ColorColumn ctermbg = darkgray
+
+
+#### TCSH ####
+
+Install tcsh.
+
+ apt-get install tcsh
+
+Change the default shell for new users by editing `/etc/adduser.conf`, setting
+the `DSHELL` variable to `/bin/tcsh`. Then use the `chsh` command to change the
+shell for root and ataylor. Create `~/.cshrc` in ataylor's and root's homedir
+with the following contents.
+
+ # .cshrc - csh resource script, read at beginning of execution by each shell
+
+ alias h history 25
+ alias j jobs -l
+ alias la ls -aF
+ alias lf ls -FA
+ alias ll ls -lAF --color
+ alias ls ls --color
+
+ # These are normally set through /etc/login.conf. You may override them here
+ # if wanted.
+ set path = (/sbin /bin /usr/sbin /usr/bin /usr/local/sbin /usr/local/bin $HOME/bin)
+
+ setenv EDITOR vim
+ setenv PAGER more
+
+ if ($?prompt) then
+ # An interactive shell -- set some stuff up
+ set prompt = "%N@%m:%~ %# "
+ set promptchars = "%#"
+
+ set filec
+ set history = 1000
+ set savehist = (1000 merge)
+ set autolist = ambiguous
+ # Use history to aid expansion
+ set autoexpand
+ set autorehash
+ set mail = (/var/mail/$USER)
+ if ( $?tcsh ) then
+ bindkey "^W" backward-delete-word
+ bindkey -k up history-search-backward
+ bindkey -k down history-search-forward
+ endif
+
+ endif
+
+
+#### XScreensaver ####
+
+Install Xscreensaver and configure screen locking.
+
+ apt-get install xscreensaver xscreensaver-data
+
+Run `xscreensaver-demo` and select some screensavers. If inspiration doesn't
+strike, do single screensaver mode with the `abstractile` hack; it looks good
+on pretty much any hardware. Remember to enable screen locking.
+
+Add the following line to `~/.xinitrc`.
+
+ /bin/xscreensaver -nosplash &
+
+
+#### ZFS Snapshots ####
+
+In order to configure automatic ZFS snapshots, use the `auto-zfs-snapshot`
+package.
+
+ apt-get install auto-zfs-snapshot
+
+In addition to the snapshot script itself, this package includes automatically
+enabled cron entries, but it will only snapshot filesystems with the
+`com.sun:auto-snapshot` property set to `true`. Since we already manually set
+that property to `false` for `/var/cache` and `/var/tmp`, simply set it to
+`true` for the two parent pools and allow filesystems to inherit wherever
+possible.
+
+ zfs set com.sun:auto-snapshot=true rpool
+ zfs set com.sun:auto-snapshot=true bpool
+
+Verify that relevant filesystems inherited the property.
+
+ zfs get com.sun:auto-snapshot
+
+After waiting 15+ minutes, verify that snapshots begin to appear.
+
+ zfs list -t snapshot
+
+
+#### ZFS Scrubs ####
+
+Automate ZFS scrubs by creating `/etc/cron.d/zfs-scrubs` with the following
+contents.
+
+ PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin
+ 0 0 0 * * root /sbin/zpool scrub rpool
+ 0 0 0 * * root /sbin/zpool scrub bpool
+
+
+#### Status Updates ####
+
+In order to receive status updates like failed drive notifications, we must
+first configure the system to send email through the SGK mail server. Rather
+than use `exim4` as provided by the base system, instead use `msmtp`.
+
+ apt-get install msmtp-mta
+
+Create the file `/etc/msmtprc` with the following contents.
+
+ # Set default values for all following accounts.
+ defaults
+ auth on
+ tls on
+ tls_trust_file /etc/ssl/certs/ca-certificates.crt
+ tls_starttls off
+
+ # Account: subgeniuskitty
+ account default
+ host mail.subgeniuskitty.com
+ port 465
+ from ataylor@subgeniuskitty.com
+ user ataylor@subgeniuskitty.com
+ password <plaintext-password>
+
+Create the file `/etc/cron.d/status-emails` with the following contents.
+
+ PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin
+ SHELL=/bin/bash
+ 0 0 * * 0 root /sbin/zpool status | echo -e "Subject:FROSTBURG: zpool status\n\n $(cat -)" | msmtp ataylor@subgeniuskitty.com
+
+--------------------------------------------------------------------------------
+
+
+## Xeon Phi Kernel Module ##
+
+It appears that Linux kernel version 4.19.0 included with Debian 10.9 already
+has some sort of in-tree kernel support for these Xeon Phi coprocessor cards as
+seen in the final lines of the following diagnostic output.
+
+ root@frostburg:~ # lspci | grep -i Co-processor
+ 02:00.0 Co-processor: Intel Corporation Xeon Phi coprocessor 5100 series (rev 11)
+ root@frostburg:~ # lspci -s 02:00.0 -vv
+ 02:00.0 Co-processor: Intel Corporation Xeon Phi coprocessor 5100 series (rev 11)
+ Subsystem: Intel Corporation Xeon Phi coprocessor 5100 series
+ Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
+ Latency: 0, Cache Line Size: 64 bytes
+ Interrupt: pin A routed to IRQ 69
+ NUMA node: 0
+ Region 0: Memory at 21c00000000 (64-bit, prefetchable) [size=8G]
+ Region 4: Memory at cb900000 (64-bit, non-prefetchable) [size=128K]
+ Capabilities: [44] Power Management version 3
+ Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot-,D3cold-)
+ Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
+ Capabilities: [4c] Express (v2) Endpoint, MSI 00
+ DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
+ ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W
+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
+ RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
+ MaxPayload 256 bytes, MaxReadReq 512 bytes
+ DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
+ LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <4us, L1 unlimited
+ ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
+ LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
+ DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
+ LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
+ Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
+ Compliance De-emphasis: -6dB
+ LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
+ EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
+ Capabilities: [88] MSI: Enable- Count=1/16 Maskable- 64bit+
+ Address: 0000000000000000 Data: 0000
+ Capabilities: [98] MSI-X: Enable+ Count=16 Masked-
+ Vector table: BAR=4 offset=00017000
+ PBA: BAR=4 offset=00018000
+ Capabilities: [100 v1] Advanced Error Reporting
+ UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
+ UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
+ UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
+ CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
+ Kernel driver in use: mic
+ Kernel modules: mic_host
+
+However, as no virtual network device automatically showed up, and since the
+Intel manuals are plastered with warnings about using exact, sanctioned
+combinations of kernel module, MPSS software, and Phi firmware, I decided to
+avoid the kernel module included with the system and instead attempt porting
+the kernel module source code included with MPSS onto a newer Linux kernel. At
+a minimum, it appears the timer API has changed, as well as some utility
+functions related to requesting block interrupt assignments.
+
projects / website_subgeniuskitty.com / commitdiff