- Installation of Debian 10.9 (buster) root on encrypted ZFS mirror with
automated snapshots and scrubs.
- - (TODO) Porting the Intel kernel module to Linux kernel version 4.19.0.
+ - Porting the Xeon Phi kernel module to newer versions of the Linux kernel.
- (TODO) Installing MPSS toolkit on Debian (or CentOS VM).
Install various no-config-required userland packages before continuing.
apt-get install net-tools bzip2 zip ntp htop xterm screen git \
- build-essential pciutils smartmontools gdb valgrind
+ build-essential pciutils smartmontools gdb valgrind wget \
+ texlive texlive-latex-extra graphviz firefox sysfsutils
#### X Window Manager ####
alias j jobs -l
alias la ls -aF
alias lf ls -FA
- alias ll ls -lAF --color
+ alias ll ls -lF --color
alias ls ls --color
# These are normally set through /etc/login.conf. You may override them here
/bin/xscreensaver -nosplash &
+#### Go Toolchain ####
+
+The version of Go provided via `apt-get` is always out of date, so all Go
+installs on this server are done via tarball from the <https://golang.com>
+website. Go 1.16.3 is used for this example but the newest version of Go may be
+found at <https://golang.org/dl/>.
+
+Previous versions of Go are installed entirely under `/usr/local/go`. Delete
+the entire `/usr/local/go` directory before proceeding.
+
+ wget https://golang.org/dl/go1.16.3.linux-amd64.tar.gz
+ tar -C /usr/local -xzf go1.16.3.linux-amd64.tar.gz
+
+If this is the first time installing Go on the system, update everyone's
+`$PATH` to include `/usr/local/go/bin`. Remember to update files under
+`/etc/skel` at the same time.
+
+
#### ZFS Snapshots ####
In order to configure automatic ZFS snapshots, use the `auto-zfs-snapshot`
SHELL=/bin/bash
0 0 * * 0 root /sbin/zpool status | echo -e "Subject:FROSTBURG: zpool status\n\n $(cat -)" | msmtp ataylor@subgeniuskitty.com
+
+#### IRC Environment ####
+
+IRC is used for collaboration on the server. First install daemon and client.
+
+ apt-get install ngircd irssi
+
+Configure the server by editing `/etc/ngircd/ngircd.conf`. The defaults are
+mostly acceptable but the server must be given a name and restricted to only
+listen for local connections. While we're at it, the max nick length is only 9
+by default and should be increased. Note that these values need to be inserted
+under the appropriate category, as shown below, but the categories already
+exist in the config file.
+
+ [Global]
+ Name = frostburg.subgeniuskitty.com
+ Info = Frostburg - Private IRC Server
+ Listen = 127.0.0.1
+ [Limits]
+ MaxNickLength = 32
+
+Restart the server and verify it listens on the correct addresses.
+
+ # systemctl restart ngircd
+ # netstat -an | grep LISTEN
+ tcp 0 0 127.0.0.1:6667 0.0.0.0:* LISTEN
+
+Startup a client in screen for each user.
+
+ screen -dR irc
+ irssi
+ /connect localhost
+ /join #channel
+
+
+#### Public SSH Access ####
+
+Although frostburg is on a private subnet, I want public SSH access. The
+easiest way to set this up is via a reverse SSH tunnel to one of the public
+subgeniuskitty.com servers.
+
+This section refers to three machines:
+
+ - The **server** is frostburg.subgeniuskitty.com, a machine which we desire
+ to access across the internet despite residing on a private subnet.
+
+ - The **endpoint** is a server with public IP address which will serve as an
+ access portal for the *server*.
+
+ - The **client** is the human user's workstation, the machine which is
+ attempting to login to the *server* via the *endpoint*.
+
+First, setup appropriate login credentials on the *server*, which in this case
+is `frostburg.subgeniuskitty.com`. Ignore any warnings about `/home/username`
+already existing or not being owned by the correct user. These are simply a
+side effect of using ZFS since we must create the homedir before adding the
+user, but we can't change ownership until after the new user exists.
+
+ server:~ # zfs create rpool/home/username
+ server:~ # adduser username
+ server:~ # cp -a /etc/skel/. /home/username
+ server:~ # chown -R username:username /home/username
+ server:~ # zfs snapshot rpoot/home/username@account_creation
+
+If necessary for the intended tasks, add the user to any relevant groups with
+something like the following command.
+
+ server:~ # usermod -a -G netdev,plugdev,sudo,video username
+
+The user will also need login credentials on the *endpoint*. These credentials
+don't need to allow anything other than simply SSHing through to the *server*.
+
+ endpoint:~ # adduser username
+
+With appropriate credentials successfully created, move on to setting up a
+reverse SSH tunnel from *server* to *endpoint*.
+
+First, create an SSH key on the *server* with no passphrase and authorize it
+for logins on the *endpoint*. This will be used to bring the tunnel up when the
+machine boots. If a non-empty passphrase is specified, you will need to type it
+during the boot process.
+
+ server:~ # ssh-keygen
+ server:~ # scp /root/.ssh/id_rsa.pub username@endpoint:/home/username/temp_key_file
+ server:~ # ssh username@endpoint
+ (login requires password)
+ endpoint:~ % mkdir -p /home/username/.ssh
+ endpoint:~ % mv /home/username/temp_key_file /home/username/.ssh/authorized_keys
+ endpoint:~ % logout
+ server:~ # ssh username@endpoint
+ (login does not require password)
+ endpoint:~ % logout
+ server:~ # mv /root/.ssh/id_rsa rtunnel_nopwd
+ server:~ # mv /root/.ssh/id_rsa.pub rtunnel_nopwd.pub
+
+Next, create the tunnel using AutoSSH to maintain a long-term connection.
+
+ server:~ # apt-get install autossh
+ server:~ # vi /etc/systemd/system/autossh-tunnel.service
+ [Unit]
+ Description=AutoSSH tunnel between frostburg.SGK and www.SGK
+ After=network-online.target
+
+ [Service]
+ Environment="AUTOSSH_GATETIME=0"
+ ExecStart=/bin/autossh -N -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -i /root/.ssh/rtunnel_nopwd -R 4242:localhost:22 username@endpoint
+
+ [Install]
+ WantedBy=multi-user.target
+ server:~ # systemctl daemon-reload
+ server:~ # systemctl start autossh-tunnel.service
+ server:~ # systemctl enable autossh-tunnel.service
+
+At this point the SSH tunnel is operational. Let's make things a little easier
+for the user by storing most of the config options in an SSH config file.
+
+ endpoint:~ # su - username
+ endpoint:~ % vi /home/username/.ssh/config
+ Host server
+ Hostname localhost
+ User username
+ Port 4242
+
+Now, when we execute `ssh server`, it is equivalent to the command
+`ssh -p 4242 username@localhost`, much easier to remember.
+
+It's time to test everything out. Starting from the *client*, you should now be
+able to login to the *server* via the *endpoint*.
+
+ client:~ % ssh username@endpoint
+ endpoint:~ % ssh server
+ server:~ %
+
--------------------------------------------------------------------------------
## Xeon Phi Kernel Module ##
-It appears that Linux kernel version 4.19.0 included with Debian 10.9 already
+It appears that Linux kernel version 4.19.181 included with Debian 10.9 already
has some sort of in-tree kernel support for these Xeon Phi coprocessor cards as
-seen in the final lines of the following diagnostic output.
+seen in the final lines of the following diagnostic output. Also note that the
+card allocated an 8GB PCIe MMIO region, indicating that the 64-bit BAR setting
+in the BIOS is working as intended.
root@frostburg:~ # lspci | grep -i Co-processor
02:00.0 Co-processor: Intel Corporation Xeon Phi coprocessor 5100 series (rev 11)
root@frostburg:~ # lspci -s 02:00.0 -vv
02:00.0 Co-processor: Intel Corporation Xeon Phi coprocessor 5100 series (rev 11)
- Subsystem: Intel Corporation Xeon Phi coprocessor 5100 series
- Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
- Latency: 0, Cache Line Size: 64 bytes
- Interrupt: pin A routed to IRQ 69
- NUMA node: 0
+ <snip>
Region 0: Memory at 21c00000000 (64-bit, prefetchable) [size=8G]
- Region 4: Memory at cb900000 (64-bit, non-prefetchable) [size=128K]
- Capabilities: [44] Power Management version 3
- Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot-,D3cold-)
- Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
- Capabilities: [4c] Express (v2) Endpoint, MSI 00
- DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
- ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W
- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
- MaxPayload 256 bytes, MaxReadReq 512 bytes
- DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
- LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <4us, L1 unlimited
- ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
- LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
- DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
- Compliance De-emphasis: -6dB
- LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
- Capabilities: [88] MSI: Enable- Count=1/16 Maskable- 64bit+
- Address: 0000000000000000 Data: 0000
- Capabilities: [98] MSI-X: Enable+ Count=16 Masked-
- Vector table: BAR=4 offset=00017000
- PBA: BAR=4 offset=00018000
- Capabilities: [100 v1] Advanced Error Reporting
- UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
- UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
- AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
+ <snip>
Kernel driver in use: mic
Kernel modules: mic_host
-However, as no virtual network device automatically showed up, and since the
-Intel manuals are plastered with warnings about using exact, sanctioned
-combinations of kernel module, MPSS software, and Phi firmware, I decided to
-avoid the kernel module included with the system and instead attempt porting
-the kernel module source code included with MPSS onto a newer Linux kernel. At
-a minimum, it appears the timer API has changed, as well as some utility
-functions related to requesting block interrupt assignments.
+However, since the Intel manuals are plastered with warnings about using exact,
+sanctioned combinations of kernel module, MPSS software, and Phi firmware, I
+decided to avoid the kernel module included with the system and instead attempt
+porting the kernel module source code included with MPSS onto a newer Linux
+kernel. Once I have everything operational and understand how it *should* work,
+then I can try the open-source driver.
+
+I have updated the Intel kernel driver to work with newer Linux kernels. My
+work is based upon the kernel source included with MPSS 3.8.6, the latest/last
+release from Intel. Since the Xeon Phi x100 series is EOL, I don't think Intel
+intends to release any more versions of MPSS. Check `README.md` in my
+[xeon-phi-kernel-module](https://git.subgeniuskitty.com/xeon-phi-kernel-module/.git)
+git repo for up-to-date information regarding kernel version compatibility.
+
+Before compiling the kernel module, verify that relevant kernel headers are
+installed.
+
+ % uname -a
+ Linux frostburg 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux
+ % dpkg -l | grep linux-header
+ ii linux-headers-4.19.0-16-amd64 4.19.181-1 amd64 Header files for Linux 4.19.0-16-amd64
+ ii linux-headers-4.19.0-16-common 4.19.181-1 all Common header files for Linux 4.19.0-16
+ ii linux-headers-amd64 4.19+105+deb10u11 amd64 Header files for Linux amd64 configuration (meta-package)
+
+Download and compile my updated version of the Intel kernel driver. Sample
+compilation output is included below.
+
+ % git clone git://git.subgeniuskitty.com/xeon-phi-kernel-module/
+ % cd xeon-phi-kernel-module/
+ % make clean all
+ make -C /lib/modules/4.19.0-16-amd64/build M=xeon-phi-kernel-module modules \
+ INSTALL_MOD_PATH=
+ make[1]: Entering directory '/usr/src/linux-headers-4.19.0-16-amd64'
+ CC [M] xeon-phi-kernel-module/dma/mic_dma_lib.o
+ CC [M] xeon-phi-kernel-module/dma/mic_dma_md.o
+ CC [M] xeon-phi-kernel-module/host/acptboot.o
+ CC [M] xeon-phi-kernel-module/host/ioctl.o
+ CC [M] xeon-phi-kernel-module/host/linpm.o
+ CC [M] xeon-phi-kernel-module/host/linpsmi.o
+ CC [M] xeon-phi-kernel-module/host/linscif_host.o
+ CC [M] xeon-phi-kernel-module/host/linsysfs.o
+ CC [M] xeon-phi-kernel-module/host/linux.o
+ CC [M] xeon-phi-kernel-module/host/linvcons.o
+ CC [M] xeon-phi-kernel-module/host/linvnet.o
+ CC [M] xeon-phi-kernel-module/host/micpsmi.o
+ CC [M] xeon-phi-kernel-module/host/micscif_pm.o
+ CC [M] xeon-phi-kernel-module/host/pm_ioctl.o
+ CC [M] xeon-phi-kernel-module/host/pm_pcstate.o
+ CC [M] xeon-phi-kernel-module/host/tools_support.o
+ CC [M] xeon-phi-kernel-module/host/uos_download.o
+ CC [M] xeon-phi-kernel-module/host/vhost/mic_vhost.o
+ CC [M] xeon-phi-kernel-module/host/vhost/mic_blk.o
+ CC [M] xeon-phi-kernel-module/host/vmcore.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_api.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_debug.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_fd.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_intr.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_nm.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_nodeqp.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_ports.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_rb.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_rma_dma.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_rma_list.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_rma.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_select.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_smpt.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_sysfs.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_va_gen.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_va_node.o
+ CC [M] xeon-phi-kernel-module/vnet/micveth_dma.o
+ CC [M] xeon-phi-kernel-module/vnet/micveth_param.o
+ LD [M] xeon-phi-kernel-module/mic.o
+ Building modules, stage 2.
+ MODPOST 1 modules
+ CC xeon-phi-kernel-module/mic.mod.o
+ LD [M] xeon-phi-kernel-module/mic.ko
+ make[1]: Leaving directory '/usr/src/linux-headers-4.19.0-16-amd64'
+
+At this point you can manually load/install the new kernel module (`mic.ko`)
+which is found in the current directory, or execute `make install`. The latter
+command also installs the SCIF header file, as well as putting some config files
+under `/usr/local/etc/`. The information in those config files won't be picked
+up by the system (we will install configs in the correct location in a moment),
+but it is useful as a reference. Sample `make install` output is shown below.
+
+ # make install
+ make -C /lib/modules/4.19.0-16-amd64/build M=/home/ataylor/xeon-phi-kernel-module modules_install \
+ INSTALL_MOD_PATH=
+ make[1]: Entering directory '/usr/src/linux-headers-4.19.0-16-amd64'
+ INSTALL /home/ataylor/xeon-phi-kernel-module/mic.ko
+ DEPMOD 4.19.0-16-amd64
+ Warning: modules_install: missing 'System.map' file. Skipping depmod.
+ make[1]: Leaving directory '/usr/src/linux-headers-4.19.0-16-amd64'
+ install -d /usr/local/etc/sysconfig/modules
+ install mic.modules /usr/local/etc/sysconfig/modules
+ install -d /usr/local/etc/modprobe.d
+ install -m644 mic.conf /usr/local/etc/modprobe.d
+ install -d /usr/local/etc/udev/rules.d
+ install -m644 udev-mic.rules /usr/local/etc/udev/rules.d/50-udev-mic.rules
+ install -d /lib/modules/4.19.0-16-amd64
+ install -m644 Module.symvers /lib/modules/4.19.0-16-amd64/scif.symvers
+ install -d /usr/src/linux-headers-4.19.0-16-amd64/include/modules
+ install -m644 include/scif.h /usr/src/linux-headers-4.19.0-16-amd64/include/modules
+
+Create the file `/etc/modprobe.d/mic.conf` with the following contents,
+intended to accomplish two things. First, blacklist the in-tree MIC kernel
+module that shipped with our kernel, including all associated modules, and
+second, configure the Intel MIC kernel module which we just built and installed.
+The options shown are drawn from the defaults in
+`/usr/local/etc/modprobe.d/mic.conf`.
+
+ # Blacklist the in-tree kernel modules associated with the Knight's Corner Xeon
+ # Phi so that we can load the Intel kernel module.
+
+ # These two modules depend on the various bus modules that follow.
+ blacklist mic_host
+ blacklist mic_x100_dma
+
+ blacklist cosm_bus
+ blacklist vop_bus
+ blacklist scif_bus
+ blacklist mic_bus
+
+ # ^^^------ Blacklisting the in-tree MIC kernel module.
+ # ==============================================================================
+ # vvv------ Configuring the Intel MIC kernel module.
+
+ # The following options apply to the Intel Many Integrated Core (MIC) driver.
+ # Unless otherwise noted, the value "1" enables the feature and "0" disables
+ # it.
+ #
+ # Option: p2p
+ # Description: Enables use of SCIF interface peer to peer communication.
+ #
+ # Option: p2p_proxy
+ # Description: Enables use of SCIF P2P Proxy DMA which converts DMA
+ # reads into DMA writes for performance on certain Intel
+ # platforms.
+ #
+ # Option: reg_cache
+ # Description: Enables SCIF Registration Caching.
+ #
+ # Option: huge_page
+ # Description: Enables SCIF Huge Page Support.
+ #
+ # Option: watchdog
+ # Description: Enables SCIF watchdog for Lost Node detection.
+ #
+ # Option: watchdog_auto_reboot
+ # Description: Configures behavior of MIC host driver upon detection of a lost
+ # node. This option is a nop if watchdog=0. Setting value "1"
+ # allows host driver to reboot node back to "online" state,
+ # whereas value "0" only allows the host driver to reset the node
+ # back to "ready" state, leaving the user responsible for rebooting
+ # the node (or not).
+ #
+ # Option: crash_dump
+ # Description: Enables uOS Kernel Crash Dump Captures.
+ #
+ # Option: ulimit
+ # Description: Enables ulimit checks on max locked memory for scif_register.
+ #
+ options mic reg_cache=1 huge_page=1 watchdog=1 watchdog_auto_reboot=1 crash_dump=1 p2p=1 p2p_proxy=1 ulimit=0
+ options mic_host reg_cache=1 huge_page=1 watchdog=1 watchdog_auto_reboot=1 crash_dump=1 p2p=1 p2p_proxy=1 ulimit=0
+
+Finally, add the line `mic` to the file `/etc/modules-load.d/modules.conf`,
+instructing the system to load this kernel module on boot, then run `depmod` to
+ensure the system is aware of the new kernel module, followed by a reboot to
+verify everything works.
+
+After the system comes back up, verify that the module loaded with your desired
+options using the `systool` command, sample output below.
+
+ # systool -v -m mic
+ Module = "mic"
+
+ Attributes:
+ coresize = "741376"
+ initsize = "0"
+ initstate = "live"
+ refcnt = "0"
+ taint = "OE"
+ uevent = <store method only>
+
+ Parameters:
+ crash_dump = "Y"
+ huge_page = "Y"
+ msi = "Y"
+ p2p_proxy = "Y"
+ p2p = "Y"
+ pm_qos_cpu_dma_lat = "-1"
+ psmi = "N"
+ ramoops_count = "4"
+ reg_cache = "Y"
+ ulimit = "N"
+ vnet = "dma"
+ vnet_addr = "0"
+ vnet_num_buffers = "62"
+ watchdog_auto_reboot= "Y"
+ watchdog = "Y"
+
+ Sections:
+ <snip>
+
+
+--------------------------------------------------------------------------------
+
+
+## Intel MPSS ##