X-Git-Url: http://git.subgeniuskitty.com/website_subgeniuskitty.com/.git/blobdiff_plain/50ab15738e8dd6ea084e0944a35c9b9c07cc66df..8ccbbc343858f80f1b0e5dcb6dfe6c0fd2a4d2ce:/data/notes/xeon_phi_server.md diff --git a/data/notes/xeon_phi_server.md b/data/notes/xeon_phi_server.md index 6682fb2..5d448b9 100644 --- a/data/notes/xeon_phi_server.md +++ b/data/notes/xeon_phi_server.md @@ -29,7 +29,7 @@ The information on this page includes: - Installation of Debian 10.9 (buster) root on encrypted ZFS mirror with automated snapshots and scrubs. - - (TODO) Porting the Intel kernel module to Linux kernel version 4.19.0. + - Porting the Xeon Phi kernel module to newer versions of the Linux kernel. - (TODO) Installing MPSS toolkit on Debian (or CentOS VM). @@ -564,7 +564,8 @@ Reboot the computer and remove the USB stick. Installation is complete. Install various no-config-required userland packages before continuing. apt-get install net-tools bzip2 zip ntp htop xterm screen git \ - build-essential pciutils smartmontools gdb valgrind + build-essential pciutils smartmontools gdb valgrind wget \ + texlive texlive-latex-extra graphviz firefox sysfsutils #### X Window Manager #### @@ -658,7 +659,7 @@ permissions so it's used for any future users on the system. alias j jobs -l alias la ls -aF alias lf ls -FA - alias ll ls -lAF --color + alias ll ls -lF --color alias ls ls --color # These are normally set through /etc/login.conf. You may override them here @@ -705,6 +706,24 @@ Add the following line to `~/.xinitrc`. /bin/xscreensaver -nosplash & +#### Go Toolchain #### + +The version of Go provided via `apt-get` is always out of date, so all Go +installs on this server are done via tarball from the +website. Go 1.16.3 is used for this example but the newest version of Go may be +found at . + +Previous versions of Go are installed entirely under `/usr/local/go`. Delete +the entire `/usr/local/go` directory before proceeding. + + wget https://golang.org/dl/go1.16.3.linux-amd64.tar.gz + tar -C /usr/local -xzf go1.16.3.linux-amd64.tar.gz + +If this is the first time installing Go on the system, update everyone's +`$PATH` to include `/usr/local/go/bin`. Remember to update files under +`/etc/skel` at the same time. + + #### ZFS Snapshots #### In order to configure automatic ZFS snapshots, use the `auto-zfs-snapshot` @@ -772,69 +791,367 @@ Create the file `/etc/cron.d/status-emails` with the following contents. SHELL=/bin/bash 0 0 * * 0 root /sbin/zpool status | echo -e "Subject:FROSTBURG: zpool status\n\n $(cat -)" | msmtp ataylor@subgeniuskitty.com + +#### IRC Environment #### + +IRC is used for collaboration on the server. First install daemon and client. + + apt-get install ngircd irssi + +Configure the server by editing `/etc/ngircd/ngircd.conf`. The defaults are +mostly acceptable but the server must be given a name and restricted to only +listen for local connections. While we're at it, the max nick length is only 9 +by default and should be increased. Note that these values need to be inserted +under the appropriate category, as shown below, but the categories already +exist in the config file. + + [Global] + Name = frostburg.subgeniuskitty.com + Info = Frostburg - Private IRC Server + Listen = 127.0.0.1 + [Limits] + MaxNickLength = 32 + +Restart the server and verify it listens on the correct addresses. + + # systemctl restart ngircd + # netstat -an | grep LISTEN + tcp 0 0 127.0.0.1:6667 0.0.0.0:* LISTEN + +Startup a client in screen for each user. + + screen -dR irc + irssi + /connect localhost + /join #channel + + +#### Public SSH Access #### + +Although frostburg is on a private subnet, I want public SSH access. The +easiest way to set this up is via a reverse SSH tunnel to one of the public +subgeniuskitty.com servers. + +This section refers to three machines: + + - The **server** is frostburg.subgeniuskitty.com, a machine which we desire + to access across the internet despite residing on a private subnet. + + - The **endpoint** is a server with public IP address which will serve as an + access portal for the *server*. + + - The **client** is the human user's workstation, the machine which is + attempting to login to the *server* via the *endpoint*. + +First, setup appropriate login credentials on the *server*, which in this case +is `frostburg.subgeniuskitty.com`. Ignore any warnings about `/home/username` +already existing or not being owned by the correct user. These are simply a +side effect of using ZFS since we must create the homedir before adding the +user, but we can't change ownership until after the new user exists. + + server:~ # zfs create rpool/home/username + server:~ # adduser username + server:~ # cp -a /etc/skel/. /home/username + server:~ # chown -R username:username /home/username + server:~ # zfs snapshot rpoot/home/username@account_creation + +If necessary for the intended tasks, add the user to any relevant groups with +something like the following command. + + server:~ # usermod -a -G netdev,plugdev,sudo,video username + +The user will also need login credentials on the *endpoint*. These credentials +don't need to allow anything other than simply SSHing through to the *server*. + + endpoint:~ # adduser username + +With appropriate credentials successfully created, move on to setting up a +reverse SSH tunnel from *server* to *endpoint*. + +First, create an SSH key on the *server* with no passphrase and authorize it +for logins on the *endpoint*. This will be used to bring the tunnel up when the +machine boots. If a non-empty passphrase is specified, you will need to type it +during the boot process. + + server:~ # ssh-keygen + server:~ # scp /root/.ssh/id_rsa.pub username@endpoint:/home/username/temp_key_file + server:~ # ssh username@endpoint + (login requires password) + endpoint:~ % mkdir -p /home/username/.ssh + endpoint:~ % mv /home/username/temp_key_file /home/username/.ssh/authorized_keys + endpoint:~ % logout + server:~ # ssh username@endpoint + (login does not require password) + endpoint:~ % logout + server:~ # mv /root/.ssh/id_rsa rtunnel_nopwd + server:~ # mv /root/.ssh/id_rsa.pub rtunnel_nopwd.pub + +Next, create the tunnel using AutoSSH to maintain a long-term connection. + + server:~ # apt-get install autossh + server:~ # vi /etc/systemd/system/autossh-tunnel.service + [Unit] + Description=AutoSSH tunnel between frostburg.SGK and www.SGK + After=network-online.target + + [Service] + Environment="AUTOSSH_GATETIME=0" + ExecStart=/bin/autossh -N -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -i /root/.ssh/rtunnel_nopwd -R 4242:localhost:22 username@endpoint + + [Install] + WantedBy=multi-user.target + server:~ # systemctl daemon-reload + server:~ # systemctl start autossh-tunnel.service + server:~ # systemctl enable autossh-tunnel.service + +At this point the SSH tunnel is operational. Let's make things a little easier +for the user by storing most of the config options in an SSH config file. + + endpoint:~ # su - username + endpoint:~ % vi /home/username/.ssh/config + Host server + Hostname localhost + User username + Port 4242 + +Now, when we execute `ssh server`, it is equivalent to the command +`ssh -p 4242 username@localhost`, much easier to remember. + +It's time to test everything out. Starting from the *client*, you should now be +able to login to the *server* via the *endpoint*. + + client:~ % ssh username@endpoint + endpoint:~ % ssh server + server:~ % + -------------------------------------------------------------------------------- ## Xeon Phi Kernel Module ## -It appears that Linux kernel version 4.19.0 included with Debian 10.9 already +It appears that Linux kernel version 4.19.181 included with Debian 10.9 already has some sort of in-tree kernel support for these Xeon Phi coprocessor cards as -seen in the final lines of the following diagnostic output. +seen in the final lines of the following diagnostic output. Also note that the +card allocated an 8GB PCIe MMIO region, indicating that the 64-bit BAR setting +in the BIOS is working as intended. root@frostburg:~ # lspci | grep -i Co-processor 02:00.0 Co-processor: Intel Corporation Xeon Phi coprocessor 5100 series (rev 11) root@frostburg:~ # lspci -s 02:00.0 -vv 02:00.0 Co-processor: Intel Corporation Xeon Phi coprocessor 5100 series (rev 11) - Subsystem: Intel Corporation Xeon Phi coprocessor 5100 series - Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ - Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Region 0: Memory at 21c00000000 (64-bit, prefetchable) [size=8G] - Region 4: Memory at cb900000 (64-bit, non-prefetchable) [size=128K] - Capabilities: [44] Power Management version 3 - Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot-,D3cold-) - Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- - Capabilities: [4c] Express (v2) Endpoint, MSI 00 - DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us - ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W - DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- - RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ - MaxPayload 256 bytes, MaxReadReq 512 bytes - DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- - LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <4us, L1 unlimited - ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- - LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ - ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- - LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- - DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported - DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled - LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- - Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- - Compliance De-emphasis: -6dB - LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- - EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- - Capabilities: [88] MSI: Enable- Count=1/16 Maskable- 64bit+ - Address: 0000000000000000 Data: 0000 - Capabilities: [98] MSI-X: Enable+ Count=16 Masked- - Vector table: BAR=4 offset=00017000 - PBA: BAR=4 offset=00018000 - Capabilities: [100 v1] Advanced Error Reporting - UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- - UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- - UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- - CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- - CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ - AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- + Kernel driver in use: mic Kernel modules: mic_host -However, as no virtual network device automatically showed up, and since the -Intel manuals are plastered with warnings about using exact, sanctioned -combinations of kernel module, MPSS software, and Phi firmware, I decided to -avoid the kernel module included with the system and instead attempt porting -the kernel module source code included with MPSS onto a newer Linux kernel. At -a minimum, it appears the timer API has changed, as well as some utility -functions related to requesting block interrupt assignments. +However, since the Intel manuals are plastered with warnings about using exact, +sanctioned combinations of kernel module, MPSS software, and Phi firmware, I +decided to avoid the kernel module included with the system and instead attempt +porting the kernel module source code included with MPSS onto a newer Linux +kernel. Once I have everything operational and understand how it *should* work, +then I can try the open-source driver. + +I have updated the Intel kernel driver to work with newer Linux kernels. My +work is based upon the kernel source included with MPSS 3.8.6, the latest/last +release from Intel. Since the Xeon Phi x100 series is EOL, I don't think Intel +intends to release any more versions of MPSS. Check `README.md` in my +[xeon-phi-kernel-module](https://git.subgeniuskitty.com/xeon-phi-kernel-module/.git) +git repo for up-to-date information regarding kernel version compatibility. + +Before compiling the kernel module, verify that relevant kernel headers are +installed. + + % uname -a + Linux frostburg 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux + % dpkg -l | grep linux-header + ii linux-headers-4.19.0-16-amd64 4.19.181-1 amd64 Header files for Linux 4.19.0-16-amd64 + ii linux-headers-4.19.0-16-common 4.19.181-1 all Common header files for Linux 4.19.0-16 + ii linux-headers-amd64 4.19+105+deb10u11 amd64 Header files for Linux amd64 configuration (meta-package) + +Download and compile my updated version of the Intel kernel driver. Sample +compilation output is included below. + + % git clone git://git.subgeniuskitty.com/xeon-phi-kernel-module/ + % cd xeon-phi-kernel-module/ + % make clean all + make -C /lib/modules/4.19.0-16-amd64/build M=xeon-phi-kernel-module modules \ + INSTALL_MOD_PATH= + make[1]: Entering directory '/usr/src/linux-headers-4.19.0-16-amd64' + CC [M] xeon-phi-kernel-module/dma/mic_dma_lib.o + CC [M] xeon-phi-kernel-module/dma/mic_dma_md.o + CC [M] xeon-phi-kernel-module/host/acptboot.o + CC [M] xeon-phi-kernel-module/host/ioctl.o + CC [M] xeon-phi-kernel-module/host/linpm.o + CC [M] xeon-phi-kernel-module/host/linpsmi.o + CC [M] xeon-phi-kernel-module/host/linscif_host.o + CC [M] xeon-phi-kernel-module/host/linsysfs.o + CC [M] xeon-phi-kernel-module/host/linux.o + CC [M] xeon-phi-kernel-module/host/linvcons.o + CC [M] xeon-phi-kernel-module/host/linvnet.o + CC [M] xeon-phi-kernel-module/host/micpsmi.o + CC [M] xeon-phi-kernel-module/host/micscif_pm.o + CC [M] xeon-phi-kernel-module/host/pm_ioctl.o + CC [M] xeon-phi-kernel-module/host/pm_pcstate.o + CC [M] xeon-phi-kernel-module/host/tools_support.o + CC [M] xeon-phi-kernel-module/host/uos_download.o + CC [M] xeon-phi-kernel-module/host/vhost/mic_vhost.o + CC [M] xeon-phi-kernel-module/host/vhost/mic_blk.o + CC [M] xeon-phi-kernel-module/host/vmcore.o + CC [M] xeon-phi-kernel-module/micscif/micscif_api.o + CC [M] xeon-phi-kernel-module/micscif/micscif_debug.o + CC [M] xeon-phi-kernel-module/micscif/micscif_fd.o + CC [M] xeon-phi-kernel-module/micscif/micscif_intr.o + CC [M] xeon-phi-kernel-module/micscif/micscif_nm.o + CC [M] xeon-phi-kernel-module/micscif/micscif_nodeqp.o + CC [M] xeon-phi-kernel-module/micscif/micscif_ports.o + CC [M] xeon-phi-kernel-module/micscif/micscif_rb.o + CC [M] xeon-phi-kernel-module/micscif/micscif_rma_dma.o + CC [M] xeon-phi-kernel-module/micscif/micscif_rma_list.o + CC [M] xeon-phi-kernel-module/micscif/micscif_rma.o + CC [M] xeon-phi-kernel-module/micscif/micscif_select.o + CC [M] xeon-phi-kernel-module/micscif/micscif_smpt.o + CC [M] xeon-phi-kernel-module/micscif/micscif_sysfs.o + CC [M] xeon-phi-kernel-module/micscif/micscif_va_gen.o + CC [M] xeon-phi-kernel-module/micscif/micscif_va_node.o + CC [M] xeon-phi-kernel-module/vnet/micveth_dma.o + CC [M] xeon-phi-kernel-module/vnet/micveth_param.o + LD [M] xeon-phi-kernel-module/mic.o + Building modules, stage 2. + MODPOST 1 modules + CC xeon-phi-kernel-module/mic.mod.o + LD [M] xeon-phi-kernel-module/mic.ko + make[1]: Leaving directory '/usr/src/linux-headers-4.19.0-16-amd64' + +At this point you can manually load/install the new kernel module (`mic.ko`) +which is found in the current directory, or execute `make install`. The latter +command also installs the SCIF header file, as well as putting some config files +under `/usr/local/etc/`. The information in those config files won't be picked +up by the system (we will install configs in the correct location in a moment), +but it is useful as a reference. Sample `make install` output is shown below. + + # make install + make -C /lib/modules/4.19.0-16-amd64/build M=/home/ataylor/xeon-phi-kernel-module modules_install \ + INSTALL_MOD_PATH= + make[1]: Entering directory '/usr/src/linux-headers-4.19.0-16-amd64' + INSTALL /home/ataylor/xeon-phi-kernel-module/mic.ko + DEPMOD 4.19.0-16-amd64 + Warning: modules_install: missing 'System.map' file. Skipping depmod. + make[1]: Leaving directory '/usr/src/linux-headers-4.19.0-16-amd64' + install -d /usr/local/etc/sysconfig/modules + install mic.modules /usr/local/etc/sysconfig/modules + install -d /usr/local/etc/modprobe.d + install -m644 mic.conf /usr/local/etc/modprobe.d + install -d /usr/local/etc/udev/rules.d + install -m644 udev-mic.rules /usr/local/etc/udev/rules.d/50-udev-mic.rules + install -d /lib/modules/4.19.0-16-amd64 + install -m644 Module.symvers /lib/modules/4.19.0-16-amd64/scif.symvers + install -d /usr/src/linux-headers-4.19.0-16-amd64/include/modules + install -m644 include/scif.h /usr/src/linux-headers-4.19.0-16-amd64/include/modules + +Create the file `/etc/modprobe.d/mic.conf` with the following contents, +intended to accomplish two things. First, blacklist the in-tree MIC kernel +module that shipped with our kernel, including all associated modules, and +second, configure the Intel MIC kernel module which we just built and installed. +The options shown are drawn from the defaults in +`/usr/local/etc/modprobe.d/mic.conf`. + + # Blacklist the in-tree kernel modules associated with the Knight's Corner Xeon + # Phi so that we can load the Intel kernel module. + + # These two modules depend on the various bus modules that follow. + blacklist mic_host + blacklist mic_x100_dma + + blacklist cosm_bus + blacklist vop_bus + blacklist scif_bus + blacklist mic_bus + + # ^^^------ Blacklisting the in-tree MIC kernel module. + # ============================================================================== + # vvv------ Configuring the Intel MIC kernel module. + + # The following options apply to the Intel Many Integrated Core (MIC) driver. + # Unless otherwise noted, the value "1" enables the feature and "0" disables + # it. + # + # Option: p2p + # Description: Enables use of SCIF interface peer to peer communication. + # + # Option: p2p_proxy + # Description: Enables use of SCIF P2P Proxy DMA which converts DMA + # reads into DMA writes for performance on certain Intel + # platforms. + # + # Option: reg_cache + # Description: Enables SCIF Registration Caching. + # + # Option: huge_page + # Description: Enables SCIF Huge Page Support. + # + # Option: watchdog + # Description: Enables SCIF watchdog for Lost Node detection. + # + # Option: watchdog_auto_reboot + # Description: Configures behavior of MIC host driver upon detection of a lost + # node. This option is a nop if watchdog=0. Setting value "1" + # allows host driver to reboot node back to "online" state, + # whereas value "0" only allows the host driver to reset the node + # back to "ready" state, leaving the user responsible for rebooting + # the node (or not). + # + # Option: crash_dump + # Description: Enables uOS Kernel Crash Dump Captures. + # + # Option: ulimit + # Description: Enables ulimit checks on max locked memory for scif_register. + # + options mic reg_cache=1 huge_page=1 watchdog=1 watchdog_auto_reboot=1 crash_dump=1 p2p=1 p2p_proxy=1 ulimit=0 + options mic_host reg_cache=1 huge_page=1 watchdog=1 watchdog_auto_reboot=1 crash_dump=1 p2p=1 p2p_proxy=1 ulimit=0 + +Finally, add the line `mic` to the file `/etc/modules-load.d/modules.conf`, +instructing the system to load this kernel module on boot, then run `depmod` to +ensure the system is aware of the new kernel module, followed by a reboot to +verify everything works. + +After the system comes back up, verify that the module loaded with your desired +options using the `systool` command, sample output below. + + # systool -v -m mic + Module = "mic" + + Attributes: + coresize = "741376" + initsize = "0" + initstate = "live" + refcnt = "0" + taint = "OE" + uevent = + + Parameters: + crash_dump = "Y" + huge_page = "Y" + msi = "Y" + p2p_proxy = "Y" + p2p = "Y" + pm_qos_cpu_dma_lat = "-1" + psmi = "N" + ramoops_count = "4" + reg_cache = "Y" + ulimit = "N" + vnet = "dma" + vnet_addr = "0" + vnet_num_buffers = "62" + watchdog_auto_reboot= "Y" + watchdog = "Y" + + Sections: + + + +-------------------------------------------------------------------------------- + + +## Intel MPSS ##