+These instructions apply to Debian 10 (buster). Use them as a reference,
+adapting them to your specific machine.
+
+Before compiling the kernel module, verify that relevant kernel headers are
+installed.
+
+ % uname -a
+ Linux frostburg 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux
+ % dpkg -l | grep linux-header
+ ii linux-headers-4.19.0-16-amd64 4.19.181-1 amd64 Header files for Linux 4.19.0-16-amd64
+ ii linux-headers-4.19.0-16-common 4.19.181-1 all Common header files for Linux 4.19.0-16
+ ii linux-headers-amd64 4.19+105+deb10u11 amd64 Header files for Linux amd64 configuration (meta-package)
+
+Now download and compile the updated version of the Intel kernel driver found
+in this repository. Sample compilation output is included below.
+
+ % git clone git://git.subgeniuskitty.com/xeon-phi-kernel-module/
+ % cd xeon-phi-kernel-module/
+ % make clean all
+ make -C /lib/modules/4.19.0-16-amd64/build M=xeon-phi-kernel-module modules \
+ INSTALL_MOD_PATH=
+ make[1]: Entering directory '/usr/src/linux-headers-4.19.0-16-amd64'
+ CC [M] xeon-phi-kernel-module/dma/mic_dma_lib.o
+ CC [M] xeon-phi-kernel-module/dma/mic_dma_md.o
+ CC [M] xeon-phi-kernel-module/host/acptboot.o
+ CC [M] xeon-phi-kernel-module/host/ioctl.o
+ CC [M] xeon-phi-kernel-module/host/linpm.o
+ CC [M] xeon-phi-kernel-module/host/linpsmi.o
+ CC [M] xeon-phi-kernel-module/host/linscif_host.o
+ CC [M] xeon-phi-kernel-module/host/linsysfs.o
+ CC [M] xeon-phi-kernel-module/host/linux.o
+ CC [M] xeon-phi-kernel-module/host/linvcons.o
+ CC [M] xeon-phi-kernel-module/host/linvnet.o
+ CC [M] xeon-phi-kernel-module/host/micpsmi.o
+ CC [M] xeon-phi-kernel-module/host/micscif_pm.o
+ CC [M] xeon-phi-kernel-module/host/pm_ioctl.o
+ CC [M] xeon-phi-kernel-module/host/pm_pcstate.o
+ CC [M] xeon-phi-kernel-module/host/tools_support.o
+ CC [M] xeon-phi-kernel-module/host/uos_download.o
+ CC [M] xeon-phi-kernel-module/host/vhost/mic_vhost.o
+ CC [M] xeon-phi-kernel-module/host/vhost/mic_blk.o
+ CC [M] xeon-phi-kernel-module/host/vmcore.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_api.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_debug.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_fd.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_intr.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_nm.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_nodeqp.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_ports.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_rb.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_rma_dma.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_rma_list.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_rma.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_select.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_smpt.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_sysfs.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_va_gen.o
+ CC [M] xeon-phi-kernel-module/micscif/micscif_va_node.o
+ CC [M] xeon-phi-kernel-module/vnet/micveth_dma.o
+ CC [M] xeon-phi-kernel-module/vnet/micveth_param.o
+ LD [M] xeon-phi-kernel-module/mic.o
+ Building modules, stage 2.
+ MODPOST 1 modules
+ CC xeon-phi-kernel-module/mic.mod.o
+ LD [M] xeon-phi-kernel-module/mic.ko
+ make[1]: Leaving directory '/usr/src/linux-headers-4.19.0-16-amd64'
+
+At this point you can manually load/install the new kernel module (`mic.ko`)
+which is found in the current directory, or you can `make install`. The latter
+command also installs the SCIF header file, as well as putting some config files
+under `/usr/local/etc/`. The information in those config files won't be picked
+up by the system (we will install configs in the correct location in a moment),
+but it is useful as a reference. Sample `make install` output is shown below.
+
+ # make install
+ make -C /lib/modules/4.19.0-16-amd64/build M=/home/ataylor/xeon-phi-kernel-module modules_install \
+ INSTALL_MOD_PATH=
+ make[1]: Entering directory '/usr/src/linux-headers-4.19.0-16-amd64'
+ INSTALL /home/ataylor/xeon-phi-kernel-module/mic.ko
+ DEPMOD 4.19.0-16-amd64
+ Warning: modules_install: missing 'System.map' file. Skipping depmod.
+ make[1]: Leaving directory '/usr/src/linux-headers-4.19.0-16-amd64'
+ install -d /usr/local/etc/sysconfig/modules
+ install mic.modules /usr/local/etc/sysconfig/modules
+ install -d /usr/local/etc/modprobe.d
+ install -m644 mic.conf /usr/local/etc/modprobe.d
+ install -d /usr/local/etc/udev/rules.d
+ install -m644 udev-mic.rules /usr/local/etc/udev/rules.d/50-udev-mic.rules
+ install -d /lib/modules/4.19.0-16-amd64
+ install -m644 Module.symvers /lib/modules/4.19.0-16-amd64/scif.symvers
+ install -d /usr/src/linux-headers-4.19.0-16-amd64/include/modules
+ install -m644 include/scif.h /usr/src/linux-headers-4.19.0-16-amd64/include/modules
+
+Create the file `/etc/modprobe.d/mic.conf` with the following contents,
+intended to accomplish two things. First, blacklist the in-tree MIC kernel
+module that shipped with our kernel, including all associated modules, and
+second, configure the Intel MIC kernel module which we just built and installed.
+The options shown are drawn from the defaults in
+`/usr/local/etc/modprobe.d/mic.conf`.
+
+ # Blacklist the in-tree kernel modules associated with the Knight's Corner Xeon
+ # Phi so that we can load the Intel kernel module.
+
+ # These two modules depend on the various bus modules that follow.
+ blacklist mic_host
+ blacklist mic_x100_dma
+
+ blacklist cosm_bus
+ blacklist vop_bus
+ blacklist scif_bus
+ blacklist mic_bus
+
+ # ^^^------ Blacklisting the in-tree MIC kernel module.
+ # ==============================================================================
+ # vvv------ Configuring the Intel MIC kernel module.
+
+ # The following options apply to the Intel Many Integrated Core (MIC) driver.
+ # Unless otherwise noted, the value "1" enables the feature and "0" disables
+ # it.
+ #
+ # Option: p2p
+ # Description: Enables use of SCIF interface peer to peer communication.
+ #
+ # Option: p2p_proxy
+ # Description: Enables use of SCIF P2P Proxy DMA which converts DMA
+ # reads into DMA writes for performance on certain Intel
+ # platforms.
+ #
+ # Option: reg_cache
+ # Description: Enables SCIF Registration Caching.
+ #
+ # Option: huge_page
+ # Description: Enables SCIF Huge Page Support.
+ #
+ # Option: watchdog
+ # Description: Enables SCIF watchdog for Lost Node detection.
+ #
+ # Option: watchdog_auto_reboot
+ # Description: Configures behavior of MIC host driver upon detection of a lost
+ # node. This option is a nop if watchdog=0. Setting value "1"
+ # allows host driver to reboot node back to "online" state,
+ # whereas value "0" only allows the host driver to reset the node
+ # back to "ready" state, leaving the user responsible for rebooting
+ # the node (or not).
+ #
+ # Option: crash_dump
+ # Description: Enables uOS Kernel Crash Dump Captures.
+ #
+ # Option: ulimit
+ # Description: Enables ulimit checks on max locked memory for scif_register.
+ #
+ options mic reg_cache=1 huge_page=1 watchdog=1 watchdog_auto_reboot=1 crash_dump=1 p2p=1 p2p_proxy=1 ulimit=0
+ options mic_host reg_cache=1 huge_page=1 watchdog=1 watchdog_auto_reboot=1 crash_dump=1 p2p=1 p2p_proxy=1 ulimit=0
+
+Finally, add the line `mic` to the file `/etc/modules-load.d/modules.conf`,
+instructing the system to load this kernel module on boot, then run `depmod` to
+ensure the system is aware of the new kernel module, followed by a reboot to
+verify everything works.
+
+After the system comes back up, verify that the module loaded with your desired
+options using the `systool` command, sample output below.
+
+ # systool -v -m mic
+ Module = "mic"
+
+ Attributes:
+ coresize = "741376"
+ initsize = "0"
+ initstate = "live"
+ refcnt = "0"
+ taint = "OE"
+ uevent = <store method only>
+
+ Parameters:
+ crash_dump = "Y"
+ huge_page = "Y"
+ msi = "Y"
+ p2p_proxy = "Y"
+ p2p = "Y"
+ pm_qos_cpu_dma_lat = "-1"
+ psmi = "N"
+ ramoops_count = "4"
+ reg_cache = "Y"
+ ulimit = "N"
+ vnet = "dma"
+ vnet_addr = "0"
+ vnet_num_buffers = "62"
+ watchdog_auto_reboot= "Y"
+ watchdog = "Y"
+
+ Sections:
+ <snip>
projects / xeon-phi-kernel-module / commitdiff