This repository contains source code for the Intel kernel module distributed with MPSS 3.8.6 for the Xeon Phi x100 family (Knights Corner). It has been modified for compatibility with newer Linux kernels.
Verified to build and run on Debian 10.9 (buster) with Linux kernel 4.19.181-1 and Intel Xeon Phi 5110P cards.
These instructions apply to Debian 10 (buster). Use them as a reference, adapting them to your specific machine.
Before compiling the kernel module, verify that relevant kernel headers are installed.
% uname -a
Linux frostburg 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux
% dpkg -l | grep linux-header
ii linux-headers-4.19.0-16-amd64 4.19.181-1 amd64 Header files for Linux 4.19.0-16-amd64
ii linux-headers-4.19.0-16-common 4.19.181-1 all Common header files for Linux 4.19.0-16
ii linux-headers-amd64 4.19+105+deb10u11 amd64 Header files for Linux amd64 configuration (meta-package)
Now download and compile the updated version of the Intel kernel driver found in this repository. Sample compilation output is included below.
% git clone git://git.subgeniuskitty.com/xeon-phi-kernel-module/
% cd xeon-phi-kernel-module/
% make clean all
make -C /lib/modules/4.19.0-16-amd64/build M=xeon-phi-kernel-module modules \
INSTALL_MOD_PATH=
make[1]: Entering directory '/usr/src/linux-headers-4.19.0-16-amd64'
CC [M] xeon-phi-kernel-module/dma/mic_dma_lib.o
CC [M] xeon-phi-kernel-module/dma/mic_dma_md.o
CC [M] xeon-phi-kernel-module/host/acptboot.o
CC [M] xeon-phi-kernel-module/host/ioctl.o
CC [M] xeon-phi-kernel-module/host/linpm.o
CC [M] xeon-phi-kernel-module/host/linpsmi.o
CC [M] xeon-phi-kernel-module/host/linscif_host.o
CC [M] xeon-phi-kernel-module/host/linsysfs.o
CC [M] xeon-phi-kernel-module/host/linux.o
CC [M] xeon-phi-kernel-module/host/linvcons.o
CC [M] xeon-phi-kernel-module/host/linvnet.o
CC [M] xeon-phi-kernel-module/host/micpsmi.o
CC [M] xeon-phi-kernel-module/host/micscif_pm.o
CC [M] xeon-phi-kernel-module/host/pm_ioctl.o
CC [M] xeon-phi-kernel-module/host/pm_pcstate.o
CC [M] xeon-phi-kernel-module/host/tools_support.o
CC [M] xeon-phi-kernel-module/host/uos_download.o
CC [M] xeon-phi-kernel-module/host/vhost/mic_vhost.o
CC [M] xeon-phi-kernel-module/host/vhost/mic_blk.o
CC [M] xeon-phi-kernel-module/host/vmcore.o
CC [M] xeon-phi-kernel-module/micscif/micscif_api.o
CC [M] xeon-phi-kernel-module/micscif/micscif_debug.o
CC [M] xeon-phi-kernel-module/micscif/micscif_fd.o
CC [M] xeon-phi-kernel-module/micscif/micscif_intr.o
CC [M] xeon-phi-kernel-module/micscif/micscif_nm.o
CC [M] xeon-phi-kernel-module/micscif/micscif_nodeqp.o
CC [M] xeon-phi-kernel-module/micscif/micscif_ports.o
CC [M] xeon-phi-kernel-module/micscif/micscif_rb.o
CC [M] xeon-phi-kernel-module/micscif/micscif_rma_dma.o
CC [M] xeon-phi-kernel-module/micscif/micscif_rma_list.o
CC [M] xeon-phi-kernel-module/micscif/micscif_rma.o
CC [M] xeon-phi-kernel-module/micscif/micscif_select.o
CC [M] xeon-phi-kernel-module/micscif/micscif_smpt.o
CC [M] xeon-phi-kernel-module/micscif/micscif_sysfs.o
CC [M] xeon-phi-kernel-module/micscif/micscif_va_gen.o
CC [M] xeon-phi-kernel-module/micscif/micscif_va_node.o
CC [M] xeon-phi-kernel-module/vnet/micveth_dma.o
CC [M] xeon-phi-kernel-module/vnet/micveth_param.o
LD [M] xeon-phi-kernel-module/mic.o
Building modules, stage 2.
MODPOST 1 modules
CC xeon-phi-kernel-module/mic.mod.o
LD [M] xeon-phi-kernel-module/mic.ko
make[1]: Leaving directory '/usr/src/linux-headers-4.19.0-16-amd64'
At this point you can manually load/install the new kernel module (mic.ko
)
which is found in the current directory, or you can make install
. The latter
command also installs the SCIF header file, as well as putting some config files
under /usr/local/etc/
. The information in those config files won’t be picked
up by the system (we will install configs in the correct location in a moment),
but it is useful as a reference. Sample make install
output is shown below.
# make install
make -C /lib/modules/4.19.0-16-amd64/build M=/home/ataylor/xeon-phi-kernel-module modules_install \
INSTALL_MOD_PATH=
make[1]: Entering directory '/usr/src/linux-headers-4.19.0-16-amd64'
INSTALL /home/ataylor/xeon-phi-kernel-module/mic.ko
DEPMOD 4.19.0-16-amd64
Warning: modules_install: missing 'System.map' file. Skipping depmod.
make[1]: Leaving directory '/usr/src/linux-headers-4.19.0-16-amd64'
install -d /usr/local/etc/sysconfig/modules
install mic.modules /usr/local/etc/sysconfig/modules
install -d /usr/local/etc/modprobe.d
install -m644 mic.conf /usr/local/etc/modprobe.d
install -d /usr/local/etc/udev/rules.d
install -m644 udev-mic.rules /usr/local/etc/udev/rules.d/50-udev-mic.rules
install -d /lib/modules/4.19.0-16-amd64
install -m644 Module.symvers /lib/modules/4.19.0-16-amd64/scif.symvers
install -d /usr/src/linux-headers-4.19.0-16-amd64/include/modules
install -m644 include/scif.h /usr/src/linux-headers-4.19.0-16-amd64/include/modules
Create the file /etc/modprobe.d/mic.conf
with the following contents,
intended to accomplish two things. First, blacklist the in-tree MIC kernel
module that shipped with our kernel, including all associated modules, and
second, configure the Intel MIC kernel module which we just built and installed.
The options shown are drawn from the defaults in
/usr/local/etc/modprobe.d/mic.conf
.
# Blacklist the in-tree kernel modules associated with the Knight's Corner Xeon
# Phi so that we can load the Intel kernel module.
# These two modules depend on the various bus modules that follow.
blacklist mic_host
blacklist mic_x100_dma
blacklist cosm_bus
blacklist vop_bus
blacklist scif_bus
blacklist mic_bus
# ^^^------ Blacklisting the in-tree MIC kernel module.
# ==============================================================================
# vvv------ Configuring the Intel MIC kernel module.
# The following options apply to the Intel Many Integrated Core (MIC) driver.
# Unless otherwise noted, the value "1" enables the feature and "0" disables
# it.
#
# Option: p2p
# Description: Enables use of SCIF interface peer to peer communication.
#
# Option: p2p_proxy
# Description: Enables use of SCIF P2P Proxy DMA which converts DMA
# reads into DMA writes for performance on certain Intel
# platforms.
#
# Option: reg_cache
# Description: Enables SCIF Registration Caching.
#
# Option: huge_page
# Description: Enables SCIF Huge Page Support.
#
# Option: watchdog
# Description: Enables SCIF watchdog for Lost Node detection.
#
# Option: watchdog_auto_reboot
# Description: Configures behavior of MIC host driver upon detection of a lost
# node. This option is a nop if watchdog=0. Setting value "1"
# allows host driver to reboot node back to "online" state,
# whereas value "0" only allows the host driver to reset the node
# back to "ready" state, leaving the user responsible for rebooting
# the node (or not).
#
# Option: crash_dump
# Description: Enables uOS Kernel Crash Dump Captures.
#
# Option: ulimit
# Description: Enables ulimit checks on max locked memory for scif_register.
#
options mic reg_cache=1 huge_page=1 watchdog=1 watchdog_auto_reboot=1 crash_dump=1 p2p=1 p2p_proxy=1 ulimit=0
options mic_host reg_cache=1 huge_page=1 watchdog=1 watchdog_auto_reboot=1 crash_dump=1 p2p=1 p2p_proxy=1 ulimit=0
Finally, add the line mic
to the file /etc/modules-load.d/modules.conf
,
instructing the system to load this kernel module on boot, then run depmod
to
ensure the system is aware of the new kernel module, followed by a reboot to
verify everything works.
After the system comes back up, verify that the module loaded with your desired
options using the systool
command, sample output below.
# systool -v -m mic
Module = "mic"
Attributes:
coresize = "741376"
initsize = "0"
initstate = "live"
refcnt = "0"
taint = "OE"
uevent = <store method only>
Parameters:
crash_dump = "Y"
huge_page = "Y"
msi = "Y"
p2p_proxy = "Y"
p2p = "Y"
pm_qos_cpu_dma_lat = "-1"
psmi = "N"
ramoops_count = "4"
reg_cache = "Y"
ulimit = "N"
vnet = "dma"
vnet_addr = "0"
vnet_num_buffers = "62"
watchdog_auto_reboot= "Y"
watchdog = "Y"
Sections:
<snip>