Overview

This repository contains source code for the Intel kernel module distributed with MPSS 3.8.6 for the Xeon Phi x100 family (Knights Corner). It has been modified for compatibility with newer Linux kernels.

Status

Verified to build and run on Debian 10.9 (buster) with Linux kernel 4.19.181-1 and Intel Xeon Phi 5110P cards.

Instructions

These instructions apply to Debian 10 (buster). Use them as a reference, adapting them to your specific machine.

Before compiling the kernel module, verify that relevant kernel headers are installed.

% uname -a
Linux frostburg 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux
% dpkg -l | grep linux-header
ii  linux-headers-4.19.0-16-amd64    4.19.181-1                        amd64        Header files for Linux 4.19.0-16-amd64
ii  linux-headers-4.19.0-16-common   4.19.181-1                        all          Common header files for Linux 4.19.0-16
ii  linux-headers-amd64              4.19+105+deb10u11                 amd64        Header files for Linux amd64 configuration (meta-package)

Now download and compile the updated version of the Intel kernel driver found in this repository. Sample compilation output is included below.

% git clone git://git.subgeniuskitty.com/xeon-phi-kernel-module/
% cd xeon-phi-kernel-module/
% make clean all
make -C /lib/modules/4.19.0-16-amd64/build M=xeon-phi-kernel-module modules \
        INSTALL_MOD_PATH=
make[1]: Entering directory '/usr/src/linux-headers-4.19.0-16-amd64'
  CC [M]  xeon-phi-kernel-module/dma/mic_dma_lib.o
  CC [M]  xeon-phi-kernel-module/dma/mic_dma_md.o
  CC [M]  xeon-phi-kernel-module/host/acptboot.o
  CC [M]  xeon-phi-kernel-module/host/ioctl.o
  CC [M]  xeon-phi-kernel-module/host/linpm.o
  CC [M]  xeon-phi-kernel-module/host/linpsmi.o
  CC [M]  xeon-phi-kernel-module/host/linscif_host.o
  CC [M]  xeon-phi-kernel-module/host/linsysfs.o
  CC [M]  xeon-phi-kernel-module/host/linux.o
  CC [M]  xeon-phi-kernel-module/host/linvcons.o
  CC [M]  xeon-phi-kernel-module/host/linvnet.o
  CC [M]  xeon-phi-kernel-module/host/micpsmi.o
  CC [M]  xeon-phi-kernel-module/host/micscif_pm.o
  CC [M]  xeon-phi-kernel-module/host/pm_ioctl.o
  CC [M]  xeon-phi-kernel-module/host/pm_pcstate.o
  CC [M]  xeon-phi-kernel-module/host/tools_support.o
  CC [M]  xeon-phi-kernel-module/host/uos_download.o
  CC [M]  xeon-phi-kernel-module/host/vhost/mic_vhost.o
  CC [M]  xeon-phi-kernel-module/host/vhost/mic_blk.o
  CC [M]  xeon-phi-kernel-module/host/vmcore.o
  CC [M]  xeon-phi-kernel-module/micscif/micscif_api.o
  CC [M]  xeon-phi-kernel-module/micscif/micscif_debug.o
  CC [M]  xeon-phi-kernel-module/micscif/micscif_fd.o
  CC [M]  xeon-phi-kernel-module/micscif/micscif_intr.o
  CC [M]  xeon-phi-kernel-module/micscif/micscif_nm.o
  CC [M]  xeon-phi-kernel-module/micscif/micscif_nodeqp.o
  CC [M]  xeon-phi-kernel-module/micscif/micscif_ports.o
  CC [M]  xeon-phi-kernel-module/micscif/micscif_rb.o
  CC [M]  xeon-phi-kernel-module/micscif/micscif_rma_dma.o
  CC [M]  xeon-phi-kernel-module/micscif/micscif_rma_list.o
  CC [M]  xeon-phi-kernel-module/micscif/micscif_rma.o
  CC [M]  xeon-phi-kernel-module/micscif/micscif_select.o
  CC [M]  xeon-phi-kernel-module/micscif/micscif_smpt.o
  CC [M]  xeon-phi-kernel-module/micscif/micscif_sysfs.o
  CC [M]  xeon-phi-kernel-module/micscif/micscif_va_gen.o
  CC [M]  xeon-phi-kernel-module/micscif/micscif_va_node.o
  CC [M]  xeon-phi-kernel-module/vnet/micveth_dma.o
  CC [M]  xeon-phi-kernel-module/vnet/micveth_param.o
  LD [M]  xeon-phi-kernel-module/mic.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      xeon-phi-kernel-module/mic.mod.o
  LD [M]  xeon-phi-kernel-module/mic.ko
make[1]: Leaving directory '/usr/src/linux-headers-4.19.0-16-amd64'

At this point you can manually load/install the new kernel module (mic.ko) which is found in the current directory, or you can make install. The latter command also installs the SCIF header file, as well as putting some config files under /usr/local/etc/. The information in those config files won’t be picked up by the system (we will install configs in the correct location in a moment), but it is useful as a reference. Sample make install output is shown below.

# make install
make -C /lib/modules/4.19.0-16-amd64/build M=/home/ataylor/xeon-phi-kernel-module modules_install \
        INSTALL_MOD_PATH=
make[1]: Entering directory '/usr/src/linux-headers-4.19.0-16-amd64'
  INSTALL /home/ataylor/xeon-phi-kernel-module/mic.ko
  DEPMOD  4.19.0-16-amd64
Warning: modules_install: missing 'System.map' file. Skipping depmod.
make[1]: Leaving directory '/usr/src/linux-headers-4.19.0-16-amd64'
install -d /usr/local/etc/sysconfig/modules
install mic.modules /usr/local/etc/sysconfig/modules
install -d /usr/local/etc/modprobe.d
install -m644 mic.conf /usr/local/etc/modprobe.d
install -d /usr/local/etc/udev/rules.d
install -m644 udev-mic.rules /usr/local/etc/udev/rules.d/50-udev-mic.rules
install -d /lib/modules/4.19.0-16-amd64
install -m644 Module.symvers /lib/modules/4.19.0-16-amd64/scif.symvers
install -d /usr/src/linux-headers-4.19.0-16-amd64/include/modules
install -m644 include/scif.h /usr/src/linux-headers-4.19.0-16-amd64/include/modules

Create the file /etc/modprobe.d/mic.conf with the following contents, intended to accomplish two things. First, blacklist the in-tree MIC kernel module that shipped with our kernel, including all associated modules, and second, configure the Intel MIC kernel module which we just built and installed. The options shown are drawn from the defaults in /usr/local/etc/modprobe.d/mic.conf.

# Blacklist the in-tree kernel modules associated with the Knight's Corner Xeon
# Phi so that we can load the Intel kernel module.

# These two modules depend on the various bus modules that follow.
blacklist mic_host
blacklist mic_x100_dma

blacklist cosm_bus
blacklist vop_bus
blacklist scif_bus
blacklist mic_bus

# ^^^------ Blacklisting the in-tree MIC kernel module.
# ==============================================================================
# vvv------ Configuring the Intel MIC kernel module.

# The following options apply to the Intel Many Integrated Core (MIC) driver.
# Unless otherwise noted, the value "1" enables the feature and "0" disables
# it.
#
# Option:      p2p
# Description: Enables use of SCIF interface peer to peer communication.
#
# Option:      p2p_proxy
# Description: Enables use of SCIF P2P Proxy DMA which converts DMA
#              reads into DMA writes for performance on certain Intel
#              platforms.
#
# Option:      reg_cache
# Description: Enables SCIF Registration Caching.
#
# Option:      huge_page
# Description: Enables SCIF Huge Page Support.
#
# Option:      watchdog
# Description: Enables SCIF watchdog for Lost Node detection.
#
# Option:      watchdog_auto_reboot
# Description: Configures behavior of MIC host driver upon detection of a lost
#              node. This option is a nop if watchdog=0. Setting value "1"
#              allows host driver to reboot node back to "online" state,
#              whereas value "0" only allows the host driver to reset the node
#              back to "ready" state, leaving the user responsible for rebooting
#              the node (or not).
#
# Option:      crash_dump
# Description: Enables uOS Kernel Crash Dump Captures.
#
# Option:      ulimit
# Description: Enables ulimit checks on max locked memory for scif_register.
#
options mic reg_cache=1 huge_page=1 watchdog=1 watchdog_auto_reboot=1 crash_dump=1 p2p=1 p2p_proxy=1 ulimit=0
options mic_host reg_cache=1 huge_page=1 watchdog=1 watchdog_auto_reboot=1 crash_dump=1 p2p=1 p2p_proxy=1 ulimit=0

Finally, add the line mic to the file /etc/modules-load.d/modules.conf, instructing the system to load this kernel module on boot, then run depmod to ensure the system is aware of the new kernel module, followed by a reboot to verify everything works.

After the system comes back up, verify that the module loaded with your desired options using the systool command, sample output below.

# systool -v -m mic
Module = "mic"

  Attributes:
    coresize            = "741376"
    initsize            = "0"
    initstate           = "live"
    refcnt              = "0"
    taint               = "OE"
    uevent              = <store method only>

  Parameters:
    crash_dump          = "Y"
    huge_page           = "Y"
    msi                 = "Y"
    p2p_proxy           = "Y"
    p2p                 = "Y"
    pm_qos_cpu_dma_lat  = "-1"
    psmi                = "N"
    ramoops_count       = "4"
    reg_cache           = "Y"
    ulimit              = "N"
    vnet                = "dma"
    vnet_addr           = "0"
    vnet_num_buffers    = "62"
    watchdog_auto_reboot= "Y"
    watchdog            = "Y"

  Sections:
    <snip>