Commit | Line | Data |
---|---|---|
4d67cfab AT |
1 | # Overview # |
2 | ||
3 | This repository contains source code for the Intel kernel module distributed | |
4 | with MPSS 3.8.6 for the Xeon Phi x100 family (Knights Corner). It has been | |
5 | modified for compatibility with newer Linux kernels. | |
6 | ||
4d67cfab AT |
7 | |
8 | # Status # | |
9 | ||
a4b35961 AT |
10 | Verified to build and run on Debian 10.9 (buster) with Linux kernel 4.19.181-1 |
11 | and Intel Xeon Phi 5110P cards. | |
4d67cfab AT |
12 | |
13 | ||
14 | # Instructions # | |
15 | ||
a4b35961 AT |
16 | These instructions apply to Debian 10 (buster). Use them as a reference, |
17 | adapting them to your specific machine. | |
18 | ||
19 | Before compiling the kernel module, verify that relevant kernel headers are | |
20 | installed. | |
21 | ||
22 | % uname -a | |
23 | Linux frostburg 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux | |
24 | % dpkg -l | grep linux-header | |
25 | ii linux-headers-4.19.0-16-amd64 4.19.181-1 amd64 Header files for Linux 4.19.0-16-amd64 | |
26 | ii linux-headers-4.19.0-16-common 4.19.181-1 all Common header files for Linux 4.19.0-16 | |
27 | ii linux-headers-amd64 4.19+105+deb10u11 amd64 Header files for Linux amd64 configuration (meta-package) | |
28 | ||
29 | Now download and compile the updated version of the Intel kernel driver found | |
30 | in this repository. Sample compilation output is included below. | |
31 | ||
32 | % git clone git://git.subgeniuskitty.com/xeon-phi-kernel-module/ | |
33 | % cd xeon-phi-kernel-module/ | |
34 | % make clean all | |
35 | make -C /lib/modules/4.19.0-16-amd64/build M=xeon-phi-kernel-module modules \ | |
36 | INSTALL_MOD_PATH= | |
37 | make[1]: Entering directory '/usr/src/linux-headers-4.19.0-16-amd64' | |
38 | CC [M] xeon-phi-kernel-module/dma/mic_dma_lib.o | |
39 | CC [M] xeon-phi-kernel-module/dma/mic_dma_md.o | |
40 | CC [M] xeon-phi-kernel-module/host/acptboot.o | |
41 | CC [M] xeon-phi-kernel-module/host/ioctl.o | |
42 | CC [M] xeon-phi-kernel-module/host/linpm.o | |
43 | CC [M] xeon-phi-kernel-module/host/linpsmi.o | |
44 | CC [M] xeon-phi-kernel-module/host/linscif_host.o | |
45 | CC [M] xeon-phi-kernel-module/host/linsysfs.o | |
46 | CC [M] xeon-phi-kernel-module/host/linux.o | |
47 | CC [M] xeon-phi-kernel-module/host/linvcons.o | |
48 | CC [M] xeon-phi-kernel-module/host/linvnet.o | |
49 | CC [M] xeon-phi-kernel-module/host/micpsmi.o | |
50 | CC [M] xeon-phi-kernel-module/host/micscif_pm.o | |
51 | CC [M] xeon-phi-kernel-module/host/pm_ioctl.o | |
52 | CC [M] xeon-phi-kernel-module/host/pm_pcstate.o | |
53 | CC [M] xeon-phi-kernel-module/host/tools_support.o | |
54 | CC [M] xeon-phi-kernel-module/host/uos_download.o | |
55 | CC [M] xeon-phi-kernel-module/host/vhost/mic_vhost.o | |
56 | CC [M] xeon-phi-kernel-module/host/vhost/mic_blk.o | |
57 | CC [M] xeon-phi-kernel-module/host/vmcore.o | |
58 | CC [M] xeon-phi-kernel-module/micscif/micscif_api.o | |
59 | CC [M] xeon-phi-kernel-module/micscif/micscif_debug.o | |
60 | CC [M] xeon-phi-kernel-module/micscif/micscif_fd.o | |
61 | CC [M] xeon-phi-kernel-module/micscif/micscif_intr.o | |
62 | CC [M] xeon-phi-kernel-module/micscif/micscif_nm.o | |
63 | CC [M] xeon-phi-kernel-module/micscif/micscif_nodeqp.o | |
64 | CC [M] xeon-phi-kernel-module/micscif/micscif_ports.o | |
65 | CC [M] xeon-phi-kernel-module/micscif/micscif_rb.o | |
66 | CC [M] xeon-phi-kernel-module/micscif/micscif_rma_dma.o | |
67 | CC [M] xeon-phi-kernel-module/micscif/micscif_rma_list.o | |
68 | CC [M] xeon-phi-kernel-module/micscif/micscif_rma.o | |
69 | CC [M] xeon-phi-kernel-module/micscif/micscif_select.o | |
70 | CC [M] xeon-phi-kernel-module/micscif/micscif_smpt.o | |
71 | CC [M] xeon-phi-kernel-module/micscif/micscif_sysfs.o | |
72 | CC [M] xeon-phi-kernel-module/micscif/micscif_va_gen.o | |
73 | CC [M] xeon-phi-kernel-module/micscif/micscif_va_node.o | |
74 | CC [M] xeon-phi-kernel-module/vnet/micveth_dma.o | |
75 | CC [M] xeon-phi-kernel-module/vnet/micveth_param.o | |
76 | LD [M] xeon-phi-kernel-module/mic.o | |
77 | Building modules, stage 2. | |
78 | MODPOST 1 modules | |
79 | CC xeon-phi-kernel-module/mic.mod.o | |
80 | LD [M] xeon-phi-kernel-module/mic.ko | |
81 | make[1]: Leaving directory '/usr/src/linux-headers-4.19.0-16-amd64' | |
82 | ||
83 | At this point you can manually load/install the new kernel module (`mic.ko`) | |
84 | which is found in the current directory, or you can `make install`. The latter | |
85 | command also installs the SCIF header file, as well as putting some config files | |
86 | under `/usr/local/etc/`. The information in those config files won't be picked | |
87 | up by the system (we will install configs in the correct location in a moment), | |
88 | but it is useful as a reference. Sample `make install` output is shown below. | |
89 | ||
90 | # make install | |
91 | make -C /lib/modules/4.19.0-16-amd64/build M=/home/ataylor/xeon-phi-kernel-module modules_install \ | |
92 | INSTALL_MOD_PATH= | |
93 | make[1]: Entering directory '/usr/src/linux-headers-4.19.0-16-amd64' | |
94 | INSTALL /home/ataylor/xeon-phi-kernel-module/mic.ko | |
95 | DEPMOD 4.19.0-16-amd64 | |
96 | Warning: modules_install: missing 'System.map' file. Skipping depmod. | |
97 | make[1]: Leaving directory '/usr/src/linux-headers-4.19.0-16-amd64' | |
98 | install -d /usr/local/etc/sysconfig/modules | |
99 | install mic.modules /usr/local/etc/sysconfig/modules | |
100 | install -d /usr/local/etc/modprobe.d | |
101 | install -m644 mic.conf /usr/local/etc/modprobe.d | |
102 | install -d /usr/local/etc/udev/rules.d | |
103 | install -m644 udev-mic.rules /usr/local/etc/udev/rules.d/50-udev-mic.rules | |
104 | install -d /lib/modules/4.19.0-16-amd64 | |
105 | install -m644 Module.symvers /lib/modules/4.19.0-16-amd64/scif.symvers | |
106 | install -d /usr/src/linux-headers-4.19.0-16-amd64/include/modules | |
107 | install -m644 include/scif.h /usr/src/linux-headers-4.19.0-16-amd64/include/modules | |
108 | ||
109 | Create the file `/etc/modprobe.d/mic.conf` with the following contents, | |
110 | intended to accomplish two things. First, blacklist the in-tree MIC kernel | |
111 | module that shipped with our kernel, including all associated modules, and | |
112 | second, configure the Intel MIC kernel module which we just built and installed. | |
113 | The options shown are drawn from the defaults in | |
114 | `/usr/local/etc/modprobe.d/mic.conf`. | |
115 | ||
116 | # Blacklist the in-tree kernel modules associated with the Knight's Corner Xeon | |
117 | # Phi so that we can load the Intel kernel module. | |
118 | ||
119 | # These two modules depend on the various bus modules that follow. | |
120 | blacklist mic_host | |
121 | blacklist mic_x100_dma | |
122 | ||
123 | blacklist cosm_bus | |
124 | blacklist vop_bus | |
125 | blacklist scif_bus | |
126 | blacklist mic_bus | |
127 | ||
128 | # ^^^------ Blacklisting the in-tree MIC kernel module. | |
129 | # ============================================================================== | |
130 | # vvv------ Configuring the Intel MIC kernel module. | |
131 | ||
132 | # The following options apply to the Intel Many Integrated Core (MIC) driver. | |
133 | # Unless otherwise noted, the value "1" enables the feature and "0" disables | |
134 | # it. | |
135 | # | |
136 | # Option: p2p | |
137 | # Description: Enables use of SCIF interface peer to peer communication. | |
138 | # | |
139 | # Option: p2p_proxy | |
140 | # Description: Enables use of SCIF P2P Proxy DMA which converts DMA | |
141 | # reads into DMA writes for performance on certain Intel | |
142 | # platforms. | |
143 | # | |
144 | # Option: reg_cache | |
145 | # Description: Enables SCIF Registration Caching. | |
146 | # | |
147 | # Option: huge_page | |
148 | # Description: Enables SCIF Huge Page Support. | |
149 | # | |
150 | # Option: watchdog | |
151 | # Description: Enables SCIF watchdog for Lost Node detection. | |
152 | # | |
153 | # Option: watchdog_auto_reboot | |
154 | # Description: Configures behavior of MIC host driver upon detection of a lost | |
155 | # node. This option is a nop if watchdog=0. Setting value "1" | |
156 | # allows host driver to reboot node back to "online" state, | |
157 | # whereas value "0" only allows the host driver to reset the node | |
158 | # back to "ready" state, leaving the user responsible for rebooting | |
159 | # the node (or not). | |
160 | # | |
161 | # Option: crash_dump | |
162 | # Description: Enables uOS Kernel Crash Dump Captures. | |
163 | # | |
164 | # Option: ulimit | |
165 | # Description: Enables ulimit checks on max locked memory for scif_register. | |
166 | # | |
167 | options mic reg_cache=1 huge_page=1 watchdog=1 watchdog_auto_reboot=1 crash_dump=1 p2p=1 p2p_proxy=1 ulimit=0 | |
168 | options mic_host reg_cache=1 huge_page=1 watchdog=1 watchdog_auto_reboot=1 crash_dump=1 p2p=1 p2p_proxy=1 ulimit=0 | |
169 | ||
170 | Finally, add the line `mic` to the file `/etc/modules-load.d/modules.conf`, | |
171 | instructing the system to load this kernel module on boot, then run `depmod` to | |
172 | ensure the system is aware of the new kernel module, followed by a reboot to | |
173 | verify everything works. | |
174 | ||
175 | After the system comes back up, verify that the module loaded with your desired | |
176 | options using the `systool` command, sample output below. | |
177 | ||
178 | # systool -v -m mic | |
179 | Module = "mic" | |
180 | ||
181 | Attributes: | |
182 | coresize = "741376" | |
183 | initsize = "0" | |
184 | initstate = "live" | |
185 | refcnt = "0" | |
186 | taint = "OE" | |
187 | uevent = <store method only> | |
188 | ||
189 | Parameters: | |
190 | crash_dump = "Y" | |
191 | huge_page = "Y" | |
192 | msi = "Y" | |
193 | p2p_proxy = "Y" | |
194 | p2p = "Y" | |
195 | pm_qos_cpu_dma_lat = "-1" | |
196 | psmi = "N" | |
197 | ramoops_count = "4" | |
198 | reg_cache = "Y" | |
199 | ulimit = "N" | |
200 | vnet = "dma" | |
201 | vnet_addr = "0" | |
202 | vnet_num_buffers = "62" | |
203 | watchdog_auto_reboot= "Y" | |
204 | watchdog = "Y" | |
205 | ||
206 | Sections: | |
207 | <snip> | |
4d67cfab | 208 |