Commit | Line | Data |
---|---|---|
a60cd2ef AT |
1 | # Xeon Phi Setup/Build Notes # |
2 | ||
3 | These notes cover the creation of a Debian 10.9 (buster) server with ZFS root | |
4 | which serves as host to Knights Corner Xeon Phi coprocessor cards. | |
5 | ||
6 | Each of these coprocessor cards features a P54C-derived core extended to | |
7 | support the X86-64 instruction set, 4-way SMT, and a beefy 512-bit vector | |
8 | processor bolted alongside. Sixty of these cores are connected on a roughly 1 | |
9 | terabit/s bi-directional ring bus. In addition to 8GB of GDDR5 RAM, each core | |
10 | has 512kB of local cache and, via the ring bus and a distributed tag store, all | |
11 | caches are coherent and quickly accessible from remote cores. This hardware is | |
12 | packaged up on a PCIe card which presents a virtual network interface to the | |
13 | host. The coprocessor card runs Linux+BusyBox, allowing SSH access to a | |
14 | traditional Linux environment on a familiar 60-core x86-64 architecture. | |
15 | ||
16 | The hostname `frostburg.subgeniuskitty.com` stems from the original | |
17 | [FROSTBURG](https://en.wikipedia.org/wiki/FROSTBURG), a CM-5 designed by | |
18 | Thinking Machines. Although the fundamental connection topology of a fat tree | |
19 | was different than the ring used in this Xeon Phi, the systems are somewhat | |
20 | similar. Both feature a NUMA cluster of repackaged and extended commercial | |
21 | processor cores operating on independent instruction streams in a MIMD fashion | |
22 | focused on small local data stores. By coincidence, both also feature similar | |
23 | core counts and total memory size. | |
24 | ||
25 | The information on this page includes: | |
26 | ||
27 | - Hardware compatibility notes for Xeon Phi and Xeon host. | |
28 | ||
29 | - Installation of Debian 10.9 (buster) root on encrypted ZFS mirror with | |
30 | automated snapshots and scrubs. | |
31 | ||
31f1b002 | 32 | - Porting the Xeon Phi kernel module to newer versions of the Linux kernel. |
a60cd2ef AT |
33 | |
34 | - (TODO) Installing MPSS toolkit on Debian (or CentOS VM). | |
35 | ||
36 | - (TODO) Building GCC toolchain for Xeon Phi. | |
37 | ||
38 | - (TODO) Installing Intel toolchain for Xeon Phi. | |
39 | ||
40 | These notes are a high-level checklist for my reference rather than a | |
41 | step-by-step installation guide for the public. That means they make no attempt | |
42 | to explain all options at each step, rather that they mention only the options | |
43 | I use on my servers. It also means they use my domains, my file system paths, | |
44 | etc in the examples. Don't blindly copy and paste. | |
45 | ||
46 | -------------------------------------------------------------------------------- | |
47 | ||
48 | ||
49 | ## Hardware ## | |
50 | ||
51 | The host system was kept low power both figuratively and literally. It will | |
52 | primarily serve as a host for the Phi coprocessors and bridge to the network. | |
53 | ||
54 | - **Chassis:** Supermicro 2027GR-TR2 | |
55 | ||
56 | - **Motherboard:** Supermicro X9DRG-HF+II | |
57 | ||
58 | - **CPU:** 2x Xeon E5-2637 | |
59 | ||
60 | - **RAM:** 8x 4GB DDR3 RDIMM | |
61 | ||
62 | - **Storage:** 2x Intel 160GB X-25M SSD | |
63 | ||
64 | - **Payload:** 4x Intel Xeon Phi 5110P | |
65 | ||
66 | To enter the BIOS, use the `DEL` key. Similarly, a boot device selection menu | |
67 | is obtained by pressing `F11`. System will display two-character status codes | |
68 | in the bottom right corner of display. | |
69 | ||
70 | Support files are stored under `hw_support/Intel Xeon Phi/supermicro/`. | |
71 | ||
72 | ||
73 | ### Memory ### | |
74 | ||
75 | Using eight identical sticks of MT36JSZF51272PZ-1G4 RAM. These are ECC DDR3 | |
76 | 2Rx4 PC3-10600 RDIMMS operating at 1.5V. Per page 2-12 of the manual | |
77 | (`MNL_1502.pdf`), DIMMs are installed in all blue memory slots. | |
78 | ||
79 | ||
80 | ### Processors & Heatsinks ### | |
81 | ||
82 | Xeon E5-2637 CPUs selected for lower power, high frequency, cheap price, and | |
83 | 'full' PCIe lane count. They only need to be a host for the real show. Per page | |
84 | 5-7 of the chassis manual (`MNL-1564.pdf`), CPU1 requires heatsink SNK-P0048PS | |
85 | and CPU2 requires heatsink SNK-P0047PS. | |
86 | ||
87 | ||
50ab1573 | 88 | ### SAS Backplane & Motherboard SATA ### |
a60cd2ef AT |
89 | |
90 | The SAS backplane is a little odd. The first eight drive bays connect via a | |
91 | pair of SFF-8087 connectors and the last two drive bays connect via standard | |
92 | 7-pin SATA connectors. | |
93 | ||
94 | Since the motherboard provides ten 7-pin SATA connectors, two cables breaking | |
95 | out SFF-8087 to quad SATA will be required. I tried using just such a cable, | |
96 | but had no luck. There doesn't appear to be anything configurable on the | |
97 | backplane itself. The backplane manual is stored at `BPN-SAS-218A.pdf`. My | |
98 | cable was of unknown origin. Per photos on some eBay auctions, the proper | |
99 | Supermicro cable appears to be part number 672042095704. In addition to the | |
100 | four SATA connectors, this cable also bundles some sort of 4-pin header, | |
101 | presumably the SGPIO connection. | |
102 | ||
103 | In the meantime, since I only intend to use two small drives in a ZFS mirror | |
104 | for the OS and home directories, with all other storage on network shares, | |
105 | simply use the last two slots and connect with normal 30"+ SATA cables. | |
106 | ||
107 | These last two drive bay slots are connected to the two white SATA ports on the | |
108 | motherboard, with the lowest numbered drive slot connected to the rear-most | |
109 | white SATA port. When SFF-8087 connectors are eventually used to increase local | |
110 | storage, relocate the boot drives to drive slots 0 and 1, and connect these | |
111 | slots to the white SATA ports. | |
112 | ||
113 | On the motherboard, the white ports are SATA3 and the black ports are SATA2. | |
114 | The line of 2x white and 4x black SATA ports are part of the primary SATA | |
115 | controller or `I_SATA`. The other line of 4x black SATA ports is part of the | |
116 | secondary or `S_SATA` controller. Put any boot drives on the `I_SATA` ports. | |
117 | ||
118 | ||
119 | ### Xeon Phi ### | |
120 | ||
121 | Section 5.1 of the Intel Xeon Phi Coprocessor Datasheet (DocID 328209-004EN) | |
122 | mentions that connecting the card via both 2x4 and 2x3 power connectors enables | |
123 | higher sustained power draw up to 245 watts versus 225 watts of other power | |
124 | cable configurations. This chassis will easily support the higher power draw | |
125 | and heat dissipation. | |
126 | ||
127 | The Xeon Phi coprocessor cards reserve PCIe MMIO address space sufficient to | |
128 | map the entire coprocessor card's RAM. Since this is >4GB, PCIe Base Address | |
129 | Registers (BAR) of greater than 32-bit size are required. This should be | |
130 | enabled in the BIOS of this particular motherboard under | |
131 | `PCIe/PCI/PnP Configuration` -> `Above 4G Decoding`. | |
132 | ||
133 | In general, motherboards with chipsets equal to or newer than the C602 should | |
134 | work. This includes most Supermicro motherboards from the X9xxx generation or | |
135 | later. None of the Supermicro X8xxx generation motherboards appear to be | |
136 | compatible. | |
137 | ||
138 | The Xeon Phi 5110P, per the suffix, is passively cooled. Section 3 of the Intel | |
139 | Xeon Phi Coprocessor Datasheet (DocID 328209-004EN) details the cooling and | |
140 | mounting requirements. | |
141 | ||
142 | ||
143 | ### Optional Fans ### | |
144 | ||
145 | There are a number of optional fans for this chassis, all detailed in the | |
146 | chassis manual (`MNL-1564.pdf`). My machine includes the optional fan for | |
147 | another double-height, full-length PCIe card with backpanel IO slots, intended | |
148 | to support something like a GPU to drive monitors. Since the optional fan is | |
149 | installed and since the power budget easily supports it, this means the fifth | |
150 | Xeon Phi card could be installed, albeit with slower PCIe connection. | |
151 | ||
152 | Regardless, since this fan is installed, whenever fewer than four Xeon Phi | |
153 | cards are installed, preferentially locate them on the left hand side of | |
154 | chassis, near the lower numbered drive bays. | |
155 | ||
156 | ||
157 | ### Power Supply ### | |
158 | ||
159 | The system contains dual redundant power supplies. Each is capable of supplying | |
160 | 1600 watts, but only when connected to a 240 volt source. When connected to a | |
161 | 120 volt source, maximum power output is 1000 watts. | |
162 | ||
163 | ||
164 | ### Rackmount ### | |
165 | ||
166 | The chassis is over 30" long and protrudes from rear of rack by approximately | |
167 | 1/2". To avoid the rear cable snagging passing carts and elbows, chassis was | |
168 | mounted at top of rack (after empty 1U). The Supermicro rails required cutting | |
169 | four notches in the vertical posts, so this is a semi-permanent home. | |
170 | ||
171 | Inserting or extracting the server from the rack at that height requires an | |
172 | extraordinary amount of free space in front of the rack and some advance | |
173 | planning. Where possible, try to do hardware modifications in-rack. The rails | |
174 | are extremely solid even when the server is fully extended. The grey | |
175 | OS-114/WQM-4 sonar test set chassis makes a solid step stool at the ideal | |
176 | height for working on the server while installed in the rack. | |
177 | ||
178 | ||
179 | ### USB Ports ### | |
180 | ||
181 | There are only two USB ports, both located on the rear of the chassis. During | |
182 | OS installation, if a mouse is required in addition to the keyboard and USB | |
183 | install drive, then a USB hub is required. | |
184 | ||
185 | -------------------------------------------------------------------------------- | |
186 | ||
187 | ||
188 | ## Debian Buster Installation ## | |
189 | ||
190 | These installation instructions use the following XFCE Debian live image. | |
191 | ||
192 | debian-live-10.9.0-amd64-xfce.iso | |
193 | ||
194 | Both the Gnome and XFCE live images were unusably slow in GUI mode. The text | |
195 | installer was fast and responsive, as were VTYs (`Ctrl`+`Alt`+`F2`) from within | |
196 | the live environment. Only the GUIs were slow, but they were slow to the point | |
197 | of being unusable, with single keypresses registering over a dozen times. Once | |
198 | Debian was installed on the SSD and booting normally, the GUI is perfectly | |
199 | usable. Since the local terminal is only used to install and start an OpenSSH | |
200 | daemon, and since this can be done from a VTY, the issue was not investigated | |
201 | further. | |
202 | ||
203 | The root on ZFS portion of this installation process is derived from the guide | |
204 | located here: | |
205 | ||
206 | <https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Buster%20Root%20on%20ZFS.html> | |
207 | ||
208 | ||
209 | ### Remote Access ### | |
210 | ||
211 | From the `F11` BIOS boot menu, select the UEFI entry for the USB live image. | |
212 | Lacking a mouse, press `CTRL`+`ALT`+`F2` after X is running in order to access | |
213 | a text-only VTY, already logged in as the user `user`. Install an SSH server so | |
214 | the remaining install can be done over the network. | |
215 | ||
216 | apt-get update | |
217 | apt-get install openssh-server | |
218 | systemctl enable ssh | |
219 | ||
220 | From wherever you intend to complete the install, SSH into the live Debian | |
221 | environment as user `user` with password `live`. | |
222 | ||
223 | ||
224 | ### ZFS Configuration ### | |
225 | ||
226 | Edit `/etc/apt/sources.list` to include the following entries. | |
227 | ||
228 | deb http://deb.debian.org/debian/ buster main contrib | |
229 | deb http://deb.debian.org/debian/ buster-backports main contrib | |
230 | deb-src http://deb.debian.org/debian/ buster main contrib | |
231 | ||
232 | Install the ZFS kernel module. Specify `--no-install-recommends` to avoid | |
233 | picking up `zfsutils-linux` since it will fail at this point. See | |
234 | <https://github.com/openzfs/zfs/issues/9599> for more details. | |
235 | ||
236 | apt-get install -t buster-backports --no-install-recommends zfs-dkms | |
237 | modprobe zfs | |
238 | ||
239 | With the kernel module successfully loaded, proceed to install ZFS. | |
240 | ||
241 | apt-get install -t buster-backports zfsutils-linux | |
242 | ||
243 | After using `dd` to eliminate any existing partition tables, partition the | |
244 | disks for use with UEFI and ZFS. | |
245 | ||
246 | First, create a UEFI partition on each disk. | |
247 | ||
248 | sgdisk -n2:1M:+512M -t2:EF00 /dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTPO1252011L160AGN | |
249 | ||
250 | Next, create a partition for the boot pool. | |
251 | ||
252 | sgdisk -n3:0:+1G -t3:BF01 /dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTPO1252011L160AGN | |
253 | ||
254 | Finally, create a partition for the encrypted pool. | |
255 | ||
256 | sgdisk -n4:0:0 -t4:BF00 /dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTPO1252011L160AGN | |
257 | ||
258 | Now that partitioning is complete, create the boot and root pools. | |
259 | ||
260 | The boot pool uses only ZFS options supported by GRUB. | |
261 | ||
262 | zpool create \ | |
263 | -o cachefile=/etc/zfs/zpool.cache \ | |
264 | -o ashift=12 -d \ | |
265 | -o feature@async_destroy=enabled \ | |
266 | -o feature@bookmarks=enabled \ | |
267 | -o feature@embedded_data=enabled \ | |
268 | -o feature@empty_bpobj=enabled \ | |
269 | -o feature@enabled_txg=enabled \ | |
270 | -o feature@extensible_dataset=enabled \ | |
271 | -o feature@filesystem_limits=enabled \ | |
272 | -o feature@hole_birth=enabled \ | |
273 | -o feature@large_blocks=enabled \ | |
274 | -o feature@lz4_compress=enabled \ | |
275 | -o feature@spacemap_histogram=enabled \ | |
276 | -o feature@zpool_checkpoint=enabled \ | |
277 | -O acltype=posixacl -O canmount=off -O compression=lz4 \ | |
278 | -O devices=off -O normalization=formD -O relatime=on -O xattr=sa \ | |
279 | -O mountpoint=/boot -R /mnt \ | |
280 | bpool mirror \ | |
281 | /dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTPO1252011L160AGN-part3 | |
282 | /dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTHC72250AKD480MGN-part3 | |
283 | ||
284 | Now create the root pool with ZFS encryption. | |
285 | ||
286 | zpool create \ | |
287 | -o ashift=12 \ | |
288 | -O encryption=aes-256-gcm \ | |
289 | -O keylocation=prompt -O keyformat=passphrase \ | |
290 | -O acltype=posixacl -O canmount=off -O compression=lz4 \ | |
291 | -O dnodesize=auto -O normalization=formD -O relatime=on \ | |
292 | -O xattr=sa -O mountpoint=/ -R /mnt \ | |
293 | rpool mirror \ | |
294 | /dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTPO1252011L160AGN-part4 | |
295 | /dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTHC72250AKD480MGN-part4 | |
296 | ||
297 | All the pools are created, so now it's time to setup filesystems. Start with | |
298 | some containers. | |
299 | ||
300 | zfs create -o canmount=off -o mountpoint=none rpool/ROOT | |
301 | zfs create -o canmount=off -o mountpoint=none bpool/BOOT | |
302 | ||
303 | Now add filesystems for boot and root. | |
304 | ||
305 | zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/debian | |
306 | zfs mount rpool/ROOT/debian | |
307 | zfs create -o mountpoint=/boot bpool/BOOT/debian | |
308 | ||
309 | Create a filesystem to contain home directories and mount root's homedir in the | |
310 | correct location. | |
311 | ||
312 | zfs create rpool/home | |
313 | zfs create -o mountpoint=/root rpool/home/root | |
314 | chmod 700 /mnt/root | |
315 | ||
316 | Create filesystems under `/var` and exclude temporary files from snapshots. | |
317 | ||
318 | zfs create -o canmount=off rpool/var | |
319 | zfs create -o canmount=off rpool/var/lib | |
320 | zfs create rpool/var/log | |
321 | zfs create rpool/var/spool | |
322 | zfs create -o com.sun:auto-snapshot=false rpool/var/cache | |
323 | zfs create -o com.sun:auto-snapshot=false rpool/var/tmp | |
324 | chmod 1777 /mnt/var/tmp | |
325 | zfs create rpool/var/mail | |
326 | ||
327 | Create a few other misc filesystems. | |
328 | ||
329 | zfs create rpool/srv | |
330 | zfs create -o canmount=off rpool/usr | |
331 | zfs create rpool/usr/local | |
332 | ||
333 | Temporarily mount a `tmpfs` at `/run`. | |
334 | ||
335 | mkdir /mnt/run | |
336 | mount -t tmpfs tmpfs /mnt/run | |
337 | mkdir /mnt/run/lock | |
338 | ||
339 | ||
340 | ### Debian Configuration ### | |
341 | ||
342 | Install a minimal Debian system. | |
343 | ||
344 | apt-get install debootstrap | |
345 | debootstrap buster /mnt | |
346 | ||
347 | Copy the zpool cache into the new system. | |
348 | ||
349 | mkdir /mnt/etc/zfs | |
350 | cp /etc/zfs/zpool.cache /mnt/etc/zfs | |
351 | ||
352 | Set the hostname. | |
353 | ||
354 | echo frostburg > /mnt/etc/hostname | |
355 | echo "127.0.1.1 frostburg.subgeniuskitty.com frostburg" >> /mnt/etc/hosts | |
356 | ||
357 | Configure networking. | |
358 | ||
359 | vi /mnt/etc/network/interfaces.d/enp129s0f0 | |
360 | ||
361 | auto enp129s0f0 | |
362 | iface enp129s0f0 inet static | |
363 | address 192.168.1.7/24 | |
364 | gateway 192.168.1.1 | |
365 | ||
366 | vi /etc/resolv.conf | |
367 | ||
368 | search subgeniuskitty.com | |
369 | nameserver 192.168.1.1 | |
370 | ||
371 | Configure packages sources. | |
372 | ||
373 | vi /mnt/etc/apt/sources.list | |
374 | ||
375 | deb http://deb.debian.org/debian buster main contrib | |
376 | deb-src http://deb.debian.org/debian buster main contrib | |
377 | ||
378 | deb http://security.debian.org/debian-security buster/updates main contrib | |
379 | deb-src http://security.debian.org/debian-security buster/updates main contrib | |
380 | ||
381 | deb http://deb.debian.org/debian buster-updates main contrib | |
382 | deb-src http://deb.debian.org/debian buster-updates main contrib | |
383 | ||
384 | vi /mnt/etc/apt/sources.list.d/buster-backports.list | |
385 | ||
386 | deb http://deb.debian.org/debian buster-backports main contrib | |
387 | deb-src http://deb.debian.org/debian buster-backports main contrib | |
388 | ||
389 | vi /mnt/etc/apt/preferences.d/90_zfs | |
390 | ||
391 | Package: libnvpair1linux libuutil1linux libzfs2linux libzfslinux-dev libzpool2linux python3-pyzfs pyzfs-doc spl spl-dkms zfs-dkms zfs-dracut zfs-initramfs zfs-test zfsutils-linux zfsutils-linux-dev zfs-zed | |
392 | Pin: release n=buster-backports | |
393 | Pin-Priority: 990 | |
394 | ||
395 | apt-get update | |
396 | ||
397 | Chroot into the new environment. | |
398 | ||
399 | mount --rbind /dev /mnt/dev | |
400 | mount --rbind /proc /mnt/proc | |
401 | mount --rbind /sys /mnt/sys | |
402 | chroot /mnt | |
403 | ||
404 | Configure the new environment as a basic system. | |
405 | ||
406 | ln -s /proc/self/mounts /etc/mtab | |
407 | apt-get update | |
408 | export TERM=vt100 | |
409 | apt-get install console-setup locales | |
410 | dpkg-reconfigure locales tzdata keyboard-configuration console-setup | |
411 | ||
412 | Install ZFS on the new system. | |
413 | ||
414 | apt-get install dpkg-dev linux-headers-amd64 linux-image-amd64 | |
415 | apt-get install zfs-initramfs | |
416 | echo REMAKE_INITRD=yes > /etc/dkms/zfs.conf | |
417 | ||
418 | Install GRUB and configure UEFI boot partition. | |
419 | ||
420 | apt-get install dosfstools | |
421 | mkdosfs -F 32 -s 1 -n EFI /dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTPO1252011L160AGN-part2 | |
422 | mkdir /boot/efi | |
423 | echo "/dev/disk/by-id/ata-INTEL_SSDSA2M160G2GN_BTPO1252011L160AGN-part2 /boot/efi vfat defaults 0 0" >> /etc/fstab | |
424 | mount /boot/efi | |
425 | apt-get install grub-efi-amd64 shim-signed | |
426 | apt-get remove --purge os-prober | |
427 | ||
428 | Ensure the bpool is always imported, even if `/etc/zfs/zpool.cache` doesn't | |
429 | exist or doesn't include a relevant entry. | |
430 | ||
431 | vi /etc/systemd/system/zfs-import-bpool.service | |
432 | ||
433 | [Unit] | |
434 | DefaultDependencies=no | |
435 | Before=zfs-import-scan.service | |
436 | Before=zfs-import-cache.service | |
437 | ||
438 | [Service] | |
439 | Type=oneshot | |
440 | RemainAfterExit=yes | |
441 | ExecStart=/sbin/zpool import -N -o cachefile=none bpool | |
442 | # Work-around to preserve zpool cache: | |
443 | ExecStartPre=-/bin/mv /etc/zfs/zpool.cache /etc/zfs/preboot_zpool.cache | |
444 | ExecStartPost=-/bin/mv /etc/zfs/preboot_zpool.cache /etc/zfs/zpool.cache | |
445 | ||
446 | [Install] | |
447 | WantedBy=zfs-import.target | |
448 | ||
449 | systemctl enable zfs-import-bpool.service | |
450 | ||
451 | Create a `tmpfs` mounted at `/tmp`. | |
452 | ||
453 | cp /usr/share/systemd/tmp.mount /etc/systemd/system/ | |
454 | systemctl enable tmp.mount | |
455 | ||
456 | ||
457 | ### Bootloader Configuration ### | |
458 | ||
459 | Verify ZFS boot filesystem is recognized. | |
460 | ||
461 | grub-probe /boot | |
462 | ||
463 | Refresh initrd. | |
464 | ||
465 | update-initramfs -c -k all | |
466 | ||
467 | Configure GRUB by editing `/etc/default/grub`. Remove the `quiet` option from | |
468 | `GRUB_CMDLINE_LINUX_DEFAULT` and add the following two options to the | |
469 | appropriate entries. | |
470 | ||
471 | GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/debian" | |
472 | GRUB_TERMINAL=console | |
473 | ||
474 | Install GRUB to the UEFI boot partition. | |
475 | ||
476 | grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=debian-1 --recheck --no-floppy | |
477 | ||
478 | Install GRUB on the other hard drives, incrementing `-2` to `-N` as necessary. | |
479 | ||
480 | umount /boot/efi | |
481 | dd if=/dev/disk/by-id/scsi-SATA_disk1-part2 \ | |
482 | of=/dev/disk/by-id/scsi-SATA_disk2-part2 | |
483 | efibootmgr -c -g -d /dev/disk/by-id/scsi-SATA_disk2 \ | |
484 | -p 2 -L "debian-2" -l '\EFI\debian\grubx64.efi' | |
485 | mount /boot/efi | |
486 | ||
487 | Fix filesystem mount ordering. Quoting from the install reference, "We need to | |
488 | activate `zfs-mount-generator`. This makes systemd aware of the separate | |
489 | mountpoints, which is important for things like `/var/log` and `/var/tmp`. In | |
490 | turn, `rsyslog.service` depends on `var-log.mount` by way of `local-fs.target` | |
491 | and services using the `PrivateTmp` feature of systemd automatically use | |
492 | `After=var-tmp.mount`." | |
493 | ||
494 | mkdir /etc/zfs/zfs-list.cache | |
495 | touch /etc/zfs/zfs-list.cache/bpool | |
496 | touch /etc/zfs/zfs-list.cache/rpool | |
497 | zed -F | |
498 | ||
499 | From another SSH session, verify that zed updated the cache by making sure the | |
500 | previously created empty files are not empty. | |
501 | ||
502 | cat /etc/zfs/zfs-list.cache/bpool | |
503 | cat /etc/zfs/zfs-list.cache/rpool | |
504 | ||
505 | If all is well, return to the previous SSH session and terminate `zed` with | |
506 | `Ctrl`+`C`. | |
507 | ||
508 | Fix the paths to eliminate `/mnt`. | |
509 | ||
510 | sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/* | |
511 | ||
512 | ||
513 | ### Reboot ### | |
514 | ||
515 | The Debian install is almost ready for use without the live Debian host | |
516 | environment. Only a few steps remain. | |
517 | ||
518 | Do a final system update. | |
519 | ||
520 | apt-get dist-upgrade | |
521 | ||
522 | Disable log compression since ZFS is already compressing at the block level. | |
523 | ||
524 | for file in /etc/logrotate.d/* ; do | |
525 | if grep -Eq "(^|[^#y])compress" "$file" ; then | |
526 | sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file" | |
527 | fi | |
528 | done | |
529 | ||
530 | Install an SSH server so we can login again after rebooting. | |
531 | ||
532 | apt-get install openssh-server | |
533 | ||
534 | Set a root password. | |
535 | ||
536 | passwd | |
537 | ||
538 | Create a user account. | |
539 | ||
540 | zfs create rpool/home/ataylor | |
541 | adduser ataylor | |
542 | mkdir /etc/skel/.ssh && chmod 700 /etc/skel/.ssh | |
543 | cp -a /etc/skel/. /home/ataylor/ | |
544 | scp ataylor@lagavulin:/usr/home/ataylor/.ssh/id_rsa.pub /home/ataylor/.ssh/authorized_keys | |
545 | chown -R ataylor:ataylor /home/ataylor | |
546 | usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video ataylor | |
547 | ||
548 | Snapshot the install. | |
549 | ||
550 | zfs snapshot bpool/BOOT/debian@install | |
551 | zfs snapshot rpool/ROOT/debian@install | |
552 | ||
553 | Exit the chroot and unmount all filesystems. | |
554 | ||
555 | exit | |
556 | mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} | |
557 | zpool export -a | |
558 | ||
559 | Reboot the computer and remove the USB stick. Installation is complete. | |
560 | ||
561 | ||
562 | ### UNIX Userland ### | |
563 | ||
564 | Install various no-config-required userland packages before continuing. | |
565 | ||
566 | apt-get install net-tools bzip2 zip ntp htop xterm screen git \ | |
399284b1 AT |
567 | build-essential pciutils smartmontools gdb valgrind wget \ |
568 | texlive texlive-latex-extra graphviz firefox sysfsutils | |
a60cd2ef AT |
569 | |
570 | ||
571 | #### X Window Manager #### | |
572 | ||
573 | Install X and dwm to ensure all dependencies are met for running my dwm-derived | |
574 | window manager. | |
575 | ||
576 | apt-get install xorg dwm numlockx | |
577 | ||
578 | Install dependencies for building my window manager. | |
579 | ||
580 | apt-get install libx11-dev libxft-dev libxinerama-dev | |
581 | ||
582 | Copy the Hophib Modern Desktop git repo to the new server. Make the following changes: | |
583 | ||
584 | - `hhmd/src/mk.conf`: Change the installation prefix from `/hh` to | |
585 | `/home/ataylor/bin` | |
586 | ||
587 | - `hhmd/src/window_manager/Makefile`: Change library and include paths from | |
588 | `/usr/local/...` to `/usr/...` | |
589 | ||
590 | - `hhmd/src/window_manager/dwm-status.c`: Change `#include <sys/time.h>` to | |
591 | `#include <time.h>` and add `#define _GNU_SOURCE` as well as | |
592 | `#define _DEFAULT_SOURCE` to the top of the file | |
593 | ||
594 | - `hhmd/src/window_manager/dwm.c`: Add `#define _POSIX_C_SOURCE 2` to the top | |
595 | of the file. | |
596 | ||
597 | - `hhmd/src/window_manager/dwm-watchdog.sh`: Change paths and executable | |
598 | names from `/hh/...` to `/home/ataylor/bin/...` and from `wm` to `dwm`. | |
599 | ||
600 | Execute `make clean install`. Verify that `dwm`, `dwm-status` and | |
601 | `dwm-watchdog.sh` all ended up in `/home/ataylor/bin` with appropriate | |
602 | permissions. Delete the man pages that were installed in ataylor's homedir. | |
603 | ||
604 | Create `~/.xinitrc` with following contents. | |
605 | ||
606 | /usr/bin/numlockx & | |
607 | /home/ataylor/bin/dwm-status & | |
608 | /home/ataylor/bin/dwm-watchdog.sh | |
609 | ||
610 | Verify X and my window manager start successfully and that `dwm-watchdog.sh` | |
611 | keeps X and X applications alive during a window manager live restart. | |
612 | ||
613 | ||
614 | #### VIM #### | |
615 | ||
616 | Install gvim. | |
617 | ||
618 | apt-get install gvim | |
619 | ||
620 | Create `~/.vimrc` with the following contents. | |
621 | ||
622 | set nocompatible | |
623 | filetype off | |
624 | set mouse=r | |
625 | set number | |
626 | syntax on | |
627 | set tabstop=4 | |
628 | set expandtab | |
629 | ||
630 | "Folding | |
631 | "http://vim.wikia.com/wiki/Folding_for_plain_text_files_based_on_indentation | |
632 | "set foldmethod=expr | |
633 | "set foldexpr=(getline(v:lnum)=~'^$')?-1:((indent(v:lnum)<indent(v:lnum+1))?('>'.indent(v:lnum+1)):indent(v:lnum)) | |
634 | "set foldtext=getline(v:foldstart) | |
635 | "set fillchars=fold:\ "(there's a space after that \) | |
636 | "highlight Folded ctermfg=DarkGreen ctermbg=Black | |
637 | "set foldcolumn=6 | |
638 | ||
639 | " Color the 100th column. | |
640 | set colorcolumn=100 | |
641 | highlight ColorColumn ctermbg = darkgray | |
642 | ||
643 | ||
644 | #### TCSH #### | |
645 | ||
646 | Install tcsh. | |
647 | ||
648 | apt-get install tcsh | |
649 | ||
650 | Change the default shell for new users by editing `/etc/adduser.conf`, setting | |
651 | the `DSHELL` variable to `/bin/tcsh`. Then use the `chsh` command to change the | |
652 | shell for root and ataylor. Create `~/.cshrc` in ataylor's and root's homedir | |
50ab1573 AT |
653 | with the following contents. Remember to also copy it to `/etc/skel` and set |
654 | permissions so it's used for any future users on the system. | |
a60cd2ef AT |
655 | |
656 | # .cshrc - csh resource script, read at beginning of execution by each shell | |
657 | ||
658 | alias h history 25 | |
659 | alias j jobs -l | |
660 | alias la ls -aF | |
661 | alias lf ls -FA | |
399284b1 | 662 | alias ll ls -lF --color |
a60cd2ef AT |
663 | alias ls ls --color |
664 | ||
665 | # These are normally set through /etc/login.conf. You may override them here | |
666 | # if wanted. | |
667 | set path = (/sbin /bin /usr/sbin /usr/bin /usr/local/sbin /usr/local/bin $HOME/bin) | |
668 | ||
669 | setenv EDITOR vim | |
670 | setenv PAGER more | |
671 | ||
672 | if ($?prompt) then | |
673 | # An interactive shell -- set some stuff up | |
674 | set prompt = "%N@%m:%~ %# " | |
675 | set promptchars = "%#" | |
676 | ||
677 | set filec | |
678 | set history = 1000 | |
679 | set savehist = (1000 merge) | |
680 | set autolist = ambiguous | |
681 | # Use history to aid expansion | |
682 | set autoexpand | |
683 | set autorehash | |
684 | set mail = (/var/mail/$USER) | |
685 | if ( $?tcsh ) then | |
686 | bindkey "^W" backward-delete-word | |
687 | bindkey -k up history-search-backward | |
688 | bindkey -k down history-search-forward | |
689 | endif | |
690 | ||
691 | endif | |
692 | ||
693 | ||
694 | #### XScreensaver #### | |
695 | ||
696 | Install Xscreensaver and configure screen locking. | |
697 | ||
698 | apt-get install xscreensaver xscreensaver-data | |
699 | ||
700 | Run `xscreensaver-demo` and select some screensavers. If inspiration doesn't | |
701 | strike, do single screensaver mode with the `abstractile` hack; it looks good | |
702 | on pretty much any hardware. Remember to enable screen locking. | |
703 | ||
704 | Add the following line to `~/.xinitrc`. | |
705 | ||
706 | /bin/xscreensaver -nosplash & | |
707 | ||
708 | ||
399284b1 AT |
709 | #### Go Toolchain #### |
710 | ||
711 | The version of Go provided via `apt-get` is always out of date, so all Go | |
712 | installs on this server are done via tarball from the <https://golang.com> | |
713 | website. Go 1.16.3 is used for this example but the newest version of Go may be | |
714 | found at <https://golang.org/dl/>. | |
715 | ||
716 | Previous versions of Go are installed entirely under `/usr/local/go`. Delete | |
717 | the entire `/usr/local/go` directory before proceeding. | |
718 | ||
719 | wget https://golang.org/dl/go1.16.3.linux-amd64.tar.gz | |
720 | tar -C /usr/local -xzf go1.16.3.linux-amd64.tar.gz | |
721 | ||
722 | If this is the first time installing Go on the system, update everyone's | |
723 | `$PATH` to include `/usr/local/go/bin`. Remember to update files under | |
724 | `/etc/skel` at the same time. | |
725 | ||
726 | ||
a60cd2ef AT |
727 | #### ZFS Snapshots #### |
728 | ||
729 | In order to configure automatic ZFS snapshots, use the `auto-zfs-snapshot` | |
730 | package. | |
731 | ||
732 | apt-get install auto-zfs-snapshot | |
733 | ||
734 | In addition to the snapshot script itself, this package includes automatically | |
735 | enabled cron entries, but it will only snapshot filesystems with the | |
736 | `com.sun:auto-snapshot` property set to `true`. Since we already manually set | |
737 | that property to `false` for `/var/cache` and `/var/tmp`, simply set it to | |
738 | `true` for the two parent pools and allow filesystems to inherit wherever | |
739 | possible. | |
740 | ||
741 | zfs set com.sun:auto-snapshot=true rpool | |
742 | zfs set com.sun:auto-snapshot=true bpool | |
743 | ||
744 | Verify that relevant filesystems inherited the property. | |
745 | ||
746 | zfs get com.sun:auto-snapshot | |
747 | ||
748 | After waiting 15+ minutes, verify that snapshots begin to appear. | |
749 | ||
750 | zfs list -t snapshot | |
751 | ||
752 | ||
753 | #### ZFS Scrubs #### | |
754 | ||
755 | Automate ZFS scrubs by creating `/etc/cron.d/zfs-scrubs` with the following | |
756 | contents. | |
757 | ||
758 | PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin | |
759 | 0 0 0 * * root /sbin/zpool scrub rpool | |
760 | 0 0 0 * * root /sbin/zpool scrub bpool | |
761 | ||
762 | ||
763 | #### Status Updates #### | |
764 | ||
765 | In order to receive status updates like failed drive notifications, we must | |
766 | first configure the system to send email through the SGK mail server. Rather | |
767 | than use `exim4` as provided by the base system, instead use `msmtp`. | |
768 | ||
769 | apt-get install msmtp-mta | |
770 | ||
771 | Create the file `/etc/msmtprc` with the following contents. | |
772 | ||
773 | # Set default values for all following accounts. | |
774 | defaults | |
775 | auth on | |
776 | tls on | |
777 | tls_trust_file /etc/ssl/certs/ca-certificates.crt | |
778 | tls_starttls off | |
779 | ||
780 | # Account: subgeniuskitty | |
781 | account default | |
782 | host mail.subgeniuskitty.com | |
783 | port 465 | |
784 | from ataylor@subgeniuskitty.com | |
785 | user ataylor@subgeniuskitty.com | |
786 | password <plaintext-password> | |
787 | ||
788 | Create the file `/etc/cron.d/status-emails` with the following contents. | |
789 | ||
790 | PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin | |
791 | SHELL=/bin/bash | |
792 | 0 0 * * 0 root /sbin/zpool status | echo -e "Subject:FROSTBURG: zpool status\n\n $(cat -)" | msmtp ataylor@subgeniuskitty.com | |
793 | ||
a60cd2ef | 794 | |
56f1fca3 AT |
795 | #### Public SSH Access #### |
796 | ||
797 | Although frostburg is on a private subnet, I want public SSH access. The | |
798 | easiest way to set this up is via a reverse SSH tunnel to one of the public | |
799 | subgeniuskitty.com servers. | |
800 | ||
801 | This section refers to three machines: | |
802 | ||
803 | - The **server** is frostburg.subgeniuskitty.com, a machine which we desire | |
804 | to access across the internet despite residing on a private subnet. | |
805 | ||
806 | - The **endpoint** is a server with public IP address which will serve as an | |
807 | access portal for the *server*. | |
808 | ||
809 | - The **client** is the human user's workstation, the machine which is | |
810 | attempting to login to the *server* via the *endpoint*. | |
811 | ||
812 | First, setup appropriate login credentials on the *server*, which in this case | |
813 | is `frostburg.subgeniuskitty.com`. Ignore any warnings about `/home/username` | |
814 | already existing or not being owned by the correct user. These are simply a | |
815 | side effect of using ZFS since we must create the homedir before adding the | |
816 | user, but we can't change ownership until after the new user exists. | |
817 | ||
818 | server:~ # zfs create rpool/home/username | |
819 | server:~ # adduser username | |
820 | server:~ # cp -a /etc/skel/. /home/username | |
821 | server:~ # chown -R username:username /home/username | |
822 | server:~ # zfs snapshot rpoot/home/username@account_creation | |
823 | ||
824 | If necessary for the intended tasks, add the user to any relevant groups with | |
825 | something like the following command. | |
826 | ||
827 | server:~ # usermod -a -G netdev,plugdev,sudo,video username | |
828 | ||
829 | The user will also need login credentials on the *endpoint*. These credentials | |
830 | don't need to allow anything other than simply SSHing through to the *server*. | |
831 | ||
832 | endpoint:~ # adduser username | |
833 | ||
834 | With appropriate credentials successfully created, move on to setting up a | |
835 | reverse SSH tunnel from *server* to *endpoint*. | |
836 | ||
837 | First, create an SSH key on the *server* with no passphrase and authorize it | |
838 | for logins on the *endpoint*. This will be used to bring the tunnel up when the | |
839 | machine boots. If a non-empty passphrase is specified, you will need to type it | |
840 | during the boot process. | |
841 | ||
842 | server:~ # ssh-keygen | |
843 | server:~ # scp /root/.ssh/id_rsa.pub username@endpoint:/home/username/temp_key_file | |
844 | server:~ # ssh username@endpoint | |
845 | (login requires password) | |
846 | endpoint:~ % mkdir -p /home/username/.ssh | |
847 | endpoint:~ % mv /home/username/temp_key_file /home/username/.ssh/authorized_keys | |
848 | endpoint:~ % logout | |
849 | server:~ # ssh username@endpoint | |
850 | (login does not require password) | |
851 | endpoint:~ % logout | |
852 | server:~ # mv /root/.ssh/id_rsa rtunnel_nopwd | |
853 | server:~ # mv /root/.ssh/id_rsa.pub rtunnel_nopwd.pub | |
854 | ||
855 | Next, create the tunnel using AutoSSH to maintain a long-term connection. | |
856 | ||
857 | server:~ # apt-get install autossh | |
858 | server:~ # vi /etc/systemd/system/autossh-tunnel.service | |
859 | [Unit] | |
860 | Description=AutoSSH tunnel between frostburg.SGK and www.SGK | |
861 | After=network-online.target | |
862 | ||
863 | [Service] | |
864 | Environment="AUTOSSH_GATETIME=0" | |
865 | ExecStart=/bin/autossh -N -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -i /root/.ssh/rtunnel_nopwd -R 4242:localhost:22 username@endpoint | |
866 | ||
867 | [Install] | |
868 | WantedBy=multi-user.target | |
869 | server:~ # systemctl daemon-reload | |
870 | server:~ # systemctl start autossh-tunnel.service | |
871 | server:~ # systemctl enable autossh-tunnel.service | |
872 | ||
873 | At this point the SSH tunnel is operational. Let's make things a little easier | |
874 | for the user by storing most of the config options in an SSH config file. | |
875 | ||
876 | endpoint:~ # su - username | |
877 | endpoint:~ % vi /home/username/.ssh/config | |
878 | Host server | |
879 | Hostname localhost | |
880 | User username | |
881 | Port 4242 | |
882 | ||
883 | Now, when we execute `ssh server`, it is equivalent to the command | |
884 | `ssh -p 4242 username@localhost`, much easier to remember. | |
885 | ||
886 | It's time to test everything out. Starting from the *client*, you should now be | |
887 | able to login to the *server* via the *endpoint*. | |
888 | ||
889 | client:~ % ssh username@endpoint | |
890 | endpoint:~ % ssh server | |
891 | server:~ % | |
892 | ||
56f1fca3 | 893 | -------------------------------------------------------------------------------- |
a60cd2ef | 894 | |
399284b1 | 895 | |
a60cd2ef AT |
896 | ## Xeon Phi Kernel Module ## |
897 | ||
31f1b002 | 898 | It appears that Linux kernel version 4.19.181 included with Debian 10.9 already |
a60cd2ef | 899 | has some sort of in-tree kernel support for these Xeon Phi coprocessor cards as |
31f1b002 AT |
900 | seen in the final lines of the following diagnostic output. Also note that the |
901 | card allocated an 8GB PCIe MMIO region, indicating that the 64-bit BAR setting | |
902 | in the BIOS is working as intended. | |
a60cd2ef AT |
903 | |
904 | root@frostburg:~ # lspci | grep -i Co-processor | |
905 | 02:00.0 Co-processor: Intel Corporation Xeon Phi coprocessor 5100 series (rev 11) | |
906 | root@frostburg:~ # lspci -s 02:00.0 -vv | |
907 | 02:00.0 Co-processor: Intel Corporation Xeon Phi coprocessor 5100 series (rev 11) | |
31f1b002 | 908 | <snip> |
a60cd2ef | 909 | Region 0: Memory at 21c00000000 (64-bit, prefetchable) [size=8G] |
31f1b002 | 910 | <snip> |
a60cd2ef AT |
911 | Kernel driver in use: mic |
912 | Kernel modules: mic_host | |
913 | ||
31f1b002 AT |
914 | However, since the Intel manuals are plastered with warnings about using exact, |
915 | sanctioned combinations of kernel module, MPSS software, and Phi firmware, I | |
916 | decided to avoid the kernel module included with the system and instead attempt | |
917 | porting the kernel module source code included with MPSS onto a newer Linux | |
918 | kernel. Once I have everything operational and understand how it *should* work, | |
919 | then I can try the open-source driver. | |
920 | ||
921 | I have updated the Intel kernel driver to work with newer Linux kernels. My | |
922 | work is based upon the kernel source included with MPSS 3.8.6, the latest/last | |
923 | release from Intel. Since the Xeon Phi x100 series is EOL, I don't think Intel | |
924 | intends to release any more versions of MPSS. Check `README.md` in my | |
925 | [xeon-phi-kernel-module](https://git.subgeniuskitty.com/xeon-phi-kernel-module/.git) | |
926 | git repo for up-to-date information regarding kernel version compatibility. | |
927 | ||
928 | Before compiling the kernel module, verify that relevant kernel headers are | |
929 | installed. | |
930 | ||
931 | % uname -a | |
932 | Linux frostburg 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux | |
933 | % dpkg -l | grep linux-header | |
934 | ii linux-headers-4.19.0-16-amd64 4.19.181-1 amd64 Header files for Linux 4.19.0-16-amd64 | |
935 | ii linux-headers-4.19.0-16-common 4.19.181-1 all Common header files for Linux 4.19.0-16 | |
936 | ii linux-headers-amd64 4.19+105+deb10u11 amd64 Header files for Linux amd64 configuration (meta-package) | |
937 | ||
938 | Download and compile my updated version of the Intel kernel driver. Sample | |
939 | compilation output is included below. | |
940 | ||
941 | % git clone git://git.subgeniuskitty.com/xeon-phi-kernel-module/ | |
942 | % cd xeon-phi-kernel-module/ | |
943 | % make clean all | |
944 | make -C /lib/modules/4.19.0-16-amd64/build M=xeon-phi-kernel-module modules \ | |
945 | INSTALL_MOD_PATH= | |
946 | make[1]: Entering directory '/usr/src/linux-headers-4.19.0-16-amd64' | |
947 | CC [M] xeon-phi-kernel-module/dma/mic_dma_lib.o | |
948 | CC [M] xeon-phi-kernel-module/dma/mic_dma_md.o | |
949 | CC [M] xeon-phi-kernel-module/host/acptboot.o | |
950 | CC [M] xeon-phi-kernel-module/host/ioctl.o | |
951 | CC [M] xeon-phi-kernel-module/host/linpm.o | |
952 | CC [M] xeon-phi-kernel-module/host/linpsmi.o | |
953 | CC [M] xeon-phi-kernel-module/host/linscif_host.o | |
954 | CC [M] xeon-phi-kernel-module/host/linsysfs.o | |
955 | CC [M] xeon-phi-kernel-module/host/linux.o | |
956 | CC [M] xeon-phi-kernel-module/host/linvcons.o | |
957 | CC [M] xeon-phi-kernel-module/host/linvnet.o | |
958 | CC [M] xeon-phi-kernel-module/host/micpsmi.o | |
959 | CC [M] xeon-phi-kernel-module/host/micscif_pm.o | |
960 | CC [M] xeon-phi-kernel-module/host/pm_ioctl.o | |
961 | CC [M] xeon-phi-kernel-module/host/pm_pcstate.o | |
962 | CC [M] xeon-phi-kernel-module/host/tools_support.o | |
963 | CC [M] xeon-phi-kernel-module/host/uos_download.o | |
964 | CC [M] xeon-phi-kernel-module/host/vhost/mic_vhost.o | |
965 | CC [M] xeon-phi-kernel-module/host/vhost/mic_blk.o | |
966 | CC [M] xeon-phi-kernel-module/host/vmcore.o | |
967 | CC [M] xeon-phi-kernel-module/micscif/micscif_api.o | |
968 | CC [M] xeon-phi-kernel-module/micscif/micscif_debug.o | |
969 | CC [M] xeon-phi-kernel-module/micscif/micscif_fd.o | |
970 | CC [M] xeon-phi-kernel-module/micscif/micscif_intr.o | |
971 | CC [M] xeon-phi-kernel-module/micscif/micscif_nm.o | |
972 | CC [M] xeon-phi-kernel-module/micscif/micscif_nodeqp.o | |
973 | CC [M] xeon-phi-kernel-module/micscif/micscif_ports.o | |
974 | CC [M] xeon-phi-kernel-module/micscif/micscif_rb.o | |
975 | CC [M] xeon-phi-kernel-module/micscif/micscif_rma_dma.o | |
976 | CC [M] xeon-phi-kernel-module/micscif/micscif_rma_list.o | |
977 | CC [M] xeon-phi-kernel-module/micscif/micscif_rma.o | |
978 | CC [M] xeon-phi-kernel-module/micscif/micscif_select.o | |
979 | CC [M] xeon-phi-kernel-module/micscif/micscif_smpt.o | |
980 | CC [M] xeon-phi-kernel-module/micscif/micscif_sysfs.o | |
981 | CC [M] xeon-phi-kernel-module/micscif/micscif_va_gen.o | |
982 | CC [M] xeon-phi-kernel-module/micscif/micscif_va_node.o | |
983 | CC [M] xeon-phi-kernel-module/vnet/micveth_dma.o | |
984 | CC [M] xeon-phi-kernel-module/vnet/micveth_param.o | |
985 | LD [M] xeon-phi-kernel-module/mic.o | |
986 | Building modules, stage 2. | |
987 | MODPOST 1 modules | |
988 | CC xeon-phi-kernel-module/mic.mod.o | |
989 | LD [M] xeon-phi-kernel-module/mic.ko | |
990 | make[1]: Leaving directory '/usr/src/linux-headers-4.19.0-16-amd64' | |
991 | ||
992 | At this point you can manually load/install the new kernel module (`mic.ko`) | |
993 | which is found in the current directory, or execute `make install`. The latter | |
994 | command also installs the SCIF header file, as well as putting some config files | |
995 | under `/usr/local/etc/`. The information in those config files won't be picked | |
996 | up by the system (we will install configs in the correct location in a moment), | |
997 | but it is useful as a reference. Sample `make install` output is shown below. | |
998 | ||
999 | # make install | |
1000 | make -C /lib/modules/4.19.0-16-amd64/build M=/home/ataylor/xeon-phi-kernel-module modules_install \ | |
1001 | INSTALL_MOD_PATH= | |
1002 | make[1]: Entering directory '/usr/src/linux-headers-4.19.0-16-amd64' | |
1003 | INSTALL /home/ataylor/xeon-phi-kernel-module/mic.ko | |
1004 | DEPMOD 4.19.0-16-amd64 | |
1005 | Warning: modules_install: missing 'System.map' file. Skipping depmod. | |
1006 | make[1]: Leaving directory '/usr/src/linux-headers-4.19.0-16-amd64' | |
1007 | install -d /usr/local/etc/sysconfig/modules | |
1008 | install mic.modules /usr/local/etc/sysconfig/modules | |
1009 | install -d /usr/local/etc/modprobe.d | |
1010 | install -m644 mic.conf /usr/local/etc/modprobe.d | |
1011 | install -d /usr/local/etc/udev/rules.d | |
1012 | install -m644 udev-mic.rules /usr/local/etc/udev/rules.d/50-udev-mic.rules | |
1013 | install -d /lib/modules/4.19.0-16-amd64 | |
1014 | install -m644 Module.symvers /lib/modules/4.19.0-16-amd64/scif.symvers | |
1015 | install -d /usr/src/linux-headers-4.19.0-16-amd64/include/modules | |
1016 | install -m644 include/scif.h /usr/src/linux-headers-4.19.0-16-amd64/include/modules | |
1017 | ||
1018 | Create the file `/etc/modprobe.d/mic.conf` with the following contents, | |
1019 | intended to accomplish two things. First, blacklist the in-tree MIC kernel | |
1020 | module that shipped with our kernel, including all associated modules, and | |
1021 | second, configure the Intel MIC kernel module which we just built and installed. | |
1022 | The options shown are drawn from the defaults in | |
1023 | `/usr/local/etc/modprobe.d/mic.conf`. | |
1024 | ||
1025 | # Blacklist the in-tree kernel modules associated with the Knight's Corner Xeon | |
1026 | # Phi so that we can load the Intel kernel module. | |
1027 | ||
1028 | # These two modules depend on the various bus modules that follow. | |
1029 | blacklist mic_host | |
1030 | blacklist mic_x100_dma | |
1031 | ||
1032 | blacklist cosm_bus | |
1033 | blacklist vop_bus | |
1034 | blacklist scif_bus | |
1035 | blacklist mic_bus | |
1036 | ||
1037 | # ^^^------ Blacklisting the in-tree MIC kernel module. | |
1038 | # ============================================================================== | |
1039 | # vvv------ Configuring the Intel MIC kernel module. | |
1040 | ||
1041 | # The following options apply to the Intel Many Integrated Core (MIC) driver. | |
1042 | # Unless otherwise noted, the value "1" enables the feature and "0" disables | |
1043 | # it. | |
1044 | # | |
1045 | # Option: p2p | |
1046 | # Description: Enables use of SCIF interface peer to peer communication. | |
1047 | # | |
1048 | # Option: p2p_proxy | |
1049 | # Description: Enables use of SCIF P2P Proxy DMA which converts DMA | |
1050 | # reads into DMA writes for performance on certain Intel | |
1051 | # platforms. | |
1052 | # | |
1053 | # Option: reg_cache | |
1054 | # Description: Enables SCIF Registration Caching. | |
1055 | # | |
1056 | # Option: huge_page | |
1057 | # Description: Enables SCIF Huge Page Support. | |
1058 | # | |
1059 | # Option: watchdog | |
1060 | # Description: Enables SCIF watchdog for Lost Node detection. | |
1061 | # | |
1062 | # Option: watchdog_auto_reboot | |
1063 | # Description: Configures behavior of MIC host driver upon detection of a lost | |
1064 | # node. This option is a nop if watchdog=0. Setting value "1" | |
1065 | # allows host driver to reboot node back to "online" state, | |
1066 | # whereas value "0" only allows the host driver to reset the node | |
1067 | # back to "ready" state, leaving the user responsible for rebooting | |
1068 | # the node (or not). | |
1069 | # | |
1070 | # Option: crash_dump | |
1071 | # Description: Enables uOS Kernel Crash Dump Captures. | |
1072 | # | |
1073 | # Option: ulimit | |
1074 | # Description: Enables ulimit checks on max locked memory for scif_register. | |
1075 | # | |
1076 | options mic reg_cache=1 huge_page=1 watchdog=1 watchdog_auto_reboot=1 crash_dump=1 p2p=1 p2p_proxy=1 ulimit=0 | |
1077 | options mic_host reg_cache=1 huge_page=1 watchdog=1 watchdog_auto_reboot=1 crash_dump=1 p2p=1 p2p_proxy=1 ulimit=0 | |
1078 | ||
1079 | Finally, add the line `mic` to the file `/etc/modules-load.d/modules.conf`, | |
1080 | instructing the system to load this kernel module on boot, then run `depmod` to | |
1081 | ensure the system is aware of the new kernel module, followed by a reboot to | |
1082 | verify everything works. | |
1083 | ||
1084 | After the system comes back up, verify that the module loaded with your desired | |
1085 | options using the `systool` command, sample output below. | |
1086 | ||
1087 | # systool -v -m mic | |
1088 | Module = "mic" | |
1089 | ||
1090 | Attributes: | |
1091 | coresize = "741376" | |
1092 | initsize = "0" | |
1093 | initstate = "live" | |
1094 | refcnt = "0" | |
1095 | taint = "OE" | |
1096 | uevent = <store method only> | |
1097 | ||
1098 | Parameters: | |
1099 | crash_dump = "Y" | |
1100 | huge_page = "Y" | |
1101 | msi = "Y" | |
1102 | p2p_proxy = "Y" | |
1103 | p2p = "Y" | |
1104 | pm_qos_cpu_dma_lat = "-1" | |
1105 | psmi = "N" | |
1106 | ramoops_count = "4" | |
1107 | reg_cache = "Y" | |
1108 | ulimit = "N" | |
1109 | vnet = "dma" | |
1110 | vnet_addr = "0" | |
1111 | vnet_num_buffers = "62" | |
1112 | watchdog_auto_reboot= "Y" | |
1113 | watchdog = "Y" | |
1114 | ||
1115 | Sections: | |
1116 | <snip> | |
1117 | ||
1118 | ||
1119 | -------------------------------------------------------------------------------- | |
1120 | ||
1121 | ||
1122 | ## Intel MPSS ## | |
a60cd2ef | 1123 |