KNEM

Documentation


See also the interface documentation for details on how to use KNEM within your applications or MPI implementations.


Installing KNEM

KNEM configures, builds and installs as usual.

 $ ./configure --prefix=/opt/knem
 $ make
 $ make install

If building from SVN, you may need to install automake >=1.10 and autoconf >=2.61, and then prepare the script before configure:

 $ ./autogen.sh

Everything is installed in the prefix directory (/opt/knem by default), including the kernel module and udev rules file. If you want modprobe and udev to properly handle KNEM, the relevant files should be installed in /lib/modules and /etc/udev/rules.d instead. To do so:

 $ /path/to/knem/install/sbin/knem_local_install

Usually, make install will run only once per cluster and install in a shared directory, while knem_local_install will run once per machine since /lib/modules and /etc/udev/rules.d are not shared between machines.

If you ever want to uninstall KNEM, you may run:

 $ /path/to/knem/install/sbin/knem_local_install --uninstall
 $ make uninstall

It is also possible to build KNEM as a RPM package which bundles the result of both make install and knem_local_install:

 $ rpmbuild -tb knem-<version>.tar.gz

KNEM also supports DKMS (Dynamic Kernel Module Support) which lets the system automatically rebuild and install modules for new kernels. See dkms.conf for details.

Starting KNEM

To start KNEM, assuming you ran knem_local_install, just run:

 $ modprobe knem

By default, udev will set the /dev/knem device group to rdma. Thus any KNEM user should be added to this group first.

If you did not run knem_local_install, you may still load the kernel module and setup the device group manually:

 $ insmod /path/to/knem/install/lib/modules/$(uname -r)/knem.ko
 $ chgrp rdma /dev/knem

Configuring your MPI implementation to use KNEM

MPICH2 supports KNEM since version 1.1.1 (it is therefore also supported by MVAPICH2 1.5). To use it, you should add the following options to the configure line:

 --with-nemesis-local-lmt=dma --with-knem=/path/to/knem/install/

It will use KNEM starting at 64kB messages, and offload copies to DMA engine after 2MB. To change the latter threshold, use the environment variable MPICH_NEM_LMT_DMA_THRESHOLD.


Open MPI supports KNEM since version 1.5.0. To use it, you should add the following option to the configure line:

 --with-knem=/path/to/knem/install/

KNEM will be used at runtime after 4kB. You may change this threshold with --mca btl_sm_eager_limit 32768. You may also disable KNEM entirely at runtime with --mca btl_sm_use_knem 0. KNEM will not offload copies to DMA engine by default. You should use --mca btl_sm_knem_dma_min 1048576 to enable copy offload starting at 1MB.


To check that KNEM is properly used, see Getting Statistics. Make sure your users are members of the rdma group so that they are granted access to the /dev/knem device file.

If you want to port your MPI implementation to KNEM, see the interface documentation.

Getting Statistics

To check that KNEM is properly used, how it processed requests, why some of them failed, you may read statistics:

  $ cat /dev/knem
  [...]
   Requests Submitted          : 10243
   Requests Processed/DMA      : 5621
   Requests Processed/Thread   : 2
  [...]
   Requests Failed/ReadCmd     : 1
   Requests Failed/FindRegion  : 3

These statistics may also be cleared by a privileged user:

  # echo 1 > /dev/knem 

If the knem module was loaded with parameter statsverbose=1, reading from /dev/knem will also list the currently open memory regions for the current process (and even from all processes if the user is privileged).

  # cat /dev/knem
  [...]
   Pid 5d343d48 (#6)
    Region 5d344ad1 SingleUse
    Listed 1 regions
   Pid 5d343d49 (#7)
    Listed 0 regions
   Listed 8 pids

Last updated on 2010/08/25.