summaryrefslogtreecommitdiff
path: root/Documentation/mic
diff options
context:
space:
mode:
authorLinus Torvalds <torvalds@linux-foundation.org>2015-06-27 00:51:15 +0300
committerLinus Torvalds <torvalds@linux-foundation.org>2015-06-27 00:51:15 +0300
commitd87823813fe498fdd47894bd28e460a9dee8d771 (patch)
tree214eaf3babd0d61f08022fc1edd99a5128616548 /Documentation/mic
parente382608254e06c8109f40044f5e693f2e04f3899 (diff)
parent3dc196eae1db548f05e53e5875ff87b8ff79f249 (diff)
downloadlinux-d87823813fe498fdd47894bd28e460a9dee8d771.tar.xz
Merge tag 'char-misc-4.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc
Pull char/misc driver updates from Greg KH: "Here's the big char/misc driver pull request for 4.2-rc1. Lots of mei, extcon, coresight, uio, mic, and other driver updates in here. Full details in the shortlog. All of these have been in linux-next for some time with no reported problems" * tag 'char-misc-4.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc: (176 commits) mei: me: wait for power gating exit confirmation mei: reset flow control on the last client disconnection MAINTAINERS: mei: add mei_cl_bus.h to maintained file list misc: sram: sort and clean up included headers misc: sram: move reserved block logic out of probe function misc: sram: add private struct device and virt_base members misc: sram: report correct SRAM pool size misc: sram: bump error message level on unclean driver unbinding misc: sram: fix device node reference leak on error misc: sram: fix enabled clock leak on error path misc: mic: Fix reported static checker warning misc: mic: Fix randconfig build error by including errno.h uio: pruss: Drop depends on ARCH_DAVINCI_DA850 from config uio: pruss: Add CONFIG_HAS_IOMEM dependence uio: pruss: Include <linux/sizes.h> extcon: Redefine the unique id of supported external connectors without 'enum extcon' type char:xilinx_hwicap:buffer_icap - change 1/0 to true/false for bool type variable in function buffer_icap_set_configuration(). Drivers: hv: vmbus: Allocate ring buffer memory in NUMA aware fashion parport: check exclusive access before register w1: use correct lock on error in w1_seq_show() ...
Diffstat (limited to 'Documentation/mic')
-rw-r--r--Documentation/mic/mic_overview.txt28
-rwxr-xr-xDocumentation/mic/mpssd/mpss24
-rw-r--r--Documentation/mic/scif_overview.txt98
3 files changed, 125 insertions, 25 deletions
diff --git a/Documentation/mic/mic_overview.txt b/Documentation/mic/mic_overview.txt
index 77c541802ad9..1a2f2c8ec59e 100644
--- a/Documentation/mic/mic_overview.txt
+++ b/Documentation/mic/mic_overview.txt
@@ -24,6 +24,10 @@ a virtual bus called mic bus is created and virtual dma devices are
created on it by the host/card drivers. On host the channels are private
and used only by the host driver to transfer data for the virtio devices.
+The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a
+low level communications API across PCIe currently implemented for MIC.
+More details are available at scif_overview.txt.
+
Here is a block diagram of the various components described above. The
virtio backends are situated on the host rather than the card given better
single threaded performance for the host compared to MIC, the ability of
@@ -47,18 +51,18 @@ the fact that the virtio block storage backend can only be on the host.
| | | Virtio over PCIe IOCTLs |
| | +--------------------------+
+-----------+ | | | +-----------+
-| MIC DMA | | | | | MIC DMA |
-| Driver | | | | | Driver |
-+-----------+ | | | +-----------+
- | | | | |
-+---------------+ | | | +----------------+
-|MIC virtual Bus| | | | |MIC virtual Bus |
-+---------------+ | | | +----------------+
- | | | | |
- | +--------------+ | +---------------+ |
- | |Intel MIC | | |Intel MIC | |
- +---|Card Driver | | |Host Driver | |
- +--------------+ | +---------------+-----+
+| MIC DMA | | +----------+ | +-----------+ | | MIC DMA |
+| Driver | | | SCIF | | | SCIF | | | Driver |
++-----------+ | +----------+ | +-----------+ | +-----------+
+ | | | | | | |
++---------------+ | +-----+-----+ | +-----+-----+ | +---------------+
+|MIC virtual Bus| | |SCIF HW Bus| | |SCIF HW BUS| | |MIC virtual Bus|
++---------------+ | +-----------+ | +-----+-----+ | +---------------+
+ | | | | | | |
+ | +--------------+ | | | +---------------+ |
+ | |Intel MIC | | | | |Intel MIC | |
+ +---|Card Driver +----+ | | |Host Driver | |
+ +--------------+ | +----+---------------+-----+
| | |
+-------------------------------------------------------------+
| |
diff --git a/Documentation/mic/mpssd/mpss b/Documentation/mic/mpssd/mpss
index cacbdb0aefb9..582aad4811ae 100755
--- a/Documentation/mic/mpssd/mpss
+++ b/Documentation/mic/mpssd/mpss
@@ -35,6 +35,7 @@
exec=/usr/sbin/mpssd
sysfs="/sys/class/mic"
+mic_modules="mic_host mic_x100_dma scif"
start()
{
@@ -48,18 +49,15 @@ start()
fi
echo -e $"Starting MPSS Stack"
- echo -e $"Loading MIC_X100_DMA & MIC_HOST Modules"
+ echo -e $"Loading MIC drivers:" $mic_modules
- for f in "mic_host" "mic_x100_dma"
- do
- modprobe $f
- RETVAL=$?
- if [ $RETVAL -ne 0 ]; then
- failure
- echo
- return $RETVAL
- fi
- done
+ modprobe -a $mic_modules
+ RETVAL=$?
+ if [ $RETVAL -ne 0 ]; then
+ failure
+ echo
+ return $RETVAL
+ fi
# Start the daemon
echo -n $"Starting MPSSD "
@@ -170,8 +168,8 @@ unload()
stop
sleep 5
- echo -n $"Removing MIC_HOST & MIC_X100_DMA Modules: "
- modprobe -r mic_host mic_x100_dma
+ echo -n $"Removing MIC drivers:" $mic_modules
+ modprobe -r $mic_modules
RETVAL=$?
[ $RETVAL -ne 0 ] && failure || success
echo
diff --git a/Documentation/mic/scif_overview.txt b/Documentation/mic/scif_overview.txt
new file mode 100644
index 000000000000..0a280d986731
--- /dev/null
+++ b/Documentation/mic/scif_overview.txt
@@ -0,0 +1,98 @@
+The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low
+level communications API across PCIe currently implemented for MIC. Currently
+SCIF provides inter-node communication within a single host platform, where a
+node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of
+communicating over the PCIe bus while providing an API that is symmetric
+across all the nodes in the PCIe network. An important design objective for SCIF
+is to deliver the maximum possible performance given the communication
+abilities of the hardware. SCIF has been used to implement an offload compiler
+runtime and OFED support for MPI implementations for MIC coprocessors.
+
+==== SCIF API Components ====
+The SCIF API has the following parts:
+1. Connection establishment using a client server model
+2. Byte stream messaging intended for short messages
+3. Node enumeration to determine online nodes
+4. Poll semantics for detection of incoming connections and messages
+5. Memory registration to pin down pages
+6. Remote memory mapping for low latency CPU accesses via mmap
+7. Remote DMA (RDMA) for high bandwidth DMA transfers
+8. Fence APIs for RDMA synchronization
+
+SCIF exposes the notion of a connection which can be used by peer processes on
+nodes in a SCIF PCIe "network" to share memory "windows" and to communicate. A
+process in a SCIF node initiates a SCIF connection to a peer process on a
+different node via a SCIF "endpoint". SCIF endpoints support messaging APIs
+which are similar to connection oriented socket APIs. Connected SCIF endpoints
+can also register local memory which is followed by data transfer using either
+DMA, CPU copies or remote memory mapping via mmap. SCIF supports both user and
+kernel mode clients which are functionally equivalent.
+
+==== SCIF Performance for MIC ====
+DMA bandwidth comparison between the TCP (over ethernet over PCIe) stack versus
+SCIF shows the performance advantages of SCIF for HPC applications and runtimes.
+
+ Comparison of TCP and SCIF based BW
+
+ Throughput (GB/sec)
+ 8 + PCIe Bandwidth ******
+ + TCP ######
+ 7 + ************************************** SCIF %%%%%%
+ | %%%%%%%%%%%%%%%%%%%
+ 6 + %%%%
+ | %%
+ | %%%
+ 5 + %%
+ | %%
+ 4 + %%
+ | %%
+ 3 + %%
+ | %
+ 2 + %%
+ | %%
+ | %
+ 1 +
+ + ######################################
+ 0 +++---+++--+--+-+--+--+-++-+--+-++-+--+-++-+-
+ 1 10 100 1000 10000 100000
+ Transfer Size (KBytes)
+
+SCIF allows memory sharing via mmap(..) between processes on different PCIe
+nodes and thus provides bare-metal PCIe latency. The round trip SCIF mmap
+latency from the host to an x100 MIC for an 8 byte message is 0.44 usecs.
+
+SCIF has a user space library which is a thin IOCTL wrapper providing a user
+space API similar to the kernel API in scif.h. The SCIF user space library
+is distributed @ https://software.intel.com/en-us/mic-developer
+
+Here is some pseudo code for an example of how two applications on two PCIe
+nodes would typically use the SCIF API:
+
+Process A (on node A) Process B (on node B)
+
+/* get online node information */
+scif_get_node_ids(..) scif_get_node_ids(..)
+scif_open(..) scif_open(..)
+scif_bind(..) scif_bind(..)
+scif_listen(..)
+scif_accept(..) scif_connect(..)
+/* SCIF connection established */
+
+/* Send and receive short messages */
+scif_send(..)/scif_recv(..) scif_send(..)/scif_recv(..)
+
+/* Register memory */
+scif_register(..) scif_register(..)
+
+/* RDMA */
+scif_readfrom(..)/scif_writeto(..) scif_readfrom(..)/scif_writeto(..)
+
+/* Fence DMAs */
+scif_fence_signal(..) scif_fence_signal(..)
+
+mmap(..) mmap(..)
+
+/* Access remote registered memory */
+
+/* Close the endpoints */
+scif_close(..) scif_close(..)