Skip to content

Commit 8700e3e

Browse files
monis410dledford
monis410
authored and
dledford
committed
Soft RoCE driver
Soft RoCE (RXE) - The software RoCE driver ib_rxe implements the RDMA transport and registers to the RDMA core device as a kernel verbs provider. It also implements the packet IO layer. On the other hand ib_rxe registers to the Linux netdev stack as a udp encapsulating protocol, in that case RDMA, for sending and receiving packets over any Ethernet device. This yields a RDMA transport over the UDP/Ethernet network layer forming a RoCEv2 compatible device. The configuration procedure of the Soft RoCE drivers requires binding to any existing Ethernet network device. This is done with /sys interface. A userspace Soft RoCE library (librxe) provides user applications the ability to run with Soft RoCE devices. The use of rxe verbs ins user space requires the inclusion of librxe as a device specifics plug-in to libibverbs. librxe is packaged separately. Architecture: +-----------------------------------------------------------+ | Application | +-----------------------------------------------------------+ +-----------------------------------+ | libibverbs | User +-----------------------------------+ +----------------+ +----------------+ | librxe | | HW RoCE lib | +----------------+ +----------------+ +---------------------------------------------------------------+ +--------------+ +------------+ | Sockets | | RDMA ULP | +--------------+ +------------+ +--------------+ +---------------------+ | TCP/IP | | ib_core | +--------------+ +---------------------+ +------------+ +----------------+ Kernel | ib_rxe | | HW RoCE driver | +------------+ +----------------+ +------------------------------------+ | NIC driver | +------------------------------------+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +-----------------------------------------------------------+ | Application | +-----------------------------------------------------------+ +-----------------------------------+ | libibverbs | User +-----------------------------------+ +----------------+ +----------------+ | librxe | | HW RoCE lib | +----------------+ +----------------+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +--------------+ +------------+ | Sockets | | RDMA ULP | +--------------+ +------------+ +--------------+ +---------------------+ | TCP/IP | | ib_core | +--------------+ +---------------------+ +------------+ +----------------+ Kernel | ib_rxe | | HW RoCE driver | +------------+ +----------------+ +------------------------------------+ | NIC driver | +------------------------------------+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Soft RoCE resources: [1[ https://github.com/SoftRoCE/librxe-dev librxe - source code in Github [2] https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home - Soft RoCE Wiki page [3] https://github.com/SoftRoCE/librxe-dev - Soft RoCE userspace library Signed-off-by: Kamal Heib <[email protected]> Signed-off-by: Amir Vadai <[email protected]> Signed-off-by: Moni Shoua <[email protected]> Reviewed-by: Haggai Eran <[email protected]> Signed-off-by: Doug Ledford <[email protected]>
1 parent 33688ab commit 8700e3e

38 files changed

+13039
-0
lines changed

‎MAINTAINERS

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7444,6 +7444,15 @@ W: http://www.mellanox.com
74447444
Q: http://patchwork.ozlabs.org/project/netdev/list/
74457445
F: drivers/net/ethernet/mellanox/mlxsw/
74467446

7447+
SOFT-ROCE DRIVER (rxe)
7448+
M: Moni Shoua <[email protected]>
7449+
L: [email protected]
7450+
S: Supported
7451+
W: https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home
7452+
Q: http://patchwork.kernel.org/project/linux-rdma/list/
7453+
F: drivers/infiniband/hw/rxe/
7454+
F: include/uapi/rdma/rdma_user_rxe.h
7455+
74477456
MEMBARRIER SUPPORT
74487457
M: Mathieu Desnoyers <[email protected]>
74497458
M: "Paul E. McKenney" <[email protected]>

‎drivers/infiniband/Kconfig

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,7 @@ source "drivers/infiniband/ulp/iser/Kconfig"
8484
source "drivers/infiniband/ulp/isert/Kconfig"
8585

8686
source "drivers/infiniband/sw/rdmavt/Kconfig"
87+
source "drivers/infiniband/sw/rxe/Kconfig"
8788

8889
source "drivers/infiniband/hw/hfi1/Kconfig"
8990

‎drivers/infiniband/sw/Makefile

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,2 @@
11
obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt/
2+
obj-$(CONFIG_RDMA_RXE) += rxe/

‎drivers/infiniband/sw/rxe/Kconfig

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
config RDMA_RXE
2+
tristate "Software RDMA over Ethernet (RoCE) driver"
3+
depends on INET && PCI && INFINIBAND
4+
depends on NET_UDP_TUNNEL
5+
---help---
6+
This driver implements the InfiniBand RDMA transport over
7+
the Linux network stack. It enables a system with a
8+
standard Ethernet adapter to interoperate with a RoCE
9+
adapter or with another system running the RXE driver.
10+
Documentation on InfiniBand and RoCE can be downloaded at
11+
www.infinibandta.org and www.openfabrics.org. (See also
12+
siw which is a similar software driver for iWARP.)
13+
14+
The driver is split into two layers, one interfaces with the
15+
Linux RDMA stack and implements a kernel or user space
16+
verbs API. The user space verbs API requires a support
17+
library named librxe which is loaded by the generic user
18+
space verbs API, libibverbs. The other layer interfaces
19+
with the Linux network stack at layer 3.
20+
21+
To configure and work with soft-RoCE driver please use the
22+
following wiki page under "configure Soft-RoCE (RXE)" section:
23+
24+
https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home

‎drivers/infiniband/sw/rxe/Makefile

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
obj-$(CONFIG_RDMA_RXE) += rdma_rxe.o
2+
3+
rdma_rxe-y := \
4+
rxe.o \
5+
rxe_comp.o \
6+
rxe_req.o \
7+
rxe_resp.o \
8+
rxe_recv.o \
9+
rxe_pool.o \
10+
rxe_queue.o \
11+
rxe_verbs.o \
12+
rxe_av.o \
13+
rxe_srq.o \
14+
rxe_qp.o \
15+
rxe_cq.o \
16+
rxe_mr.o \
17+
rxe_dma.o \
18+
rxe_opcode.o \
19+
rxe_mmap.o \
20+
rxe_icrc.o \
21+
rxe_mcast.o \
22+
rxe_task.o \
23+
rxe_net.o \
24+
rxe_sysfs.o

0 commit comments

Comments
 (0)