Infiniband (IB) network on the nodes
Contents
19. Infiniband (IB) network on the nodes¶
We have node03 - node09 connected to FDR10 infiniband switch. The nodes now can be interconnected at substantially higher speed.
We’ll use the 192.168.7.x subnet. The last digit should match your node number.
The hostnames associated with the IP addresses go as node03ib, node04ib, …, node09ib.
This doesn’t apply to the LXC containers since they are not connected to IB.
19.1. Manual IB network setup on the nodes¶
The procedure below should be done on the nodes only.
Install IB related deb packages:
sudo apt install infiniband-diags ibutils
Install useful network tools:
sudo apt install net-tools iperf
Check if the IB interface is seen by the operating system:
sudo ibstat
You should see the output similar to below:
output from ibstat:
CA 'mlx4_0'
CA type: MT4099
Number of ports: 1
Firmware version: 2.42.5000
Hardware version: 1
Node GUID: 0x506b4b03007f8120
System image GUID: 0x506b4b03007f8123
Port 1:
State: Active
Physical state: LinkUp
Rate: 40 (FDR10)
Base lid: 34
LMC: 0
SM lid: 1
Capability mask: 0x02514868
Port GUID: 0x506b4b03007f8121
Link layer: InfiniBand
To configure a new network interface for IB, add the following block in the bottom of your /etc/network/interfaces file. Make sure to change IP address 192.168.7.7 for the correct one.
The last digit should match your node number.
/etc/network/interfaces:
auto ibp2s0
iface ibp2s0 inet static
address 192.168.7.7
netmask 255.255.255.0
pre-up echo connected > /sys/class/net/ibp2s0/mode
Load the kernel module for IP over IB:
modprobe ib_ipoib
Enable the new interface, ibp2s0:
ifup ibp2s0
Check if the interface got the assigned IP address:
ifconfig ibp2s0
or similarly:
ip addr show dev ibp2s0
It should show the IP address you assigned in /etc/network/interfaces file.
19.2. IB network setup on the nodes by using Ansible playbook¶
Below is the playbook, IB_add.yml, to accomplish the same tasks.
Use it in case you rebuild the node and need quick IB deployment.
---
- name: enable IB network interface
hosts: ceph_nodes
vars:
IP: "{{ ansible_default_ipv4.address }}"
tasks:
- name: Install IB packages
ansible.builtin.apt:
pkg:
- infiniband-diags
- ibutils
- name: Install network tools
ansible.builtin.apt:
pkg:
- net-tools
- iperf
- name: Define IB ip address for the host. Its on 192.168.7. network
ansible.builtin.set_fact:
IBip: "{{ IP.split('.')[0] }}.{{ IP.split('.')[1] }}.7.{{ IP.split('.')[3] }}"
- name: modify /etc/network/interfaces
ansible.builtin.blockinfile:
path: /etc/network/interfaces
block: |
auto ibp2s0
iface ibp2s0 inet static
address {{ IBip }}
netmask 255.255.255.0
pre-up echo connected > /sys/class/net/ibp2s0/mode
- name: load IP over IB kernel module
ansible.builtin.command: modprobe ib_ipoib
- name: bring up the IB interface
ansible.builtin.command: ifup ibp2s0
To deploy the IB on the nodes, run command
ansible-playbook IB_add.yml
19.3. Check the speed of the IP over IB network¶
On one of the nodes, for example node07, run iperf in the server mode:
sudo iperf -s
On the other node, for example node08, connect to the first node with iperf -c by using the IB ip address or name:
iperf -c node07ib
or
iperf -c 192.168.7.7
You should see about 20 Gbits/sec speed.
Run the same iperf -c command on the default hostname to compare the performance of the 1 Gbit interface:
iperf -c node07
This should show about 940 Mbits/sec.
19.4. The IB network for Ceph benchmarks¶
You can replace both the public and the cluster network on the ceph nodes with the IB network, 192.168.7.0, on the nodes and run benchmarks with the python scripts. Keep in mind that the LXC can’t use IB.