INFINIBAND: How To Check Status Of Infiniband Links On the Nodes/Hosts Using Infiniband Network? ("iblinkinfo" Command)

Below iblinkinfo command can be used.

iblinkinfo | egrep "HCA|PCIe"

Below is example snippet of above command from Exalogic rack using Infiniband networks. testingcn01-cn08 are Linux nodes and testingsn01 and sn02 are Storage nodes with IB HCA/PCIe cards. Below output shows both the links to all the linux nodes and storage heads are up.

# iblinkinfo | egrep "HCA|PCIe"
CA: testingsn01 PCIe 1:
CA: testingsn02 PCIe 1:
CA: testingcn08 EL-C 192.168.5.108 HCA-1:
          12    5[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>      21    2[  ] "testingcn02 EL-C 192.168.5.102 HCA-1" ( )
          12    6[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>      19    2[  ] "testingcn01 EL-C 192.168.5.101 HCA-1" ( )
          12    7[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>      30    2[  ] "testingcn04 EL-C 192.168.5.104 HCA-1" ( )
          12    8[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>      28    2[  ] "testingcn03 EL-C 192.168.5.103 HCA-1" ( )
          12    9[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       6    2[  ] "testingcn06 EL-C 192.168.5.106 HCA-1" ( )
          12   10[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>      13    2[  ] "testingcn05 EL-C 192.168.5.105 HCA-1" ( )
          12   11[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>      17    2[  ] "testingcn07 EL-C 192.168.5.107 HCA-1" ( )
          12   18[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>      15    2[  ] "testingcn08 EL-C 192.168.5.108 HCA-1" ( )
          12   31[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       4    2[  ] "testingsn02 PCIe 1" ( )
          12   32[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       2    2[  ] "testingsn01 PCIe 1" ( )
CA: testingcn07 EL-C 192.168.5.107 HCA-1:
CA: testingcn05 EL-C 192.168.5.105 HCA-1:
CA: testingcn06 EL-C 192.168.5.106 HCA-1:
CA: testingcn03 EL-C 192.168.5.103 HCA-1:
CA: testingcn04 EL-C 192.168.5.104 HCA-1:
CA: testingcn02 EL-C 192.168.5.102 HCA-1:
          27    5[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>      20    1[  ] "testingcn02 EL-C 192.168.5.102 HCA-1" ( )
          27    6[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>      18    1[  ] "testingcn01 EL-C 192.168.5.101 HCA-1" ( )
          27    7[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>      29    1[  ] "testingcn04 EL-C 192.168.5.104 HCA-1" ( )
          27    8[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>      22    1[  ] "testingcn03 EL-C 192.168.5.103 HCA-1" ( )
          27    9[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       5    1[  ] "testingcn06 EL-C 192.168.5.106 HCA-1" ( )
          27   10[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       7    1[  ] "testingcn05 EL-C 192.168.5.105 HCA-1" ( )
          27   11[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>      16    1[  ] "testingcn07 EL-C 192.168.5.107 HCA-1" ( )
          27   18[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>      14    1[  ] "testingcn08 EL-C 192.168.5.108 HCA-1" ( )
          27   31[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       3    1[  ] "testingsn02 PCIe 1" ( )
          27   32[  ] ==( 4X          10.0 Gbps Active/  LinkUp)==>       1    1[  ] "testingsn01 PCIe 1" ( )
CA: testingcn01 EL-C 192.168.5.101 HCA-1:

Products to which Article Applies

All Hosts using Infiniband Fabric, Oracle Engineered Systems using Infiniband Fabric.

Additional Reference

https://docs.oracle.com/cd/E86824_01/html/E54764/iblinkinfo-1m.html

 

tarun boyella