Blog

※本ブログの内容は個人的見解であり、所属組織及び企業の意見を代弁するものではありません。 / # The contents expressed here are only my personal opinions.

NICでVLAN毎にQoS(帯域上限)を設定する方法(Linux)

VLAN毎にQoS(帯域上限)をNICで設定する例を説明します。

例えば40GbEネットワークを検討する場合、ある特定のトラフィックが帯域を使い切ってしまうことが懸念され、10GbE複数本を統合するのに躊躇してしまう、というような場合、QoS設定をしておくことで帯域をトラフィック毎に使い分けることができます。（実際には非常に速いオールフラッシュストレージ等を接続しない限り、40Gbpsを使い切ることはめったにないと思いますが、運用上安心できると思います）

※Mellanox NICの場合の例

構成例：

  • OS : Ubuntu14.04LTS
  • NIC : Mellanox ConnectX-3Pro (40GbE) Dual Port
  • Driver : MLNX OFED 2.2-1.0.1
  • MTU=1500 byte (デフォルト)

 QoS設定例：

f:id:kz_tomo:20140811165926j:plain

VLAN=10 (IP address=12.0.10.2): 1Gbps

VLAN=20 (IP address=12.0.20.2): 5Gbps

VLAN=30 (IP address=12.0.30.2): 帯域制限なし

 ※下記では一方のNICのみの設定ですが、通信ペア共に設定します（NICからの送信帯域を制限する設定）。

1.VLANインタフェース作成

# vconfig add eth1 10
# vconfig add eth1 20
# vconfig add eth1 30

2.IPアドレス割り当て（ここまではQoSとは直接関係ない設定）

# ifconfig eth1.10 12.0.10.2 netmask 255.255.255.0
# ifconfig eth1.20 12.0.20.2 netmask 255.255.255.0
# ifconfig eth1.30 12.0.30.2 netmask 255.255.255.0

3.送信Socket priorityをUser priority(UP)に割り当て

//VLAN 10 : socket priority 0-7(all) => user priority 1
//VLAN 20 : socket priority 0-7(all) => user priority 3
//VLAN 30 : socket priority 0-7(all) => user priority 5

# for i in {0..7}; do vconfig set_egress_map eth1.10 $i 1 ; done
# for i in {0..7}; do vconfig set_egress_map eth1.20 $i 3 ; done
# for i in {0..7}; do vconfig set_egress_map eth1.30 $i 5 ; done

＜設定確認＞

送信priority 0~7とuser priorityとのマッピングが下記で確認できます。

# cat /proc/net/vlan/eth1.10
eth1.10 VID: 10 REORDER_HDR: 1 dev->priv_flags: 1
total frames received 0
total bytes received 0
Broadcast/Multicast Rcvd 0

total frames transmitted 8
total bytes transmitted 648
Device: bond0
INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
EGRESS priority mappings: 0:1 1:1 2:1 3:1 4:1 5:1 6:1 7:1

# cat /proc/net/vlan/eth1.20
eth1.20 VID: 20 REORDER_HDR: 1 dev->priv_flags: 1
total frames received 0
total bytes received 0
Broadcast/Multicast Rcvd 0

total frames transmitted 8
total bytes transmitted 648
Device: bond0
INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
EGRESS priority mappings: 0:3 1:3 2:3 3:3 4:3 5:3 6:3 7:3

# cat /proc/net/vlan/eth1.30
eth1.30 VID: 30 REORDER_HDR: 1 dev->priv_flags: 1
total frames received 0
total bytes received 0
Broadcast/Multicast Rcvd 0

total frames transmitted 8
total bytes transmitted 648
Device: bond0
INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
EGRESS priority mappings: 0:5 1:5 2:5 3:5 4:5 5:5 6:5 7:5

4.NICにQoS設定（Mellanox NICの場合の例）

// user priority 0 => TC(Traffic Class) 0
// user priority 1 => TC(Traffic Class) 0
// user priority 2 => TC(Traffic Class) 1
// user priority 3 => TC(Traffic Class) 1
// user priority 4 => TC(Traffic Class) 2
// user priority 5 => TC(Traffic Class) 2
// user priority 6 => TC(Traffic Class) 3
// user priority 7 => TC(Traffic Class) 3
// Transmission algorithm = strict for TC0-3
// TC minimum bandwidth = 0% for TC0-3 (strict)
// Rate limit : 1Gbps:TC0, 5Gbps:TC1

というように設定するコマンドが下記です。
# mlnx_qos -i eth1 --prio_tc=0,0,1,1,2,2,3,3 --tsa=strict,strict,strict,strict --tcbw=0,0,0,0 --ratelimit=1,5

 

5.性能検証

実際にQoSが効いていることを確認します（本例ではiperfで測定を実施）。

VLAN=10（帯域制限=1Gbps）

# iperf -c 12.0.10.1 -P 16
------------------------------------------------------------
Client connecting to 12.0.10.1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 18] local 12.0.10.2 port 51263 connected with 12.0.10.1 port 5001
[ 5] local 12.0.10.2 port 51250 connected with 12.0.10.1 port 5001
[ 4] local 12.0.10.2 port 51249 connected with 12.0.10.1 port 5001
[ 3] local 12.0.10.2 port 51248 connected with 12.0.10.1 port 5001
[ 6] local 12.0.10.2 port 51251 connected with 12.0.10.1 port 5001
[ 7] local 12.0.10.2 port 51252 connected with 12.0.10.1 port 5001
[ 8] local 12.0.10.2 port 51253 connected with 12.0.10.1 port 5001
[ 9] local 12.0.10.2 port 51254 connected with 12.0.10.1 port 5001
[ 10] local 12.0.10.2 port 51255 connected with 12.0.10.1 port 5001
[ 11] local 12.0.10.2 port 51256 connected with 12.0.10.1 port 5001
[ 13] local 12.0.10.2 port 51257 connected with 12.0.10.1 port 5001
[ 12] local 12.0.10.2 port 51258 connected with 12.0.10.1 port 5001
[ 14] local 12.0.10.2 port 51259 connected with 12.0.10.1 port 5001
[ 15] local 12.0.10.2 port 51260 connected with 12.0.10.1 port 5001
[ 16] local 12.0.10.2 port 51261 connected with 12.0.10.1 port 5001
[ 17] local 12.0.10.2 port 51262 connected with 12.0.10.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 153 MBytes 128 Mbits/sec
[ 5] 0.0-10.0 sec 146 MBytes 123 Mbits/sec
[ 7] 0.0-10.0 sec 10.6 MBytes 8.90 Mbits/sec
[ 17] 0.0-10.0 sec 139 MBytes 117 Mbits/sec
[ 8] 0.0-10.0 sec 154 MBytes 128 Mbits/sec
[ 14] 0.0-10.0 sec 146 MBytes 123 Mbits/sec
[ 13] 0.0-10.0 sec 139 MBytes 116 Mbits/sec
[ 15] 0.0-10.0 sec 150 MBytes 126 Mbits/sec
[ 10] 0.0-10.0 sec 144 MBytes 121 Mbits/sec
[ 9] 0.0-10.1 sec 11.2 MBytes 9.37 Mbits/sec
[ 6] 0.0-10.1 sec 10.4 MBytes 8.62 Mbits/sec
[ 18] 0.0-10.1 sec 10.5 MBytes 8.71 Mbits/sec
[ 11] 0.0-10.1 sec 11.0 MBytes 9.12 Mbits/sec
[ 12] 0.0-10.1 sec 10.4 MBytes 8.61 Mbits/sec
[ 4] 0.0-10.1 sec 10.5 MBytes 8.70 Mbits/sec
[ 16] 0.0-10.1 sec 11.2 MBytes 9.32 Mbits/sec
[SUM] 0.0-10.1 sec 1.23 GBytes 1.04 Gbits/sec

VLAN=20 (帯域制限=5Gbps）

# iperf -c 12.0.20.1 -P 16
------------------------------------------------------------
Client connecting to 12.0.20.1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 18] local 12.0.20.2 port 36567 connected with 12.0.20.1 port 5001
[ 4] local 12.0.20.2 port 36553 connected with 12.0.20.1 port 5001
[ 5] local 12.0.20.2 port 36554 connected with 12.0.20.1 port 5001
[ 6] local 12.0.20.2 port 36555 connected with 12.0.20.1 port 5001
[ 3] local 12.0.20.2 port 36552 connected with 12.0.20.1 port 5001
[ 7] local 12.0.20.2 port 36556 connected with 12.0.20.1 port 5001
[ 8] local 12.0.20.2 port 36557 connected with 12.0.20.1 port 5001
[ 10] local 12.0.20.2 port 36559 connected with 12.0.20.1 port 5001
[ 9] local 12.0.20.2 port 36558 connected with 12.0.20.1 port 5001
[ 11] local 12.0.20.2 port 36560 connected with 12.0.20.1 port 5001
[ 12] local 12.0.20.2 port 36561 connected with 12.0.20.1 port 5001
[ 14] local 12.0.20.2 port 36562 connected with 12.0.20.1 port 5001
[ 13] local 12.0.20.2 port 36563 connected with 12.0.20.1 port 5001
[ 16] local 12.0.20.2 port 36564 connected with 12.0.20.1 port 5001
[ 15] local 12.0.20.2 port 36565 connected with 12.0.20.1 port 5001
[ 17] local 12.0.20.2 port 36566 connected with 12.0.20.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 7] 0.0-10.0 sec 735 MBytes 617 Mbits/sec
[ 11] 0.0-10.0 sec 694 MBytes 583 Mbits/sec
[ 14] 0.0-10.0 sec 627 MBytes 526 Mbits/sec
[ 18] 0.0-10.0 sec 697 MBytes 584 Mbits/sec
[ 4] 0.0-10.0 sec 572 MBytes 479 Mbits/sec
[ 6] 0.0-10.0 sec 735 MBytes 616 Mbits/sec
[ 3] 0.0-10.0 sec 78.0 MBytes 65.4 Mbits/sec
[ 9] 0.0-10.0 sec 572 MBytes 479 Mbits/sec
[ 12] 0.0-10.0 sec 77.8 MBytes 65.2 Mbits/sec
[ 16] 0.0-10.0 sec 78.1 MBytes 65.5 Mbits/sec
[ 15] 0.0-10.0 sec 749 MBytes 627 Mbits/sec
[ 5] 0.0-10.0 sec 80.1 MBytes 67.1 Mbits/sec
[ 8] 0.0-10.0 sec 78.4 MBytes 65.6 Mbits/sec
[ 10] 0.0-10.0 sec 78.1 MBytes 65.3 Mbits/sec
[ 13] 0.0-10.0 sec 78.4 MBytes 65.5 Mbits/sec
[ 17] 0.0-10.0 sec 78.5 MBytes 65.6 Mbits/sec
[SUM] 0.0-10.0 sec 5.87 GBytes 5.02 Gbits/sec

VLAN=30（帯域制限なし）

# iperf -c 12.0.30.1 -P 16
------------------------------------------------------------
Client connecting to 12.0.30.1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 12.0.30.2 port 41096 connected with 12.0.30.1 port 5001
[ 4] local 12.0.30.2 port 41097 connected with 12.0.30.1 port 5001
[ 5] local 12.0.30.2 port 41098 connected with 12.0.30.1 port 5001
[ 18] local 12.0.30.2 port 41110 connected with 12.0.30.1 port 5001
[ 6] local 12.0.30.2 port 41099 connected with 12.0.30.1 port 5001
[ 7] local 12.0.30.2 port 41100 connected with 12.0.30.1 port 5001
[ 9] local 12.0.30.2 port 41102 connected with 12.0.30.1 port 5001
[ 8] local 12.0.30.2 port 41101 connected with 12.0.30.1 port 5001
[ 10] local 12.0.30.2 port 41103 connected with 12.0.30.1 port 5001
[ 11] local 12.0.30.2 port 41104 connected with 12.0.30.1 port 5001
[ 12] local 12.0.30.2 port 41105 connected with 12.0.30.1 port 5001
[ 13] local 12.0.30.2 port 41106 connected with 12.0.30.1 port 5001
[ 14] local 12.0.30.2 port 41107 connected with 12.0.30.1 port 5001
[ 15] local 12.0.30.2 port 41108 connected with 12.0.30.1 port 5001
[ 16] local 12.0.30.2 port 41109 connected with 12.0.30.1 port 5001
[ 17] local 12.0.30.2 port 41111 connected with 12.0.30.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 3.80 GBytes 3.26 Gbits/sec
[ 4] 0.0-10.0 sec 1.97 GBytes 1.69 Gbits/sec
[ 5] 0.0-10.0 sec 3.08 GBytes 2.65 Gbits/sec
[ 6] 0.0-10.0 sec 1.95 GBytes 1.68 Gbits/sec
[ 7] 0.0-10.0 sec 1.72 GBytes 1.48 Gbits/sec
[ 12] 0.0-10.0 sec 3.54 GBytes 3.04 Gbits/sec
[ 13] 0.0-10.0 sec 2.09 GBytes 1.80 Gbits/sec
[ 14] 0.0-10.0 sec 3.70 GBytes 3.18 Gbits/sec
[ 18] 0.0-10.0 sec 3.57 GBytes 3.07 Gbits/sec
[ 9] 0.0-10.0 sec 1.86 GBytes 1.59 Gbits/sec
[ 8] 0.0-10.0 sec 4.06 GBytes 3.49 Gbits/sec
[ 10] 0.0-10.0 sec 3.58 GBytes 3.07 Gbits/sec
[ 11] 0.0-10.0 sec 1.75 GBytes 1.51 Gbits/sec
[ 15] 0.0-10.0 sec 3.42 GBytes 2.94 Gbits/sec
[ 16] 0.0-10.0 sec 2.05 GBytes 1.76 Gbits/sec
[ 17] 0.0-10.0 sec 1.87 GBytes 1.60 Gbits/sec
[SUM] 0.0-10.0 sec 44.0 GBytes 37.8 Gbits/sec

 6. 考察

性能測定の結果、各VLANトラフィックが設定通りの帯域制限で動作していることが確認できました。

各VLANにて同時にトラフィックを流した場合は、単独で流した場合よりは変動幅はあるものの、運用上問題ないレベルで性能確認ができています。

本設定はサーバ再起動すると残らないので、起動時に実行されるスクリプトで実行するよう設定することで運用することができます。

7.補足 (bonding構成の場合)

補足として、bonding構成の場合についてです。

例として、eth1とeth2をbondingし、bond0とした場合ですが、

  • 「3.送信Socket priorityをUser priority(UP)に割り当て」まではeth1の代わりにbond0を指定
  • 「4.NICにQoS設定（Mellanox NICの場合の例）」では、eth1とeth2の両方でコマンドを実行（下記）

# mlnx_qos -i eth1 --prio_tc=0,0,1,1,2,2,3,3 --tsa=strict,strict,strict,strict --tcbw=0,0,0,0 --ratelimit=1,5

# mlnx_qos -i eth2 --prio_tc=0,0,1,1,2,2,3,3 --tsa=strict,strict,strict,strict --tcbw=0,0,0,0 --ratelimit=1,5

でＯＫです。

 

10GbE２本、冗長含めて計４本を１つのサーバに接続して運用されている仮想化環境やクラウド環境は多いと思いますが、今回紹介した方法を用いれば、Dual portの40GbEカード１枚に統合し、しかも帯域制限でVLANトラフィック毎に帯域を分けることができます。（NIC送信帯域を制限する方法なので、スイッチの設定は必要ありません）