测试openvpn做二层桥接的性能.
作者：互联网
测试一下openvpn做二层桥接时. 能达到什么样的性能.
操作系统: debian10 内核 4.19.0-16-amd64
cpu:Xeon E5640 @ 2.67GHz2 每台服务器
通信物理网卡: BCM5709 Gigabit Ethernet3 bond rr模式 默认驱动.
eno2,eno3,eno4连了三条电口网线. 两台服务器直连.
bond网卡是链路聚合网卡.用的都是 rr方式.
tap网卡是openvpn用的二层网卡.
veth0和client中的eth0是虚拟的网卡对.
网络连接.
(ClientA 2.0.0.1/24)----(LinuxA 1.0.0.254/24) — (1.0.0.253/24 LinuxB ) —(2.0.0.2/24 ClientB)
LinuxA与LinuxB 之间连接一个二层openvpn
ClientA和ClinentB都是用net namespace建立的测试客户端. 生成虚拟网卡对与LinuxA和LinuxB通信
LinuxA与LinuxB 启用一个桥接口. 把openvpn产生的tap接口 和 虚拟网卡对加入 桥接接口.
最后实现的是. ClientA 与 ClientB的虚拟网卡对在同一个广播域内. 测试通信
=====================
LinuxA
ip link add br6 type bridge
ip addr flush dev br6
ip link set br6 up
openvpn --remote 1.0.0.253 --port 1051 --dev tap01 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon
ip link set tap01 down
ip addr flush dev tap01
ip link set tap01 master br6
ip link set tap01 up
##################
ip netns add ca
ip netns exec ca /usr/bin/screen -d -m -t ca
ip link add veth0 type veth peer name B
ip link set B netns ca
ip netns exec ca ip link set B name eth0
#lsns
4026532348 net 4 19660 root /usr/bin/SCREEN -d -m
4026532404 mnt 2 19660 root /usr/bin/SCREEN -d -m
#进 screen的pid
nsenter -n -t 19660
#这命令第一行也能取到?
#ip netns pids ca
=============
ip link set veth0 down
ip addr flush dev veth0
ip link set veth0 master br6
ip link set veth0 up
================
ClientA
ip addr add 2.0.0.1/24 dev eth0
ip link set eth0 up
###########################################
LinuxB
ip link add br6 type bridge
ip addr flush dev br6
ip link set br6 up
openvpn --remote 1.0.0.254 --port 1051 --dev tap01 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon
ip link set tap01 down
ip addr flush dev tap01
ip link set tap01 master br6
ip link set tap01 up
ip netns add ca
ip netns exec ca /usr/bin/screen -d -m -t ca
ip link add veth0 type veth peer name B
ip link set B netns ca
ip netns exec ca ip link set B name eth0
#这命令第一行也能取到?
#ip netns pids ca
=============
ip link set veth0 down
ip addr flush dev veth0
ip link set veth0 master br6
ip link set veth0 up
ClientB
ip addr add 2.0.0.2/24 dev eth0
ip link set eth0 up
##########################
测试效果.
cat /dev/zero |nc -l -p 2587 -q1
nc 2.0.0.2 2587 > /dev/null
单个进程.
流量
tap01: 11.95 Mb/s In 463.44 Mb/s Out - 21788.9 p/s In 40453.8 p/s Out
openvpn进程cpu占用
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21256 root 10 -10 11700 4944 4196 S 66.1 0.0 11:23.74 openvpn
##################
下面用多条openvpn来进行测试一下.
LinuxA
openvpn --remote 1.0.0.253 --port 1051 --dev tap01 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon
openvpn --remote 1.0.0.253 --port 1052 --dev tap02 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon
openvpn --remote 1.0.0.253 --port 1053 --dev tap03 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon
openvpn --remote 1.0.0.253 --port 1054 --dev tap04 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon
openvpn --remote 1.0.0.253 --port 1055 --dev tap05 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon
linuxB
openvpn --remote 1.0.0.254 --port 1051 --dev tap01 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon
openvpn --remote 1.0.0.254 --port 1052 --dev tap02 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon
openvpn --remote 1.0.0.254 --port 1053 --dev tap03 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon
openvpn --remote 1.0.0.254 --port 1054 --dev tap04 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon
openvpn --remote 1.0.0.254 --port 1055 --dev tap05 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon
都执行
ip link add bond1 type bond
for i in 1 2 3 4 5
do
dev=“tap0${i}”
ip link set $dev down
ip addr flush dev $dev
ip link set $dev master bond1
ip link set $dev up
done
ip link set bond1 down
ip addr flush dev bond1
ip link set bond1 master br6
ip link set bond1 up
##################
5个进程同时测试
veth0: 823.24 Mb/s In 39.70 Mb/s Out - 33410.3 p/s In 63845.4 p/s Out
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22505 root 10 -10 11700 4828 4080 R 60.5 0.0 2:34.19 openvpn
22325 root 10 -10 11700 4804 4056 S 59.5 0.0 2:35.90 openvpn
22497 root 10 -10 11700 4972 4224 R 58.5 0.0 2:35.04 openvpn
22491 root 10 -10 11700 4916 4168 R 56.8 0.0 2:32.48 openvpn
22484 root 10 -10 11700 4944 4196 S 56.5 0.0 2:34.34 openvpn
================
在LinuxA - B 两台之间测试 5进程
eno2: 286.49 Mb/s In 20.77 Mb/s Out - 24233.3 p/s In 20907.4 p/s Out
eno3: 286.26 Mb/s In 20.76 Mb/s Out - 24212.5 p/s In 20892.9 p/s Out
eno4: 286.42 Mb/s In 20.78 Mb/s Out - 24225.4 p/s In 20922.0 p/s Out
tap01: 167.42 Mb/s In 7.85 Mb/s Out - 14615.6 p/s In 12567.3 p/s Out
tap02: 166.68 Mb/s In 7.86 Mb/s Out - 14551.0 p/s In 12572.5 p/s Out
tap03: 166.28 Mb/s In 7.85 Mb/s Out - 14516.5 p/s In 12567.3 p/s Out
tap04: 166.81 Mb/s In 7.84 Mb/s Out - 14563.2 p/s In 12547.7 p/s Out
tap05: 165.72 Mb/s In 7.86 Mb/s Out - 14467.4 p/s In 12572.8 p/s Out
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22325 root 10 -10 11700 4804 4056 R 67.7 0.0 4:51.64 openvpn
22505 root 10 -10 11700 4828 4080 S 60.0 0.0 4:45.39 openvpn
22497 root 10 -10 11700 4972 4224 R 59.7 0.0 4:49.64 openvpn
22484 root 10 -10 11700 4944 4196 S 59.3 0.0 4:46.80 openvpn
22491 root 10 -10 11700 4916 4168 R 59.3 0.0 4:45.10 openvpn
性能不是差在网卡对上.
linuxB-ClientB
之间测试
veth0: 73.00 Mb/s In 20389.52 Mb/s Out - 138248.5 p/s In 162817.3 p/s Out
性能也不是差在物理网卡上. 尽管 broadcom的网卡在linux中经常性能有问题
eno2: 939.99 Mb/s In 5.33 Mb/s Out - 77403.6 p/s In 8268.9 p/s Out
eno3: 941.26 Mb/s In 5.32 Mb/s Out - 77509.0 p/s In 8244.0 p/s Out
eno4: 939.69 Mb/s In 5.34 Mb/s Out - 77379.7 p/s In 8291.1 p/s Out
还是差在openvpn通信上吧.
==========================
找几个openvpn通信的参数试一下.
–fast-io 没看出变化, 也许能每接口差10Mbit/s?
–txqueuelen 1000 加上没用.
–sndbuf 0 --rcvbuf 0 加上就慢了.
LinuxA
openvpn --remote 1.0.0.253 --port 1051 --dev tap01 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.253 --port 1052 --dev tap02 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.253 --port 1053 --dev tap03 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.253 --port 1054 --dev tap04 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.253 --port 1055 --dev tap05 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.253 --port 1056 --dev tap06 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.253 --port 1057 --dev tap07 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.253 --port 1058 --dev tap08 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.253 --port 1059 --dev tap09 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.253 --port 1060 --dev tap010 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
linuxB
openvpn --remote 1.0.0.254 --port 1051 --dev tap01 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.254 --port 1052 --dev tap02 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.254 --port 1053 --dev tap03 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.254 --port 1054 --dev tap04 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.254 --port 1055 --dev tap05 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.254 --port 1056 --dev tap06 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.254 --port 1057 --dev tap07 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.254 --port 1058 --dev tap08 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.254 --port 1059 --dev tap09 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
openvpn --remote 1.0.0.254 --port 1060 --dev tap010 --ping 9 --link-mtu 1500 --nice -10 --auth none --cipher none --daemon --fast-io
for i in 1 2 3 4 5 6 7 8 9 10
do
dev=“tap0${i}”
ip link set $dev down
ip addr flush dev $dev
ip link set $dev master bond1
ip link set $dev up
done
eno2: 390.03 Mb/s In 27.38 Mb/s Out - 32990.0 p/s In 27524.1 p/s Out
eno3: 389.72 Mb/s In 27.38 Mb/s Out - 32963.6 p/s In 27536.6 p/s Out
eno4: 389.69 Mb/s In 27.35 Mb/s Out - 32960.9 p/s In 27503.7 p/s Out
tap01: 112.79 Mb/s In 5.17 Mb/s Out - 9846.4 p/s In 8254.4 p/s Out
tap010: 113.75 Mb/s In 5.17 Mb/s Out - 9930.4 p/s In 8257.9 p/s Out
tap02: 113.63 Mb/s In 5.18 Mb/s Out - 9919.2 p/s In 8263.6 p/s Out
tap03: 113.57 Mb/s In 5.18 Mb/s Out - 9914.6 p/s In 8262.9 p/s Out
tap04: 113.56 Mb/s In 5.18 Mb/s Out - 9913.7 p/s In 8261.1 p/s Out
tap05: 112.73 Mb/s In 5.18 Mb/s Out - 9841.1 p/s In 8265.7 p/s Out
tap06: 113.87 Mb/s In 5.17 Mb/s Out - 9940.3 p/s In 8260.7 p/s Out
tap07: 113.22 Mb/s In 5.18 Mb/s Out - 9884.6 p/s In 8269.6 p/s Out
tap08: 113.47 Mb/s In 5.18 Mb/s Out - 9905.8 p/s In 8266.3 p/s Out
tap09: 113.05 Mb/s In 5.18 Mb/s Out - 9869.3 p/s In 8266.7 p/s Out
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20732 root 10 -10 11700 2688 1956 S 59.1 0.0 0:56.17 openvpn
20759 root 10 -10 11700 2800 2068 R 58.5 0.0 0:55.34 openvpn
20777 root 10 -10 11700 2844 2112 S 58.5 0.0 0:56.55 openvpn
20747 root 10 -10 11700 2660 1928 S 57.8 0.0 0:55.88 openvpn
20771 root 10 -10 11700 2760 2028 R 57.8 0.0 0:56.31 openvpn
20739 root 10 -10 11700 2728 1996 R 57.5 0.0 0:55.65 openvpn
20753 root 10 -10 11700 2756 2024 S 57.5 0.0 0:55.54 openvpn
20783 root 10 -10 11700 2764 2028 S 57.1 0.0 0:55.99 openvpn
20765 root 10 -10 11700 2660 1932 R 56.8 0.0 0:56.05 openvpn
20725 root 10 -10 11700 2792 2064 R 56.1 0.0 0:55.58 openvpn
简单增加vpn的条数. 性能也不能线性提高.cpu的idle 已经 50%了.
cpu5已经跑满.这个cpu上的 si很高.
手动改了一下中断分配. si的分担好了一点儿. 不过对于通信量没啥效果.
调也没用就对了. 应该还是差在openvpn上. 也许是内核到用户进程之间的问题?
反正用内核管理的bond,网卡对之类的. 性能没问题.
eno2: 442.11 Mb/s In 30.05 Mb/s Out - 37393.9 p/s In 30134.2 p/s Out
eno3: 441.51 Mb/s In 30.01 Mb/s Out - 37344.0 p/s In 30091.8 p/s Out
eno4: 442.08 Mb/s In 30.04 Mb/s Out - 37390.8 p/s In 30122.0 p/s Out
tap01: 128.07 Mb/s In 5.68 Mb/s Out - 11180.1 p/s In 9038.5 p/s Out
tap010: 128.26 Mb/s In 5.68 Mb/s Out - 11197.2 p/s In 9034.8 p/s Out
tap02: 127.84 Mb/s In 5.69 Mb/s Out - 11160.8 p/s In 9040.6 p/s Out
tap03: 128.09 Mb/s In 5.69 Mb/s Out - 11182.5 p/s In 9040.2 p/s Out
tap04: 129.35 Mb/s In 5.69 Mb/s Out - 11291.9 p/s In 9038.9 p/s Out
tap05: 128.76 Mb/s In 5.68 Mb/s Out - 11240.0 p/s In 9037.2 p/s Out
tap06: 127.92 Mb/s In 5.69 Mb/s Out - 11166.3 p/s In 9037.3 p/s Out
tap07: 129.28 Mb/s In 5.69 Mb/s Out - 11286.6 p/s In 9046.0 p/s Out
tap08: 128.60 Mb/s In 5.68 Mb/s Out - 11225.9 p/s In 9035.3 p/s Out
tap09: 128.13 Mb/s In 5.68 Mb/s Out - 11185.6 p/s In 9034.6 p/s Out
============
以下没用.
cat /dev/zero | nc -l -p 2001 -q1 &
cat /dev/zero | nc -l -p 2002 -q1 &
cat /dev/zero | nc -l -p 2003 -q1 &
cat /dev/zero | nc -l -p 2004 -q1 &
cat /dev/zero | nc -l -p 2005 -q1 &
nohup nc 2.0.0.253 2001 > /dev/null &
nohup nc 2.0.0.253 2002 > /dev/null &
nohup nc 2.0.0.253 2003 > /dev/null &
nohup nc 2.0.0.253 2004 > /dev/null &
nohup nc 2.0.0.253 2005 > /dev/null &
kill -9 $(pidof nc)
team网卡聚合没调明白. 以后有机会再看.
标签：10,二层,Mb,桥接,dev,--,link,openvpn,Out 来源： https://blog.csdn.net/zhanzc1/article/details/117574886