docker容器运行Openwrt以访问k8s.gcr.io
作者：互联网
目录
docker容器运行Openwrt以访问k8s.gcr.io
为什么有这个需求呢？原因是在个人电脑的虚拟机上部署K8S集群拉去一些镜像比较麻烦，虽然可以用阿里的镜像仓库来解决一些问题, 但是有的镜像还是得自己找资源下载就很不方便，比如velero，所以才有了这样的想法。Openwrt在这里充当了一个网关的角色, 通过它我们下载外面网络的镜像。
1.网络情况描述
这里用的虚拟机为VMware，虚拟机网络用的NAT网络，详情如下：
| 虚拟机NAT网段 | 172.16.0.0/16 |
|---|---|
| NAT网关地址 | 172.16.0.2 |
| openwrt宿主机地址 | 172.16.6.234 |
| openwrt容器地址 | 172.16.6.235 |
| 测试用虚拟机 | 172.16.6.233 |
2.宿主机环境
宿主机系统为Ubuntu 20.04，需要安装docker以运行openwrt，可以采用apt安装或者二进制安装，docker安装完毕可配置镜像加速。详细步骤略。
[root@openwrt ~]# uname -a
Linux openwrt 5.4.0-110-generic #124-Ubuntu SMP Thu Apr 14 19:46:19 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
网卡情况,IP为172.16.6.234的网卡为ens33:
[root@openwrt ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:e4:bb:98 brd ff:ff:ff:ff:ff:ff
inet 172.16.6.234/16 brd 172.16.255.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fee4:bb98/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:de:7e:c5:bf brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
3. 运行Openwrt容器
3.1宿主机开启网卡混杂模式
网卡ens33开启网卡混杂模式
[root@openwrt ~]# ip link set ens33 promisc on
[root@openwrt ~]# echo $?
0
3.2 创建虚拟网络
虚拟网络名称为macnet，驱动为macvlan模式，subnet 为NAT网络的网段，geteway修改为NAT网络网关, 依赖的物理网卡为ens33,注意macvlan网络和宿主机不通(宿主机ping不通openwrt容器)
#创建虚拟网络,注意网卡名别写错
root@openwrt ~]# docker network create -d macvlan --subnet=172.16.0.0/16 --gateway=172.16.0.2 -o parent=ens33 macnet
ab933a53aeeb707319b02908dc175d5103c4cc1fa26bc8590892d6cf3d857bbc
#查看新创建的macvlan网络
[root@openwrt ~]# docker network ls
NETWORK ID NAME DRIVER SCOPE
4a24b0686a57 bridge bridge local
7ed97f38433e host host local
ab933a53aeeb macnet macvlan local
c7f790dd236a none null local
3.3 拉取Openwrt镜像
#拉取镜像
[root@openwrt ~]# docker pull sulinggg/openwrt:x86_64
x86_64: Pulling from sulinggg/openwrt
Digest: sha256:4d6f3503950c2c14b6cee86c3c1d8fb1b931edc4a829d555ff051bcd46eb22c6
Status: Image is up to date for sulinggg/openwrt:x86_64
docker.io/sulinggg/openwrt:x86_64
#保存镜像到本地
[root@openwrt ~]# docker save sulinggg/openwrt:x86_64 > /tmp/openwrt-x86_64.tar
[root@openwrt ~]# ls /tmp/ | grep open
openwrt-x86_64.tar
#导入本地镜像
docker load -i openwrt-x86_64.tar
3.4 运行Openwrt容器
[root@openwrt ~]# docker run --restart always --name openwrt -d --network macnet --privileged sulinggg/openwrt:x86_64 /sbin/init
657b9a0e30ccea3b2184064cf2f91fb7d0e2911523df5bc1681772465237defc
[root@openwrt ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
657b9a0e30cc sulinggg/openwrt:x86_64 "/sbin/init" 36 seconds ago Up 35 seconds openwrt
3.5 进入Openwrt容器配置网络
进入Openwrt容器配置静态IP地址,配置管理页面登录密码
[root@openwrt ~]# docker exec -ti openwrt bash
#网络配置如下
bash-5.1# cat /etc/config/network
config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option packet_steering '1'
config interface 'lan'
option type 'bridge'
option ifname 'eth0'
option proto 'static'
option netmask '255.255.0.0'
option ipaddr '172.16.6.235'
option gateway '172.16.0.2'
option dns '172.16.0.2'
#ip查看,br-lan的IP为172.16.6.235
bash-5.1# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
4: eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP group default
link/ether 02:42:ac:10:00:01 brd ff:ff:ff:ff:ff:ff link-netnsid 0
10: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 02:42:ac:10:00:01 brd ff:ff:ff:ff:ff:ff
inet 172.16.6.235/16 brd 172.16.255.255 scope global br-lan
valid_lft forever preferred_lft forever
#测试网络
bash-5.1# ping baidu.com
PING baidu.com (220.181.38.251): 56 data bytes
64 bytes from 220.181.38.251: seq=0 ttl=128 time=9.275 ms
64 bytes from 220.181.38.251: seq=1 ttl=128 time=9.090 ms
64 bytes from 220.181.38.251: seq=2 ttl=128 time=12.958 ms
^C
--- baidu.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 9.090/10.441/12.958 ms
#配置管理页面登录密码
bash-5.1# passwd root
Changing password for root
New password:
Bad password: too short
Retype password:
passwd: password for root changed by root
4. 测试Openwrt地址
在另一台虚拟机(172.16.6.233)测试Openwrt地址是否ping通
root@test ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:df:29:54 brd ff:ff:ff:ff:ff:ff
inet 172.16.6.233/16 brd 172.16.255.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fedf:2954/64 scope link
valid_lft forever preferred_lft forever
[root@test ~]# ping 172.16.6.235
PING 172.16.6.235 (172.16.6.235) 56(84) bytes of data.
64 bytes from 172.16.6.235: icmp_seq=1 ttl=64 time=0.328 ms
64 bytes from 172.16.6.235: icmp_seq=2 ttl=64 time=1.46 ms
64 bytes from 172.16.6.235: icmp_seq=3 ttl=64 time=1.03 ms
64 bytes from 172.16.6.235: icmp_seq=4 ttl=64 time=0.990 ms
^C
--- 172.16.6.235 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3034ms
rtt min/avg/max/mdev = 0.328/0.950/1.456/0.403 ms
在笔记本电脑浏览器输入Openwrt地址 172.16.6.235 即可访问管理面板, 首先在网络-->接口确认网络正常, dns解析正常, 然后在服务中选一个合适的工具添加节点即可。
5.测试用虚拟机访问k8s.gcr.io
将测试用虚拟机网关地址指向Openwrt地址 172.16.6.235
#测试用虚拟机网卡配置文件
[root@test ~]# cat /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
ethernets:
ens33:
addresses:
- 172.16.6.233/16
gateway4: 172.16.6.235
nameservers:
addresses:
- 223.5.5.5
- 114.114.114.114
version: 2
#curl访问k8s.gcr.io，能正常返回结果
[root@test ~]# curl k8s.gcr.io
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://k8s.gcr.io/">here</A>.
</BODY></HTML>
#直接拉取 pause镜像
[root@test ~]# docker pull k8s.gcr.io/pause:3.1
3.1: Pulling from pause
67ddbfb20a22: Pull complete
Digest: sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea
Status: Downloaded newer image for k8s.gcr.io/pause:3.1
k8s.gcr.io/pause:3.1
6.其他应用场景
有了这个网关，可以将harbor镜像仓库宿主机的网关指向Openwrt，并且建立相关的镜像代理仓库即可通过harbor来拉取镜像。
如果需要将openwrt和harbor运行在同一个宿主机的话需要将宿主机与openwrt所在的macvlan网络打通，可以参考以下这篇文章。
参考链接：https://cloud.tencent.com/developer/article/1907799
因为在部署
OpenWrt时使用到了Docker的macvlan模式 ，通俗来讲就是在树莓派的物理网卡虚拟出来两个虚拟网卡。可以让宿主机和Docker同时接入网络并使用不同的IP，但是在设计该模式的时候为了安全禁止了宿主机与容器的直接通信。解决办法就是在宿主机再建立一个macvlan然后修改路由，使数据经过新建的macvlan传到容器内的macvlan即可。
- 新建接口
添加一个名为
mynet的macvlan接口ip link add mynet link eth0 type macvlan mode bridge复制
- 分配IP并启用
需按照本地网络环境，不能照抄
设置
mynet的IP为192.168.5.248ip addr add 192.168.5.248 dev mynet ip link set mynet up复制
- 添加静态路由
192.168.5.250为 OpenWrt 容器的IPip route add 192.168.5.250 dev mynet复制
- 在宿主机测试一下
root@ubuntu:~# ping 192.168.5.250 -c 3 PING 192.168.5.250 (192.168.5.250) 56(84) bytes of data. 64 bytes from 192.168.5.250: icmp_seq=1 ttl=64 time=0.415 ms 64 bytes from 192.168.5.250: icmp_seq=2 ttl=64 time=0.338 ms 64 bytes from 192.168.5.250: icmp_seq=3 ttl=64 time=0.296 ms --- 192.168.5.250 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2052ms rtt min/avg/max/mdev = 0.296/0.349/0.415/0.049 ms复制
- 写入开机自启脚本
在
/etc/rc.local加入以下内容，并添加「可执行」权限chmod a+x /etc/rc.localip link add mynet link eth0 type macvlan mode bridge ip addr add 192.168.5.248 dev mynet ip link set mynet up ip route add 192.168.5.250 dev mynet
个人网站：https://cxupup.com
标签：00,ff,lft,gcr,64,io,172.16,docker,openwrt 来源： https://www.cnblogs.com/cxupup/p/16299858.html