需求说明

• 提供路由服务，并透明代理
• 提供 VPN 服务
• 提供影视库功能
• 提供照片备份功能
• 迅雷下载电影

心路历程

最开始纠结是 PVE 还是直接跑 Linux ，但是想了一下都 all in one 了干脆直接 Linux 省的套娃。

最后整体思路：

• 创建 gateway 的 netns 作为软路由，里面跑 dnmasq 提供 dhcp 。配合 iptables 和 ipset 实现全局流量走 clash 透明代理。源码
• 使用自己实现的 wg 工具来配置 wireguard 提供 VPN 服务。源码
• 影音库 Jellyfin 运行在 lxc 容器中，透传核显提供 GPU 解码。
• immich 提供照片备份功能，通过 nerdctl 跑 immich 的 docker compose 服务。
• 按照迅雷网盘版提供磁力下载功能。源码
• nginx 提供 https 反向代理。
• 服务管理，基本通过 systemd

硬件说明

小黄鱼淘了一台 4 网口的 AMD 主板,16G 内存，CPU 型号忘了，看了一下是 es 的，无所谓了

root@shawn-aio:~# grep name /proc/cpuinfo | cut -f2 -d: | uniq -c
      4  AMD Eng Sample
root@shawn-aio:~# free -h
               total        used        free      shared  buff/cache   available
Mem:            14Gi       1.9Gi       9.8Gi       200Mi       3.4Gi        12Gi
Swap:          4.0Gi          0B       4.0Gi

网络结构

eno1 做 wan 口，通过 iptables nat 表做 masquerade 给内部提供外网访问能力。wan 口只开放 vpn 端口。

iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE

root@shawn-aio:~# iptables -S INPUT
-P INPUT DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -i br-mgmt -j ACCEPT
-A INPUT -i eno1 -p udp -m udp --dport 51820 -j ACCEPT
-A INPUT -i wg0 -j ACCEPT
-A INPUT -i lxcbr0 -j ACCEPT
-A INPUT -i br-0ad574e582c7 -j ACCEPT
-A INPUT -i nerdctl0 -j ACCEPT

创建 br-mgmt ，接入 enp3s0 ，enp4s0 ，enp5s0 做 lan 口，同时通过 veth 将 netns gateway 接到 br-mgmt ，并在 netns 中提供 dhcp ，下发默认路由和 dns 地址。

root@shawn-aio:~# ip a show br-mgmt
6: br-mgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ea:a4:3d:6c:ce:d0 brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.254/24 brd 172.20.0.255 scope global br-mgmt
       valid_lft forever preferred_lft forever
    inet6 fe80::e8a4:3dff:fe6c:ced0/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
root@shawn-aio:~# ip netns exec gateway ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host proto kernel_lo
       valid_lft forever preferred_lft forever
12: gTb@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:ef:a4:d0:0c:e9 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.20.0.1/24 scope global gTb
       valid_lft forever preferred_lft forever
    inet6 fe80::50ef:a4ff:fed0:ce9/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
root@shawn-aio:~# ip l show master br-mgmt
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-mgmt state UP mode DEFAULT group default qlen 1000
    link/ether 20:76:93:5b:14:4f brd ff:ff:ff:ff:ff:ff
4: enp4s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master br-mgmt state DOWN mode DEFAULT group default qlen 1000
    link/ether 20:76:93:5b:14:50 brd ff:ff:ff:ff:ff:ff
5: enp5s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master br-mgmt state DOWN mode DEFAULT group default qlen 1000
    link/ether 20:76:93:5b:14:51 brd ff:ff:ff:ff:ff:ff
13: bTg@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-mgmt state UP mode DEFAULT group default qlen 1000
    link/ether aa:36:50:e4:db:81 brd ff:ff:ff:ff:ff:ff link-netns gateway
15: veth18XBnR@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-mgmt state UP mode DEFAULT group default qlen 1000
    link/ether fe:e5:47:3f:90:3b brd ff:ff:ff:ff:ff:ff link-netnsid 1
16: vethdxnWc4@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-mgmt state UP mode DEFAULT group default qlen 1000
    link/ether fe:c1:4c:94:06:ba brd ff:ff:ff:ff:ff:ff link-netnsid 2

服务详情

nginx 反向代理

root@shawn-aio:~# systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled)
     Active: active (running) since Fri 2025-05-09 08:38:57 CST; 1 week 0 days ago
 Invocation: a6039cd62e984eb0bd649511a52f5060
       Docs: man:nginx(8)
   Main PID: 6945 (nginx)
      Tasks: 5 (limit: 17733)
     Memory: 8.6M (peak: 13.6M)
        CPU: 1.111s
     CGroup: /system.slice/nginx.service
             ├─6945 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
             ├─6946 "nginx: worker process"
             ├─6947 "nginx: worker process"
             ├─6948 "nginx: worker process"
             └─6949 "nginx: worker process"

May 09 08:38:57 shawn-aio systemd[1]: Starting nginx.service - A high performance web server and a reverse proxy server...
May 09 08:38:57 shawn-aio systemd[1]: Started nginx.service - A high performance web server and a reverse proxy server.

软路由和透明代理

自己写了个脚本，创建了一个 netns 后，在 netns 中跑 dnsmasq 提供 dhcp 服务。里面跑 clash ，并设置 iptables 规则来实现透明代理。

root@shawn-aio:~# systemctl status gateway
● gateway.service - Gateway service enable tproxy
     Loaded: loaded (/etc/systemd/system/gateway.service; enabled; preset: enabled)
     Active: active (running) since Fri 2025-05-09 08:38:57 CST; 1 week 0 days ago
 Invocation: 76718511daa5473bad8931fedd4d54c9
   Main PID: 6873 (gateway.sh)
      Tasks: 3 (limit: 17733)
     Memory: 5.1M (peak: 6.8M)
        CPU: 1d 22h 47min 56.956s
     CGroup: /system.slice/gateway.service
             ├─6873 /usr/bin/bash /opt/gateway/gateway.sh start
             ├─6973 /usr/sbin/dnsmasq -C /opt/gateway/dnsmasq.conf
             └─6974 bash /opt/gateway/tproxy.sh start

May 15 08:25:45 shawn-aio dnsmasq-dhcp[6973]: DHCPREQUEST(gTb) 172.20.0.165 c8:a3:62:57:df:69
May 15 08:25:45 shawn-aio dnsmasq-dhcp[6973]: DHCPACK(gTb) 172.20.0.165 c8:a3:62:57:df:69 luchengdeMBP
May 15 13:53:13 shawn-aio dnsmasq-dhcp[6973]: DHCPREQUEST(gTb) 172.20.0.165 c8:a3:62:57:df:69
May 15 13:53:13 shawn-aio dnsmasq-dhcp[6973]: DHCPACK(gTb) 172.20.0.165 c8:a3:62:57:df:69 luchengdeMBP
May 15 19:10:29 shawn-aio dnsmasq-dhcp[6973]: DHCPREQUEST(gTb) 172.20.0.165 c8:a3:62:57:df:69
May 15 19:10:29 shawn-aio dnsmasq-dhcp[6973]: DHCPACK(gTb) 172.20.0.165 c8:a3:62:57:df:69 luchengdeMBP
May 16 00:33:54 shawn-aio dnsmasq-dhcp[6973]: DHCPREQUEST(gTb) 172.20.0.165 c8:a3:62:57:df:69
May 16 00:33:54 shawn-aio dnsmasq-dhcp[6973]: DHCPACK(gTb) 172.20.0.165 c8:a3:62:57:df:69 luchengdeMBP
May 16 06:08:01 shawn-aio dnsmasq-dhcp[6973]: DHCPREQUEST(gTb) 172.20.0.165 c8:a3:62:57:df:69
May 16 06:08:01 shawn-aio dnsmasq-dhcp[6973]: DHCPACK(gTb) 172.20.0.165 c8:a3:62:57:df:69 luchengdeMBP
root@shawn-aio:~# ip netns
gateway (id: 0)

VPN 服务

使用场景很简单，感觉没必要用 tailscale 或者 netmaker 。同时自己不太喜欢界面，所以自己模仿 wg-easy 写了个 wg 服务来提供 VPN 服务。客户端直接鉴权是模仿 k8s apiserver 和 kubectl 通过 tls 证书鉴权。

root@shawn-aio:~# wgctl subnet list
UUID                                  Name     Address       Public Key
e4212eda-233f-11f0-8903-2076935b144e  default  10.67.0.1/24  <pub key>
root@shawn-aio:~# wgctl peer list -s e4212eda-233f-11f0-8903-2076935b144e
UUID                                  User     Address       Public Key                                    Enable
f9615f20-233f-11f0-8903-2076935b144e  lucheng  10.67.0.2/24  <pub key>  true
root@shawn-aio:~# wg
interface: wg0
  public key: <server pubkey>
  private key: (hidden)
  listening port: 51820

peer: <peer pubkey>
  endpoint: 10.28.66.18:36303
  allowed ips: 10.67.0.2/32
  latest handshake: 29 minutes, 23 seconds ago
  transfer: 5.14 MiB received, 35.12 MiB sent

影音功能

lxc 跑 ubuntu 在里面装 jellyfin

root@shawn-aio:~# lxc-ls -f
NAME     STATE   AUTOSTART GROUPS IPV4       IPV6 UNPRIVILEGED
jellyfin RUNNING 1         -      172.20.0.2 -    false
ubuntu   RUNNING 1         -      172.20.0.3 -    false

照片备份

root@shawn-aio:~# nerdctl ps
CONTAINER ID    IMAGE                                                COMMAND                   CREATED       STATUS    PORTS                     NAMES
05a100ce77d8    quay.io/shawnlu0127/immich/immich-server:v1.131.1    "tini -- /bin/bash s…"    6 days ago    Up        0.0.0.0:2283->2283/tcp    immich_server
414297ba5e5f    quay.io/shawnlu0127/immich/postgres:pg14-v0.2.0      "docker-entrypoint.s…"    6 days ago    Up                                  immich_postgres
6fa52ce3d9e1    quay.io/shawnlu0127/immich/redis:6.2-alpine          "docker-entrypoint.s…"    6 days ago    Up                                  immich_redis