ieee 802.3ae-2002
_
running self-hosted services requires network connectivity, there is no escape from networking, in my opinion. a good network foundation is key to the stable and smooth operation of any modern infrastructure. in the era of cloud and virtual services, the network level is often invisible to end users as well as people who manage all the underlying network infrastructure.
today, I still believe that networking is an important part of the infrastructure, which is nicely reflected in my hobbies as well. now, I host my own infrastructure (and try to keep it modern) at home, and it's a nice little project that keeps me busy and happy.
currently, my setup looks like this (connection diagram).
in the early stages (it was 2014 if i remember correctly) i would just use a decent WiFi router, back then that was ASUS router with asuswrt-merlin[0] firmware, and connect the server directly to it. then, i have discovered a pihole at around 2015, got an Raspberry Pi with the only possibility to connect via LAN, plus i already had a proliant server, qnap nas and then another LAN device that required LAN connectivity. then i thought, let me get a switch. the next state was getting a switch, and i got a cisco switch (SG200-26) and started to grow my network slowly including the LAN connection to all my workstations since I got very bad experience with WiFi plus back then there was the key reinstallation [KRACK][1] vulnerability with WPA2 that just kept me using physical cabel. everything was in one network and i haven't been using much of the security features of that cisco only to introduce vlans in 2019 when i got my first firewall (UniFi gateway[*]) and then later added even more segmentation with the opensense coming in the game.
*i have replaced unifi stuff very quickly from the moment i got it, primarily because of the lack of flexibility, weird and confusing terminology and the "ecosystem" nature. all the equipment got sold at swapfest @c-base.
image: ieee802.3 and ieee802.3q dataformat comparison, source: internetworking, technische grundladen und anwendungen
the cisco SG200-26 served me well for years, it was a decent little switch, honestly. however, over the time ports started flapping up and down, crc errors popping up like weeds, and the switch just giving up and putting ports into err-disabled mode. the hardware was basically dying, choking on the all the heavy backup traffic that wasn't built to handle gracefully. originally, i got this cisco switch without working power supply, the switch was refusing to start and I have diagnosed that the power supply capacitor was swollen and electrolyte has escaped, it was an easy fix and i was really proud to make it work again. meanwhile, the model was EOL[2] by 2023 already and UI became painfully slow + all the port issues, etc...
*cisco small business sg200 series had an EOS (end of sale) after just 6 years which i find very short for that period and the successor sg350 was targeted for more larger corps rather than small business, everything after that became meraki which essentially became all cloud dependent.
so, clap-clap to cisco serving me for more than 10 years, i couldn't save it again, and honestly i felt like that is time for an upgrade. i built couple of servers, 2 NAS devices, lots of VMs, containers, personal data backups and syncs, cisco was just a bottleneck. plus, i already got Intel NUC with 2.5Gibt NIC, firewall with 2.5Gig interfaces and 3xIntel 10G 2P x520 adapters rescued from servers that were thrown away by the ship company I worked for (yes, shame on you HAL for throwing away all those Dell server carelessly)
image: das netz, source: self, Deutsches Technikmuseum Berlin
then, there was a question of powering some of the PoE capable devices like Netgear AP and temperature sensor device. the thought on having PoE switch gave me the fantasy of removing those extra PoE injectors that were consuming power sockets.
when cisco announced EOL i developed the fear of having something that cannot be patched anymore therefore i put all my network management in the separate vlan without external internet access and restricted inbound traffic. this vlan still exists and all the network management is still there!
first of all, why not cisco again? well, cisco is fully-blown cloud dependent bro now with their meraki series and catalyst was simply expensive. then, cisco is not open source, i mean there are some really honorable mentions, however here SONiC could be a choice but looking at their hardware support list the switches there cost 3K+ EUR, and those are mostly enterprise grade hardware switches and those are not easy to find used. then the consumer oss software could be vyos, opx, openwrt or milk-v but those are still niche products with not that broad hardware support.
i already run oss on my other networking hardware, openwrt on my WAX220 APs and opnsense on my firewall, however moving to a oss switch os seams like a real hustle.
fully open hardware ethernet switches are still uncommon compared to opensource routers or firewalls. open source routers and firewalls became practical much earlier because routing can run efficiently on general purpose consumer CPUs, switching at 10/25/100/400+ Gbps is different, it usually requires highly specialized ASICs with enormous packet processing throughput and very low latency, that is very expensive for home use. switch ASICs are usually proprietary (broadcom, marvell, etc) & ASIC SDKs are often closed. so “open switch” usually means, open NOS, open management plane, open or whitebox hardware, ONIE environment and standardized APIs.
image: openwrt one open source & open hardware router, source: openwrt.org
# the openness of internet infrastructure can be an illusion. while it is possible to use open source os, software, and hardware in local networks, there exists no completely open source path across the entire internet. Even if you're using open source solutions from A to B, it is inevitable that some data packets will traverse networks through proprietary vendors like Cisco or Juniper, making the Internet itself a hybrid of open and closed systems. #
i needed to find a good afordable solution. d-link was a very good option, but expensive. tp-link came out with the offer of 2.5&10Gbit ports and PoE, then the idea of utilizing the 10G intel x520 and upgrade NAS devices came instantly but i was missing 10G ports to support the core. as someone who worked as network engineer in the past i got an idea to get a core switch to support the 10G backbone, here the decision fell on mikrotik CRS305-1G-4S+[very catchy name]. additionally i got designed the network to get another 8 port 2.5Gbit swithc with 10Gbit uplinks for my home office which is in a different room then the main server and network rack, again tp-link was the best option.
what i find very surprising is the fact that when you plan your network upgrade somehow you slip the idea of getting new cables and modules, this is the unexpected cost that honestly caught me unprepared. dealing with 10G network requires better cabling, also SFP+ modules or in the best case fiber-optic to support the bandwidth. luckily, the moment i rescued those Intel 10G NICs is also saved bunch of SFP+ DAC cables which ended up being a perfect fit. the only thing i got putchased was SFP+ BASE-T modules to support connectivity between main access switch and my 8 port switch via 20m CAT6 ethernet cable.
image: transceiver block diagram, source: broadcom, newcom-acorn joint workshop presentation
what i got in the end:
tp-link fans: tp-links arrived the first shocker was the the noise, sg3218xp-m2 was really noisy especially while booting, reminds a lot on pretty much any serious enterprise network switch from cisco, brocade, hp or juniper. even when fully booted the noise was just unareable, especially for me and the fact that this is going to be in the rack in hallway of my flat, no way that this is buzzing there 24/7. i tried not to panic and found the good solution to lower the noise. i've found couple of articles on the internet that mainly use noctua fans and i got a pair of NF-A4x20 FLX, the improvement was drasticall with very low thermal degradation.
core switch fans: one small design flaw is that the crs305-1g-4s+ switch can overheat in high traffic utilization especially when SFP+ base T modules are used, those can generate 90°C easily. I modified the case and added 2(two) 4mm fans to cool down the modules, after the modification system temperature went down from 70°C to approx 38°C.
tp-link switches are locally managed and i do not use neither cloud or locally hosted omada controller, mikrotik is also local only config. all network equipment management has its own vlan with many restrictions like no outbound traffic, except the ntp to local and external server. also inbound is very controlled for https and ssh only from few IPs that i use to manage them from.
i also used the chance to redo my network configuration so this was not a migration really, rather the new configuration with some upgrades and addons.
i have 8 vlans configured in total, all are terminated on opnsense firewall, inter-vlan communication is restricted with firewall rules on the firewall and intra-vlan comms are unrestricted, here i use local server or device firewall when needed. additionally i use security features like stp, storm control, arp & dhcp spoofing. PoE is also configured to deliver not more than needed for the device and save some energy. the management is available only via ssh with ssh key and/or https via nginx-proxy.
core switch is configured with all ports as trunks with port isolation allowing all vlans on all port, this gives me the flexibility to later decide to block certain vlans ports. stp is also active on the core switch on all ports.
MTU size gave me some troubles and doubts originally, I managed to configure [MTU 9000] on all devices I have only to discover that zenarmor and netmap do support up to 1500 bytes max[3] i've decided to keep it like that on all other devices. zenarmor relies on the netmap framework, which is optimized for fixed size 1500 byte buffers and lacks the dynamic memory handling required to safely process the variable, oversized packets of jumbo frames without causing crashes or data corruption. I hope they implement better support for MTU in the future.
i've used a single cable to connect core switch, servers, NAS and the switches. do i care about the redundancy, no i don't. normally in the enterprise setup N+1[4] is a religious doctrine and the way to go, however in my home setup i don't depend on anything that is hosted on servers, and any other failover or redundancy could introduce additional complexity and cost, what for? I actually have N+1 Human Redundancy. I don't need a second switch. I need a spare cable and a backup plan. That is the optimal engineering solution for a solo operator.
"In the home lab, we do not buy redundancy to guarantee eternal uptime, instead we buy resilience to ensure that when the inevitable failure occurs, our data survives and our recovery is swift. True reliability is not the absence of failure, but the presence of a backup plan and the willingness to fix it yourself."
all the tests were performed from minisforum ms-01 (it has built in dual SFP+) to hp proliant server with intel x520 NIC. the traffic path is:
image: transceiver block diagram, source: broadcom, newcom-acorn joint workshop presentation
so ideally, this would test the core and the access in full.
$ iperf3 -c 172.x.x.200 Connecting to host 172.x.x.200, port 5201 [5] local 172.x.x.221 port 52758 connected to 172.x.x.200 port 5201 [ID] Interval Transfer Bitrate Retr Cwnd [5] 0.00-1.00 sec 1.10 GBytes 9.43 Gbits/sec 0 1.68 MBytes [5] 1.00-2.00 sec 1.09 GBytes 9.40 Gbits/sec 1401 1.51 MBytes [5] 2.00-3.00 sec 1.10 GBytes 9.41 Gbits/sec 1110 1.61 MBytes [5] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 940 1.61 MBytes [5] 4.00-5.00 sec 1.09 GBytes 9.40 Gbits/sec 837 1.61 MBytes [5] 5.00-6.00 sec 1.09 GBytes 9.40 Gbits/sec 984 1.09 MBytes [5] 6.00-7.00 sec 892 MBytes 8.48 Gbits/sec 760 1.68 MBytes [5] 7.00-8.00 sec 1.09 GBytes 9.40 Gbits/sec 0 1.68 MBytes [5] 8.00-9.00 sec 1.10 GBytes 9.41 Gbits/sec 886 1.61 MBytes [5] 9.00-10.00 sec 1.09 GBytes 9.40 Gbits/sec 885 1.61 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ID] Interval Transfer Bitrate Retr [5] 0.00-10.00 sec 10.7 GBytes 9.41 Gbits/sec 7803 sender [5] 0.00-10.00 sec 10.7 GBytes 9.41 Gbits/sec receiver
throughput of 9.41 Gbits/sec is outstanding performance for a 10 Gigabit ethernet link. this is achieving approximately 94% of theoretical maximum, which is typical for well configured enterprise systems. real world factors like protocol overhead, CPU processing, and NIC efficiency typically cap out around 90-95%.
retransmissions in the test is 7803, this is the number of packets that had to be resent due to errors or congestion. that is about 780 per second. is it ideal? no. ideally, we want zero. but is it a problem, i would say no. it is just normal TCP behavior. the congestion window (cwnd) was bouncing around between 1.09 and 1.68 MBytes, which is healthy. the network was self-correcting.
there was one second where it dropped to 8.48 Gbps. probably a CPU interrupt hiccup or a switch buffer blip and it recovered instantly.
$ ethtool -S enp2s0f0np0 | grep -E "error|drop|miss" rx_errors: 0 tx_errors: 0 rx_dropped: 0 rx_missed_errors: 102 tx_dropped: 0 rx_length_errors: 0 rx_crc_errors: 0 veb.tx_errors: 0 port.tx_errors: 0 port.tx_dropped_link_down: 0 port.rx_crc_errors: 0 port.rx_length_errors: 0
i looked further to find the possible cause of the congestion, did couple of tests and found a few things, first was the NIC errors that i noticed on my ms-01. i used ethtool to check the interface stats.
CRC Errors (rx_crc_errors, port.rx_crc_errors): 0
this is the most critical metric. CRC errors indicate physical layer problems (bad cables, failing SFP+ modules, or electromagnetic interference or other). having zero confired the physical link was good.
Tx/Rx Errors and Dropped Packets: 0
no transmission failures or receive errors and the NIC itself is not dropping packets due to buffer overflows or driver issues.
the anomaly: rx_missed_errors: 102
in a 60-second test transferring 65 GB of data, receiving 102 missed packets is statistically negligible (roughly 1 packet every 600 MB). rx_missed_errors usually means the network card received a packet faster than the CPU (or the driver) could hand it off to the kernel network stack. the NIC internal buffer filled up momentarily, and it had to drop the packet. at 10G speeds, packet arrival rates are massive and if the CPU is busy with other tasks, or if the interrupt handling isn't perfectly tuned, the NIC can briefly "miss" a packet before the driver wakes up to process it.
started backup task '69e43334-e5df-4e2f-b5f7-893207da0bc0'
4% (1.5 GiB of 32.0 GiB) in 3s, read: 508.7 MiB/s, write: 350.1 MiB/s
7% (2.4 GiB of 32.0 GiB) in 6s, read: 305.7 MiB/s, write: 301.9 MiB/s
10% (3.3 GiB of 32.0 GiB) in 9s, read: 306.9 MiB/s, write: 303.7 MiB/s
12% (4.1 GiB of 32.0 GiB) in 12s, read: 282.8 MiB/s, write: 282.7 MiB/s
18% (6.0 GiB of 32.0 GiB) in 15s, read: 2.1 GiB/s, write: 129.5 MiB/s
45% (18.2 GiB of 32.0 GiB) in 18s, read: 1.7 GiB/s, write: 312.2 MiB/s
...
100% (32.0 GiB of 32.0 GiB) in 34s, read: 4.5 GiB/s, write: 0 B/s
backup is sparse: 22.87 GiB (71%) total zero data
transferred 32.00 GiB in 34 seconds (963.8 MiB/s)
some real scenario throughput test with the proxmox backup to NAS. while network link is capable of ~1.2 GB/s (9.4 Gbits/sec), write speed to the NAS is the limiting factor, hovering around 300-400 MiB/s.
read speed (proxmox): highly variable, spiking up to 2.1 GiB/s (approx. 17 Gbps). proxmox storage is fast enough to feed data into the network pipe easily. it is not the bottleneck.
write speed (NAS): stuck around 300-400 MiB/s (approx. 2.4-3.2 Gbps). the NAS is struggling to write data as fast as it arrives. this is the bottleneck.
network utilization: at peak write (400 MiB/s), here is only ~3.2 Gbps of 10G link used so only ~32% of the available network bandwidth is used.
backup is sparse means the proxmox backup detected that 71% of the data in the virtual machine disk is empty (zeros). instead of reading and sending over the network and writing 22.87 GiB of useless "empty space" to the NAS, proxmox used a deduplication optimization
everyone says 10G switches eat power like crazy. I put the tp-link access and mikrotik core switches on a smart plug to see the real deal.
total daily usage: 0.73 kWh for tp-link access and 0.22kWh for core switch. access switch PoE Load: 10.5 Watts (running 3 PoE devices).
$ show power inline System Power Limit: 240.0w System Power Consumption: 10.5w System Power Remain: 229.5w
lets do the math. 0.73 kWh over 24 hours = 30.4 Watts average, subtract the 10.5W for PoE, and the switch chassis itself is burning ~20 Watts. Is 20W a lot? Honestly? No. That is the price of keeping sixteen 2.5G/10G PHYs alive. The copper ports on these switches are power-hungry beasts compared to SFP+. But for the speed you get? It is efficient. The old Cisco was drawing similar power but delivering 1/4th the throughput.
upgrading to 10G isn't just about speed, it is about removing the friction. the old Cisco was a brick wall, the new omada is a highway. i got 9.4 Gbps throughput, zero CRC errors, and a backup that finished in less than 5min thanks to sparse magic. the power draw is reasonable (~20W for the chassis), and the 2.5G ports are perfect for everything else. so far it is stable and future proof.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。