Back in April, I upgraded my home LAN to 10Gb/s. The in-wall cabling is CAT-6 or similar, so I had to use 10GBASE-T. Now, the router I'm using, and the switch in my study, provide 10Gb/s through SFP+ cages; that meant that they needed 10GBASE-T SFP+ modules in order to connect.

That kind of module is known to run hot -- sometimes too hot to actually work. The modules in reggie, the router, appeared to be running OK (see the linked post above for charts), but the one in nigel, the study switch, was a worrying 93C. I tried sticking some mini-heatsinks on it, which seemed to help a bit. But the weather got warmer, and eventually the module overheated. I lost access to the Internet from the study, and checking the metrics showed me this:

You can see that it's "flapping": the temperature gets up to a level where the module shuts itself down for its own protection -- about 95C, I think -- and then when it has recovered, it switches on again, the temperature rises, and the process repeats.

I was able to work around the problem by switching on the air conditioning in the study. But normally I only have it on when I'm in there, and keeping aircon on 24/7 just to keep the network working felt like the wrong solution.

It was time to switch to a more power-efficient SFP+ module.

My original 10Gb/s post had quite a lot of discussion on Hacker News, and xxpor mentioned that there are two generations of 10GBASE-T SFP+ modules: old ones using a Marvell chip, and newer ones using one from Broadcom. blunden on the ServeTheHome forums made the same point. The Marvell-based ones were known to run hot, and they both recommended finding Broadcom-based ones.

I'd confirmed that the MikroTik S+RJ10 that I had in nigel was indeed a Marvell one, so the solution was pretty simple: get a better one. So I went on Amazon and picked up a 10Gtek ASF-10G-T80-INT. Checking 10Gtek's own page on that module confirmed that it used the right kind of chip (although it was a little bit garbled):

10Gtek's ASF-10G-T80 is a newest version copper transceiver, its biggest feature are ultra lowpower consumption and longer transmission distance (1.6W C10Gbps 30m，2.0W 110Gbps 80m). ASF-10G-T80 is a 10GBase mult-rate Copper RJ45 SFP+ transceiver, designed in with BROADCOM BCM84891 PHY chip following IEEE 802.3an/az and SFP+ MSA, supporting up to 80-meter transmission over CAT.6a or CAT.7.

A day or two later, it arrived. It came in a rather pretty little metal case:

Installing it took a little while, because I found removing the existing MikroTik module tricky; Willie Howe's video on YouTube helped quite a lot in showing how to disengage the latch, but I still needed to fiddle around with it quite a bit to get it out. However, that was eventually done, and the new module went in.

I plugged all of the network cables back in, switched on the switch, and (after a slightly nerve-wracking wait for it to boot up) the network was back up and running!

So, were the temperatures any better? I checked my monitoring, and:

Huh, nothing was being reported.

That made sense, though. The way I was charting those numbers was that the switch exposed them over SNMP, and then the Telegraf daemon on my router, reggie, read the numbers and sent them to InfluxDB; finally, Grafana did the charting.

I'd been reading the module temperatures in using the SNMP OID that I'd identified that the switch was providing them on (.1.3.6.1.4.1.14988.1.1.19.1.1.6.3 if you're interested), but perhaps the new module was published on a different OID. It was time to log in to the switch and take a look.

[admin@Nigel] > /interface ethernet monitor sfp-sfpplus1 once
                      name: sfp-sfpplus1
                    status: link-ok
          auto-negotiation: done
                      rate: 10Gbps
               full-duplex: yes
           tx-flow-control: yes
           rx-flow-control: yes
                 supported: 10M-baseT-half
                            10M-baseT-full
                            100M-baseT-half
                            100M-baseT-full
                            1G-baseT-half
                            1G-baseT-full
                            1G-baseX
                            2.5G-baseT
                            2.5G-baseX
                            5G-baseT
                            10G-baseT
                            10G-baseSR-LR
                            10G-baseCR
             sfp-supported: 1G-baseX
                            10G-baseSR-LR
               advertising: 1G-baseX
                            10G-baseSR-LR
  link-partner-advertising:
        sfp-module-present: yes
               sfp-rx-loss: no
              sfp-tx-fault: no
                  sfp-type: SFP/SFP+/SFP28/SFP56
        sfp-connector-type: LC
              sfp-encoding: 64B/66B
       sfp-link-length-om1: 30m
       sfp-link-length-om2: 80m
       sfp-link-length-om3: 300m
           sfp-vendor-name: Intel Corp
    sfp-vendor-part-number: FTLX8571D3BCV-IT
       sfp-vendor-revision: A
         sfp-vendor-serial: IN101Q14436
    sfp-manufacturing-date: 26-01-31
            sfp-wavelength: 850nm
           eeprom-checksum: good
                    eeprom: 0000: 03 04 07 10 00 00 00 00  00 00 00 06 67 00 00 00  ........ ....g...
                            0010: 08 03 00 1e 49 6e 74 65  6c 20 43 6f 72 70 20 20  ....Inte l Corp
                            0020: 20 20 20 20 00 00 1b 21  46 54 4c 58 38 35 37 31      ...! FTLX8571
                            0030: 44 33 42 43 56 2d 49 54  41 20 20 20 03 52 00 85  D3BCV-IT A   .R..
                            0040: 00 1a 00 00 49 4e 31 30  31 51 31 34 34 33 36 20  ....IN10 1Q14436
                            0050: 20 20 20 20 32 36 30 31  33 31 20 20 00 f0 03 96      2601 31  ....
                            0060: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........

It's saying that it's an Intel module; that in itself is not all that odd -- there are frequently compatibility issues between switches and SFP+ modules, so sometimes modules are configured to "lie" about which manufacturer made them -- and I'd specifically bought the "Intel-compatible" one on Amazon, the ASF-10G-T80-INT, because I couldn't find one that pretended to be MikroTik. Research had suggested that it would work OK, and it did.

But the really odd bits were these:

sfp-connector-type: LC
sfp-wavelength: 850nm
sfp-link-length-om1: 30m
sfp-link-length-om2: 80m
sfp-link-length-om3: 300m
sfp-vendor-part-number: FTLX8571D3BCV-IT

Not only was it impersonating an Intel module -- it was saying that it was a fibre-optic one! Perhaps if I had found the "MikroTik-compatible" option it would have been better -- though, equally, it might have just impersonated a MikroTik fibre module anyway.

Anyway, it was working -- so that was OK. But there was some bad news. If the switch was able to read a temperature from the new module, then you'd expect it to appear in that output, as sfp-temperature. So, sadly, I don't think I'll be able to monitor the temperature of the new module.

How could I tell whether it had helped, then? Well, one thing would be to simply see if there are any further instances of network flapping. I actually did the replacement just over two weeks ago, and everything has been fine as far as I can tell from using it and from the other monitoring (despite another hot week last week).

But another interesting metric is the CPU temperature for nigel over the two weeks before and after the module change:

You can see that there was a real drop-off late on 1 June, when I switched the modules, and it has been running about 5C cooler since. Of course, there's a lot that's different about the new module -- as well as having a different chipset and a mendacious EEPROM, it's likely to have different thermal coupling characteristics -- it might be shedding more or less of its heat to the SFP+ cage and thence to the switch's CPU. So it's not proof of anything, but in combination with the improved link stability, I'll take it as a win.

So, an interesting little excursion into the world of SFP+ modules -- in particular, slightly dodgy ones :-) Let's see if this one holds up better as we go through the toasty Lisbon summer.