There could be a thousand reasons something like this happens, but here's my situation:
A few months ago, when I was testing 10 Gigabit networking on the Raspberry Pi, I noticed something funny when I was doing iperf3 speed tests between devices connected through my MikroTik CRS305 10G switch.
The switch uses SFP+ jacks, and so I was testing a variety of connections—SFP+ to duplex fiber, SFP+ to RJ-45 copper, and SFP+ DAC (Direct Attach Cable) cables. One thing I found very strange was that sometimes, I would get in a situation where a particular device (Pi, my Mac, a Windows PC, or even my NAS) would get the full expected speed in one direction (e.g. my Mac to the NAS), but then if I tested it in reverse (NAS to my Mac), I would get horrible speeds—but only if I was using a 2.5 Gbps NIC on one end.
In the case of the NAS, which has 2.5G Ethernet, I was getting between 50 and 500 Mbps to it, with no consistency in the throughput.
I tested with multiple NICs, I tested both ports of my NAS, and I tested with multiple known-working-to-10G Cat6 patch cables.
And in every case, I was getting this asymmetric speed.
Eventually I bought a QNAP QSW-M2108-2C switch instead, which has 8 built-in 2.5G RJ-45 jacks, and 2 dual-mode SFP+/Copper 10G interfaces.
I plugged everything into it, forgoing fiber or DACs, and the problem went away.
So I shelved the MikroTik switch for the time being.
More MikroTik, More Problems?
Then a few weeks ago, a viewer of my YouTube channel offered me a MikroTik CRS309 switch for a song, so I had to take him up on the offer. I swapped it into my rack, and... exact same issue!
Thinking there's no possible way both MikroTik units could make one port work poorly at random, I finally ordered a new batch of RJ-45 transceivers, and started swapping them out (after going through the whack-a-mole with cables and devices to no avail).
And I found, after all that testing, that the FLYPROFiber SFP-10G-T-30M transceivers—at least some revisions—had trouble when a device negotiated 2.5G speeds through it.
As can be seen above, the FLYPRO transceivers were being identified as "multi-mode fiber", which also seemed suspect, since they are, indeed, copper RJ45 transceivers!
When connected through one of these, I'd get 2.3 Gbps one way, but the other direction would bottleneck at less than 500 Mbps, which was really horrible for editing video and backups to my NAS.
It was the Transceiver, dummy!
So yeah, when you're diagnosing network problems, don't leave any stone unturned. In my case, I was using a transceiver that I'd tested and had working with 2.5 and 10G devices, but in some cases, for some reason, it would only work one way.
Switching to MikroTik's own S+RJ10 transceiver resulted in full 2.3 Gbps bidirectional throughput.
I'm reminded of a time earlier this year when one of my Macs was getting 100 Mbps sometimes, and 10 Gbps others, seemingly depending on the direction the wind was blowing. In that case, it turns out one of the 8 wires in one keystone jack was rubbing against the shielded keystone casing, and causing the entire cable run to downrate to 100 Mbps... but only sometimes.
Always start with the patch cables. Then look at your equipment. Then check your drivers and device CPU/statistics. Then check transceivers, then jacks, terminations, and finally cabling. Networking can be... fun.
While this may not have been the direct cause, engrpiman2's post on Ars Technica was what finally jostled my brain to think to swap transceivers. Not everything that's 10G will also work well with 2.5G and 5GBASE-T devices. Lesson learned.
Comments
I want to just leave a tip regarding smoke testing the network path. If you don't care about the TCP performance of the stack on both ends, it's simplest to run iperf in UDP mode, so you can push as much traffic through the network as your endpoints can. Not having a flow control mechanism like TCP, the UDP flow will hit every interface and if you suspect a particular link, you will see a difference in the bitrate. Without drop counters, using TCP hides this link by link diagnostic tool as the entire path in one direction will be the speed capable according to the lowest common denominator, even if it's just a link that drops a packet from time to time.
Good point. You can also change the TCP window size and that can help pinpoint some weird conditions. There's a lot to iperf/iperf3 that I am still pretty green with.
lol I just got burned by this again today, testing the Radxa Taco SATA NAS board with a 2.5 Gbps Ethernet connection.
Hi Jeff,
I'm (completely unnecessarily) upgrading from a Ryzen 5 4600U Chinese mini PC to another with a Ryzen 7 5900H. The reason I'm here is that it has 2 2.5 Gbps NICs, so I naturally need to make a use of them, right? :) (Just when I significantly improved my network with a D-Link 24port managed PoE switch...) Anyway, I'm looking to you for tips on 2.5 Gbps+ switches, and this Mikrotik looks great! Quite reasonably priced for a managed 10G switch.
Anyway, thanks a lot for all the ridiculous fun projects! Pity it took me so long to find your channel :)
P.S.: The first link to the Mikrotik switch incorrectly shows/links to CRS305, which is a 5-port Gigabit switch. Might want to fix that :)
That was the one I originally used (and still have as a spare for testing), so the link/text is correct there ;)
Ah, just read the post properly. I see it's correct! Missed that the small 305 is also a 10G switch! Sorry about that :) Please ignore the previous comment.
Cheers Pete
Haha, no problem.
There is a logical explanation.
Internally your switch has an ethernet MAC connected to the SFP+ transceiver via differential pairs (commonly called serdes). The standard speeds for serdes are 1Gbit/s or 10Gbit/s.
If you use a fiber transceiver, and connect it to another switch the same way, then both switches will run auto-negotiation through the fiber to establish a working 1Gbit/s or 10Gbit/s link.
If you have a transceiver with a copper (RJ45) PHY, it's completely different. The serdes is still there, but there are actually two separate links. One link between the switch ethernet MAC and the "serdes side" of the RJ45 PHY inside the transceiver, and the other link between both RJ45 PHYs at each side of the RJ45 cable:
What happens if both links don't have the same speed ?
For example if RJ45 auto-negociates to 100Mbit/s or 10Mbit/s, then there is an issue because there is no standard way to run the serdes link at a speed lower than 1Gbit/s. So the switch would send data at 1Gbit/s, and inside your transceiver the same data would have to be sent over a 100Mbit/s copper link. The only way for this to work at all is to have buffering to absorb the rate change.
Buffering needs RAM, which is expensive and not something you want to put inside a hopefully cheap transceiver, so this was not the chosen solution back in 2004 when the problem appeared. It was solved by Cisco proprietary standard SGMII. This standard adds a special mode on the serdes link, which will still be running physically at 1Gbit/s, but will repeat the data either 10 or 100 times to emulate 100 and 10Mbit/s speed.
With SGMII, the ethernet MAC is now artificially throttled to send data on the 1 Gbit/s serdes at the same rate than the RJ45 link, so buffering takes place inside the switch where it should be (because switches do have buffers).
Fast forward in 2020, and the problem is there again with 2.5Gbit/s or 5Gbit/s capable RJ45 transceivers:
"Old" serdes can only run at 1Gbit/s or 10Gbit/s. For example an Intel x520 NIC will always report a link speed of 10Gbit/s if you use such a transceiver, and rightfully so because that's the truth.
But since the copper link runs at 2.5Gbit/s the transceiver will try to cope with link speed difference by buffering and/or using pause frames to the switch. They don't usually have enough memory to absorb large 10Gbit/s burst coming from the switch, so packet loss occurs.
What's happening in your case:
Dear Maxime,
thank you for explaining the problem. After months of testing with different SFP+ adapters and switches and experiencing horrible speeds of 150Mbit/s over WiFi, I came to the same conclusion. However, up until now, I did not entirely understood as to why. So thank you for your explanation.
I use a Fritz!Box 6690 with WiFi 6 that has one 2.5Gbit/s Ethernet port. The FB6690 connects to the 'cable'-network of my internet provider with 1150Mbit/s. Naturally, I wanted to have the full speed of the internet connection in my home network that uses 10Gbit/s fiber for all PCs. When I ordered an RJ-45/SFP+ module I achieved the full internet speed in the whole network. However, shortly after I noticed that I receive data from my Server (2x10Gbot/s fiber) via WiFi of the FB6690 only with 150Mbit/s instead of about 1.5Gbit/s (with a 2400Mbit/s link speed WiFi-6/160MHz).
I started to diagnose all network components, and learned that the problem does only exist when I download from the Server, but not when I upload to it. I had upstream speeds of 1.8Gbit/s via WiFi.
The first lesson I had to learn was that the SFP+ adapter (MikroTik S+RJ10 ) is not fully compatible with my switch TP-Link T1700G-28TQ. It was the exact issue you described here. The switch expected a 10Gbit/s link, while the bandwidth was limited to 2.5Gbit/s. When I got the MikroTik CRS305, the Link speed was detected correctly. However, I still had only about 150Mbit/s downstream from the server to the WiFi-6 Client.
I had to enable flow-control in the MikroTik switch to get higher speeds. I now get about 1.05Gbit/s downstream. This is still away from what I would expect (1.8Gbit/s would be nice), but it is at least usable. However, I noticed that quite a few link-paused messages appear in the MikroTik switch.
Also, I was curious, whether the TP-Link switch would be able to reach the same speeds with the RJ45/SFP+ module and Flow-Control enabled. It did not. I think this is because it still does not recognize the 2.5Gbit/s limitation. The interface showed 10Gbit/s to the RJ45/SFP+ module.
I thought about connecting the FB6690 directly to my Server which has 2 additional 10Gbit/s Ethernet ports to avoid the link pause issue. But I think, it will probably not negotiate 2.5Gbit/s, but instead fall back to 1Gbit/s. If anyone knows whether the "Ethernet controller: Intel Corporation Ethernet Connection X553/X557-AT 10GBASE-T (rev 11)" supports 2.5Gbit/s, please leave me a comment.
Also, I would like to know whether you think that my FB6690 has any issue, given that it is needs flow control enabled and is probably the device that asks to pause the link.
Do you have another idea, how I could configure my network so that I can get full 1.8Gbit/s downstream via WiFi using the FB6690?
Just want to chime in and say I'm experiencing the exact same thing. Recently upgraded since my ISP supports 3 gigabit. My 2.5G PC <> Router gives me 2.5g up but anywhere from 800m-1.5g down. This is happening with a 10Gtek ASF-10G-T. I'm going to try the Wiitek one tomorrow when it comes in and if that doesn't work - I'll go with the Mikrotik one.