Advertise MAC_CAPAB_OVERLAY
#920
Open
+8
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is a GLDv3-private capability, but stlouis will use this information to determine whether we should receive 10GbE equiavalent software fanout (i.e., up to 8 soft rings). Using this capability (with some stlouis changes to remove the VNIC requirement) should allow us to avoid having all fan-in from being serialised by a single softring worker thread.
Collaborator
Author
|
Conditioned on getting https://code.oxide.computer/c/illumos-gate/+/692 in... Degree of fanout is 8-way (divide EVT22200014 # mdb -k
Loading modules: [ unix genunix specfs dtrace mac cpu.generic apix cpc crypto mm random smbios zfs sata ip hook neti sockfs lofs vmm scsi_vhci arp ufs ipcc logindmux nsmb ptm ]
> ::mac_srs -r
MBLK Q SR
ADDR LINK_NAME STATE TYPE CNT BYTES CNT
fffffcfa43f471c0 cxgbe1 2400000e 000030b1 0 0 15
fffffcfa43f488c0 cxgbe1 2400000e 000030b1 0 0 15
fffffcfa43f4d180 cxgbe1 2400000e 000030b1 0 0 15
fffffcfa43f57800 cxgbe1 24000002 00003031 0 0 15
fffffcfa3575c140 igb0 2400000e 00003091 0 0 5
fffffcfa3575d840 igb0 24000002 00003011 0 0 5
fffffcfa30f2d100 opte0 24000002 00003031 0 0 40
fffffcfa30f2e800 cxgbe0 24000002 00003011 0 0 5
fffffcfa407367c0 cxgbe0 2400000e 00003091 0 0 5
fffffcfa40738080 cxgbe0 2400000e 00003091 0 0 5
fffffcfa40739780 cxgbe0 2400000e 00003091 0 0 5
>And testing with the native TCP stack atop OPTE is multiplicative, but not 8x. We are functionally capped to 3-way fanin when coming via only cxgbe1 like this. root@EVT22200009:~# zlogin a
[Connected to zone 'a' pts/5]
The illumos Project stlouis-0-g6b4a0fc6f8 January 2026
root@a:~# iperf -c 10.0.0.1
Connecting to host 10.0.0.1, port 5201
[ 4] local 10.0.0.2 port 53875 connected to 10.0.0.1 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 1.36 GBytes 11.7 Gbits/sec
[ 4] 1.00-2.00 sec 2.04 GBytes 17.5 Gbits/sec
[ 4] 2.00-3.00 sec 2.05 GBytes 17.7 Gbits/sec
[ 4] 3.00-4.00 sec 2.00 GBytes 17.2 Gbits/sec
[ 4] 4.00-5.00 sec 2.04 GBytes 17.6 Gbits/sec
[ 4] 5.00-6.00 sec 2.04 GBytes 17.6 Gbits/sec
[ 4] 6.00-7.00 sec 1.98 GBytes 17.0 Gbits/sec
[ 4] 7.00-8.00 sec 2.05 GBytes 17.6 Gbits/sec
[ 4] 8.00-9.00 sec 2.06 GBytes 17.7 Gbits/sec
[ 4] 9.00-10.00 sec 1.98 GBytes 17.0 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 19.6 GBytes 16.9 Gbits/sec sender
[ 4] 0.00-10.00 sec 19.6 GBytes 16.9 Gbits/sec receiver
iperf Done.
root@a:~# iperf -c 10.0.0.1 -P2 19:39 [218/4800]
Connecting to host 10.0.0.1, port 5201
[ 4] local 10.0.0.2 port 37560 connected to 10.0.0.1 port 5201
[ 6] local 10.0.0.2 port 34454 connected to 10.0.0.1 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 1.11 GBytes 9.52 Gbits/sec
[ 6] 0.00-1.00 sec 1.03 GBytes 8.88 Gbits/sec
[SUM] 0.00-1.00 sec 2.14 GBytes 18.4 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 1.00-2.00 sec 1.57 GBytes 13.5 Gbits/sec
[ 6] 1.00-2.00 sec 1.46 GBytes 12.5 Gbits/sec
[SUM] 1.00-2.00 sec 3.03 GBytes 26.0 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 2.00-3.00 sec 1.62 GBytes 13.9 Gbits/sec
[ 6] 2.00-3.00 sec 1.52 GBytes 13.0 Gbits/sec
[SUM] 2.00-3.00 sec 3.14 GBytes 27.0 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
<SNIP>
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 9.00-10.00 sec 1.61 GBytes 13.9 Gbits/sec
[ 6] 9.00-10.00 sec 1.51 GBytes 13.0 Gbits/sec
[SUM] 9.00-10.00 sec 3.12 GBytes 26.8 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 15.5 GBytes 13.3 Gbits/sec sender
[ 4] 0.00-10.00 sec 15.5 GBytes 13.3 Gbits/sec receiver
[ 6] 0.00-10.00 sec 14.5 GBytes 12.5 Gbits/sec sender
[ 6] 0.00-10.00 sec 14.5 GBytes 12.5 Gbits/sec receiver
[SUM] 0.00-10.00 sec 30.1 GBytes 25.8 Gbits/sec sender
[SUM] 0.00-10.00 sec 30.1 GBytes 25.8 Gbits/sec receiver
iperf Done.
root@a:~# iperf -c 10.0.0.1 -P8 19:41 [84/4800]
Connecting to host 10.0.0.1, port 5201
[ 4] local 10.0.0.2 port 41670 connected to 10.0.0.1 port 5201
[ 6] local 10.0.0.2 port 42993 connected to 10.0.0.1 port 5201
[ 8] local 10.0.0.2 port 39582 connected to 10.0.0.1 port 5201
[ 10] local 10.0.0.2 port 61995 connected to 10.0.0.1 port 5201
[ 12] local 10.0.0.2 port 49075 connected to 10.0.0.1 port 5201
[ 14] local 10.0.0.2 port 45929 connected to 10.0.0.1 port 5201
[ 16] local 10.0.0.2 port 55218 connected to 10.0.0.1 port 5201
[ 18] local 10.0.0.2 port 41348 connected to 10.0.0.1 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 127 MBytes 1.06 Gbits/sec
[ 6] 0.00-1.00 sec 330 MBytes 2.77 Gbits/sec
[ 8] 0.00-1.00 sec 127 MBytes 1.06 Gbits/sec
[ 10] 0.00-1.00 sec 429 MBytes 3.60 Gbits/sec
[ 12] 0.00-1.00 sec 366 MBytes 3.07 Gbits/sec
[ 14] 0.00-1.00 sec 292 MBytes 2.45 Gbits/sec
[ 16] 0.00-1.00 sec 355 MBytes 2.97 Gbits/sec
[ 18] 0.00-1.00 sec 328 MBytes 2.75 Gbits/sec
[SUM] 0.00-1.00 sec 2.30 GBytes 19.7 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 1.00-2.00 sec 620 MBytes 5.20 Gbits/sec
[ 6] 1.00-2.00 sec 312 MBytes 2.62 Gbits/sec
[ 8] 1.00-2.00 sec 453 MBytes 3.80 Gbits/sec
[ 10] 1.00-2.00 sec 424 MBytes 3.56 Gbits/sec
[ 12] 1.00-2.00 sec 655 MBytes 5.50 Gbits/sec
[ 14] 1.00-2.00 sec 287 MBytes 2.41 Gbits/sec
[ 16] 1.00-2.00 sec 24.0 MBytes 201 Mbits/sec
[ 18] 1.00-2.00 sec 502 MBytes 4.21 Gbits/sec
[SUM] 1.00-2.00 sec 3.20 GBytes 27.5 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 2.00-3.00 sec 602 MBytes 5.05 Gbits/sec
[ 6] 2.00-3.00 sec 478 MBytes 4.01 Gbits/sec
[ 8] 2.00-3.00 sec 347 MBytes 2.91 Gbits/sec
[ 10] 2.00-3.00 sec 558 MBytes 4.68 Gbits/sec
[ 12] 2.00-3.00 sec 527 MBytes 4.42 Gbits/sec
[ 14] 2.00-3.00 sec 290 MBytes 2.43 Gbits/sec
[ 16] 2.00-3.00 sec 313 MBytes 2.63 Gbits/sec
[ 18] 2.00-3.00 sec 255 MBytes 2.14 Gbits/sec
[SUM] 2.00-3.00 sec 3.29 GBytes 28.3 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
<SNIP>
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 9.00-10.00 sec 582 MBytes 4.88 Gbits/sec
[ 6] 9.00-10.00 sec 394 MBytes 3.31 Gbits/sec
[ 8] 9.00-10.00 sec 345 MBytes 2.89 Gbits/sec
[ 10] 9.00-10.00 sec 274 MBytes 2.29 Gbits/sec
[ 12] 9.00-10.00 sec 524 MBytes 4.40 Gbits/sec
[ 14] 9.00-10.00 sec 341 MBytes 2.86 Gbits/sec
[ 16] 9.00-10.00 sec 507 MBytes 4.25 Gbits/sec
[ 18] 9.00-10.00 sec 351 MBytes 2.95 Gbits/sec
[SUM] 9.00-10.00 sec 3.24 GBytes 27.8 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 5.50 GBytes 4.73 Gbits/sec sender
[ 4] 0.00-10.00 sec 5.50 GBytes 4.73 Gbits/sec receiver
[ 6] 0.00-10.00 sec 4.58 GBytes 3.93 Gbits/sec sender
[ 6] 0.00-10.00 sec 4.58 GBytes 3.93 Gbits/sec receiver
[ 8] 0.00-10.00 sec 4.84 GBytes 4.16 Gbits/sec sender
[ 8] 0.00-10.00 sec 4.84 GBytes 4.16 Gbits/sec receiver
[ 10] 0.00-10.00 sec 3.43 GBytes 2.95 Gbits/sec sender
[ 10] 0.00-10.00 sec 3.43 GBytes 2.95 Gbits/sec receiver
[ 12] 0.00-10.00 sec 4.68 GBytes 4.02 Gbits/sec sender
[ 12] 0.00-10.00 sec 4.68 GBytes 4.02 Gbits/sec receiver
[ 14] 0.00-10.00 sec 3.49 GBytes 3.00 Gbits/sec sender
[ 14] 0.00-10.00 sec 3.49 GBytes 3.00 Gbits/sec receiver
[ 16] 0.00-10.00 sec 2.19 GBytes 1.89 Gbits/sec sender
[ 16] 0.00-10.00 sec 2.19 GBytes 1.89 Gbits/sec receiver
[ 18] 0.00-10.00 sec 2.64 GBytes 2.27 Gbits/sec sender
[ 18] 0.00-10.00 sec 2.64 GBytes 2.27 Gbits/sec receiver
[SUM] 0.00-10.00 sec 31.4 GBytes 26.9 Gbits/sec sender
[SUM] 0.00-10.00 sec 31.4 GBytes 26.9 Gbits/sec receiver
iperf Done.
root@a:~# |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a GLDv3-private capability, but stlouis will use this information to determine whether we should receive 10GbE equiavalent software fanout (i.e., up to 8 soft rings).
Using this capability (with some stlouis changes to remove the VNIC requirement) should allow us to avoid having all fan-in from being serialised by a single softring worker thread.