FYI, one can try this out using
iperf 2.1 with
--trip-times. This gives end/end delay at the application level. One can use --trip-times when clocks
are sync'd to get the write to read latencies which are the latencies at the application level.
Note: I set up a Raspberry Pi 4 with a GPS hat from
ubutronics for solderless pulse per second. Then configured it as a PTP grandmaster. This cost me around $200.
I also added support for a very crude --near-congestion option that paces the writes based upon the weight of the RTT. The tcp_info struct is sampled and available
for other experiments though one would have to modify the source a bit. This current technique used by iperf 2.1 is designed for test networks only where all
traffic is under script control. We've had too many people measuring bloat as latency. We really need separate measurements between the two phenomena,
bloat vs latency, because they require different eng actions for a semiconductor supplier.
Below are examples over a 10G link, first with no write pacing then with it. The server output, shown first, has the latency data (as well as the net power
and little's law calculation.) (Note: use --histograms for to get full distributions.)
No write pacing
[rjmcmahon@localhost iperf2-code]$ src/iperf -s -i 1 -e
------------------------------------------------------------
Server listening on TCP port 5001 with pid 24568
Read buffer size: 128 KByte (Dist bin width=16.0 KByte)
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.1.10%enp2s0 port 5001 connected with 192.168.1.62 port 50056 (MSS=1448) (trip-times) (sock=4) (peer 2.1.0-rc) on 2021-01-07 12:11:04 (PST)
[ ID] Interval Transfer Bandwidth Burst Latency avg/min/max/stdev (cnt/size) inP NetPwr Reads=Dist
[ 1] 0.00-1.00 sec 1.09 GBytes 9.34 Gbits/sec 2.959/1.180/3.681/0.388 ms (8905/131072) 3.31 MByte 394522 18480=2459:2580:2475:2354:2203:2192:1974:2243
[ 1] 1.00-2.00 sec 1.10 GBytes 9.41 Gbits/sec 2.993/2.302/3.703/0.359 ms (8978/131072) 3.36 MByte 393209 19482=2526:2850:3102:2622:2344:2297:1867:1874
[ 1] 2.00-3.00 sec 1.10 GBytes 9.42 Gbits/sec 3.010/2.302/3.692/0.347 ms (8978/131085) 3.38 MByte 391047 19387=2563:2757:2928:2708:2432:2244:1829:1926
[ 1] 3.00-4.00 sec 1.10 GBytes 9.41 Gbits/sec 3.009/2.301/3.668/0.348 ms (8979/131060) 3.38 MByte 391094 18821=2456:2585:2660:2545:2270:2239:1906:2160
[ 1] 4.00-5.00 sec 1.10 GBytes 9.42 Gbits/sec 2.985/2.299/3.696/0.359 ms (8979/131070) 3.35 MByte 394295 19441=2509:2886:2959:2728:2336:2200:1971:1852
[ 1] 5.00-6.00 sec 1.10 GBytes 9.41 Gbits/sec 2.977/2.258/3.671/0.363 ms (8978/131082) 3.34 MByte 395352 18509=2352:2602:2464:2380:2263:2142:2095:2211
[ 1] 6.00-7.00 sec 1.10 GBytes 9.41 Gbits/sec 2.980/2.290/3.680/0.363 ms (8978/131072) 3.34 MByte 394873 18522=2407:2499:2565:2334:2213:2268:1999:2237
[ 1] 7.00-8.00 sec 1.10 GBytes 9.42 Gbits/sec 2.980/2.253/3.702/0.362 ms (8979/131073) 3.35 MByte 394972 18615=2427:2592:2493:2460:2281:2057:2062:2243
[ 1] 8.00-9.00 sec 1.10 GBytes 9.41 Gbits/sec 2.976/2.277/3.663/0.364 ms (8979/131065) 3.34 MByte 395443 18632=2338:2615:2647:2351:2192:2317:2063:2109
[ 1] 9.00-10.00 sec 1.10 GBytes 9.41 Gbits/sec 2.976/2.293/3.690/0.366 ms (8978/131076) 3.34 MByte 395416 18428=2281:2622:2497:2275:2178:2253:2129:2193
[ 1] 0.00-10.00 sec 11.0 GBytes 9.41 Gbits/sec 2.984/1.180/3.703/0.362 ms (89736/131072) 3.35 MByte 394014 188367=24320:26609:26793:24757:22712:22211:19916:21049
[rjmcmahon@localhost iperf2-code]src/iperf -c 192.168.1.10 --trip-times -i 1 -e
------------------------------------------------------------
Client connecting to 192.168.1.10, TCP port 5001 with pid 18961 (1 flows)
Write buffer size: 131072 Byte
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.1.62%enp2s0 port 50056 connected with 192.168.1.10 port 5001 (MSS=1448) (trip-times) (sock=3) (ct=0.41 ms) on 2021-01-07 12:11:04 (PST)
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT NetPwr
[ 1] 0.00-1.00 sec 1.09 GBytes 9.37 Gbits/sec 8937/0 0 1508K/1099 us 1065750
[ 1] 1.00-2.00 sec 1.10 GBytes 9.41 Gbits/sec 8975/0 0 1508K/1087 us 1082218
[ 1] 2.00-3.00 sec 1.10 GBytes 9.41 Gbits/sec 8975/0 0 1508K/1081 us 1088225
[ 1] 3.00-4.00 sec 1.10 GBytes 9.42 Gbits/sec 8984/0 0 1508K/1085 us 1085300
[ 1] 4.00-5.00 sec 1.10 GBytes 9.42 Gbits/sec 8980/0 0 1508K/1105 us 1065182
[ 1] 5.00-6.00 sec 1.10 GBytes 9.41 Gbits/sec 8975/0 0 1582K/1100 us 1069428
[ 1] 6.00-7.00 sec 1.10 GBytes 9.42 Gbits/sec 8979/0 0 1582K/1121 us 1049862
[ 1] 7.00-8.00 sec 1.10 GBytes 9.41 Gbits/sec 8976/0 0 1582K/1133 us 1038396
[ 1] 8.00-9.00 sec 1.10 GBytes 9.41 Gbits/sec 8978/0 0 1582K/1115 us 1055394
[ 1] 9.00-10.00 sec 1.10 GBytes 9.42 Gbits/sec 8986/0 0 1582K/1122 us 1049744
[ 1] 0.00-10.00 sec 11.0 GBytes 9.41 Gbits/sec 89748/0 0 1582K/1122 us 1048294
With write pacing
[rjmcmahon@localhost iperf2-code]$ src/iperf -s -i 1 -e
------------------------------------------------------------
Server listening on TCP port 5001 with pid 24702
Read buffer size: 128 KByte (Dist bin width=16.0 KByte)
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.1.10%enp2s0 port 5001 connected with 192.168.1.62 port 50072 (MSS=1448) (trip-times) (sock=4) (peer 2.1.0-rc) on 2021-01-07 12:14:59 (PST)
[ ID] Interval Transfer Bandwidth Burst Latency avg/min/max/stdev (cnt/size) inP NetPwr Reads=Dist
[ 1] 0.00-1.00 sec 1.08 GBytes 9.31 Gbits/sec 0.401/0.193/2.682/0.168 ms (8876/131084) 456 KByte 2904347 19868=3296:2404:2508:2797:3559:1778:1551:1975
[ 1] 1.00-2.00 sec 1.10 GBytes 9.41 Gbits/sec 0.400/0.219/0.627/0.053 ms (8971/131071) 460 KByte 2937822 19117=3069:2267:2307:2510:3029:1824:1683:2428
[ 1] 2.00-3.00 sec 1.09 GBytes 9.39 Gbits/sec 0.374/0.193/0.541/0.055 ms (8958/131060) 428 KByte 3143030 18942=2846:2423:2304:2417:2927:1831:1856:2338
[ 1] 3.00-4.00 sec 1.09 GBytes 9.39 Gbits/sec 0.385/0.190/0.664/0.070 ms (8952/131072) 441 KByte 3050401 19248=3041:2175:2343:2749:3320:1805:1526:2289
[ 1] 4.00-5.00 sec 1.09 GBytes 9.40 Gbits/sec 0.380/0.197/0.546/0.057 ms (8965/131075) 436 KByte 3095915 19959=3321:2398:2551:2738:3500:1840:1532:2079
[ 1] 5.00-6.00 sec 1.09 GBytes 9.39 Gbits/sec 0.369/0.198/0.536/0.051 ms (8956/131072) 423 KByte 3177431 21060=3627:2456:2886:3189:4246:1813:1190:1653
[ 1] 6.00-7.00 sec 1.09 GBytes 9.39 Gbits/sec 0.380/0.202/0.562/0.054 ms (8959/131077) 436 KByte 3086914 19263=3044:2338:2424:2505:3155:1809:1636:2352
[ 1] 7.00-8.00 sec 1.09 GBytes 9.40 Gbits/sec 0.376/0.198/0.541/0.053 ms (8965/131061) 432 KByte 3122495 19137=3079:2303:2340:2455:3017:1822:1683:2438
[ 1] 8.00-9.00 sec 1.10 GBytes 9.41 Gbits/sec 0.381/0.208/0.576/0.054 ms (8974/131073) 438 KByte 3083767 19162=3050:2269:2392:2486:3019:1891:1667:2388
[ 1] 9.00-10.00 sec 1.09 GBytes 9.40 Gbits/sec 0.371/0.194/0.582/0.057 ms (8964/131070) 425 KByte 3169244 19143=3006:2411:2303:2462:3067:1744:1760:2390
[ 1] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec 0.382/0.190/2.682/0.076 ms (89544/131072) 437 KByte 3074913 194908=31380:23444:24362:26308:32839:18161:16084:22330
[rjmcmahon@localhost iperf2-code]$ src/iperf -c 192.168.1.10 --near-congestion=0.05 --trip-times -i 1 -e
------------------------------------------------------------
Client connecting to 192.168.1.10, TCP port 5001 with pid 19320 (1 flows)
Write buffer size: 131072 Byte
TCP near-congestion delay weight set to 0.0500
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.1.62%enp2s0 port 50072 connected with 192.168.1.10 port 5001 (MSS=1448) (trip-times) (sock=3) (ct=0.40 ms) on 2021-01-07 12:14:59 (PST)
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT NetPwr
[ 1] 0.00-1.00 sec 1.08 GBytes 9.31 Gbits/sec 8881/0 0 1135K/373 us 3120427
[ 1] 1.00-2.00 sec 1.10 GBytes 9.41 Gbits/sec 8971/0 0 1135K/391 us 3007281
[ 1] 2.00-3.00 sec 1.09 GBytes 9.39 Gbits/sec 8958/0 0 1135K/331 us 3547260
[ 1] 3.00-4.00 sec 1.09 GBytes 9.39 Gbits/sec 8952/0 0 1135K/288 us 4074155
[ 1] 4.00-5.00 sec 1.09 GBytes 9.40 Gbits/sec 8965/0 0 1135K/301 us 3903855
[ 1] 5.00-6.00 sec 1.09 GBytes 9.39 Gbits/sec 8955/0 0 1135K/414 us 2835144
[ 1] 6.00-7.00 sec 1.09 GBytes 9.40 Gbits/sec 8961/0 0 1135K/470 us 2499013
[ 1] 7.00-8.00 sec 1.09 GBytes 9.40 Gbits/sec 8964/0 0 1135K/350 us 3356941
[ 1] 8.00-9.00 sec 1.10 GBytes 9.41 Gbits/sec 8973/0 0 1135K/472 us 2491756
[ 1] 9.00-10.00 sec 1.09 GBytes 9.40 Gbits/sec 8964/0 0 1135K/402 us 2922710
[ 1] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec 89547/0 0 1135K/402 us 2919642
Bob