From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) by mail.toke.dk (Postfix) with ESMTPS id 844AB95A293 for ; Mon, 18 Apr 2022 02:22:17 +0200 (CEST) Authentication-Results: mail.toke.dk; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=kTWtlNdi Received: by mail-ej1-x631.google.com with SMTP id ks6so24412390ejb.1 for ; Sun, 17 Apr 2022 17:22:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=Q3Udvc3nRHqOkTP+d5Ox6zYQSNRp26BCa1MJwDNk81o=; b=kTWtlNdiy2dwfy6Ze+iiJv/AzBVtsVMsr4maOEo8jZGSLOG106hSdD9/oahJfxY+2t lcr8pyZdCwwYr/a5zCA+QffmpG6cQXWNy/0++F8nA5vI6LFYKym90zFXjo00+07n9hQA QI8hgnRXW7SSlAaH4cHQUYHL8Cbze8rguS17LOL8Wk3aOMKdETZBD0nGikhkcSUwdqUj vAsIWjVxM1IWenhdPitq7IH6Nrz+zNp+t7H2Vb3agdYt7WMnHgQLgfLjn1g/QUKI2FXb PK0M4vGhXLMMMzd+ybaUpucrai1sehuaEpd5G9bvuZ06X1WbTlh8fP8Upox5e1MVGXhH QyXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=Q3Udvc3nRHqOkTP+d5Ox6zYQSNRp26BCa1MJwDNk81o=; b=TBhrtXMKP8xRPmHPJi1eDyMyaxnGfeGK2PloaRGstP0YTQ3xrFLvDtY+PHesYtasY5 lWHKdFVomZb/3dj8CLdtR6HLF+ksVkxqMyIN51sCtp78aEjrudDz86OjoiRjPS4D7biv CxDhY2IJmnPat9PmKBLUSkLng4yYWEF0+69T7EB+YKDqVqVjZFuHnsbSkoTOXDFk/h6v 3JrklIb+7JwVAvBKf0w2uoYqxblFy6Qnr9zhlG3BEnfK5iTI/hO7nyz0xY4A4HEV6Hbx QfBPcJMiU9ze1BuYeQb9EdpJxH6LThUle9MnquBAFjHXjebLwUQ5QTG/SspEJ6kOaICF hg4g== X-Gm-Message-State: AOAM5307hXYJDnVAY/Aeb6LMWBDZJqKjmw748KDyBqQrpOz/teptY93J jNVFYQWPb4LTml6u6VuxN0eOdn/9pxXd88Z53UM= X-Google-Smtp-Source: ABdhPJzKC9LcXg+ExylKC0vFeUVct15e2W95BYBfBynFAgcj7SKkecovaluaBoXSQSdWf9J+8VSMIhuiFTw7w1MMgr8= X-Received: by 2002:a17:906:5fc6:b0:6e8:86bd:64ab with SMTP id k6-20020a1709065fc600b006e886bd64abmr6892924ejv.538.1650241335439; Sun, 17 Apr 2022 17:22:15 -0700 (PDT) MIME-Version: 1.0 References: <14d1f01d85268$68d55f00$3a801d00$@inacomptc.com> <14d2801d85270$d9820020$8c860060$@inacomptc.com> In-Reply-To: <14d2801d85270$d9820020$8c860060$@inacomptc.com> From: Dave Taht Date: Sun, 17 Apr 2022 17:22:02 -0700 Message-ID: To: Jordan Szuch Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Message-ID-Hash: VUWAOC7VAYBCP5BI25N73BNROBGUMJ4B X-Message-ID-Hash: VUWAOC7VAYBCP5BI25N73BNROBGUMJ4B X-MailFrom: dave.taht@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: flent-users , bloat , starlink@lists.bufferbloat.net X-Mailman-Version: 3.3.5 Precedence: list Subject: [Flent-users] Re: [Starlink] an rtt_fair test? List-Id: Flent discussion list Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Sun, Apr 17, 2022 at 8:36 AM wrote: > > You're welcome! Here's the files Flent generated: https://1drv.ms/u/s!Ap4= u4Rte63FqjqoSQXX444OWLvl8FQ?e=3Df22Nt7 Thank you for the data! Part of with me unleashing this test on y'all is to validate my cloud, which was looking weird yesterday. I still haven't quite got to the problem fremont or london actually had, but it's much better now. > And thanks for the link! It honestly doesn't look too bad to implement so= I might give it a shot today if time allows. Honestly, I hope y'all are out there spending time with your families today. The Net can wait. > > I've also been contemplating replacing it with something like a Mikrotik = since cake is now integrated into them. I've been testing one at home and m= ade a post with a bunch of flent tests to the cake thread in their forum if= you're interested in some data from a Charter Spectrum cable connection: h= ttps://forum.mikrotik.com/viewtopic.php?t=3D179307#p925485 That is a really wonderful addition to that thread, thank you. For those here that haven't gone through it, the pictures help a lot, but requires you register for a free account. Perhaps at some point we can eliminate the confusion and noise from it, but I often find it useful to deeply understand what mistakes others will make. ( I have no idea why anyone is using the bucket thing on that bug thread... no buckets needed. I don't get it) Seeing that horrible 1sec of induced latency on spectrum and knowing that they have 28m subscribers whose networks still behave like that, pains me. I would so hope that their gbit service, at least, was only in the 120ms range. Pro tip: tc -s qdisc show shows you the drop/market/reschedules on the client, you might see a few reschedules under the structure of this test but no drops as "TCP small queues" keeps that together. On router, the drop ratios are geometrically relative to the bandwidth. If I have any one gripe left about the mikrotik release it is that they make backlog, packet and ,drop statistics easily available across their OSes. I'm pretty sure that at higher bandwidths their default settings for SFQ and pfifo are too low... > > The OneDrive link to my flent files should still be active in that post a= s well. > > -Jordan > > -----Original Message----- > From: Dave Taht > Sent: Sunday, April 17, 2022 10:57 AM > To: Jordan Szuch > Cc: flent-users ; bloat ; starlink@lists.bufferbloat.net > Subject: Re: [Starlink] an rtt_fair test? > > Thank you for testing!!! If you could stash the *.flent.gz files somewher= e I could get at, that would be great. You can also browse the plots using = the flent-gui to see more detail over time. > > Cake, long available as an out of tree build for the edgerouter, has both= dsl compensation and an ack-filter which would help A LOT, in your case. i= wish I'd pushed harder to get dsl compensation into the edgerouter smart q= ueues code... > > https://community.ui.com/questions/Cake-compiled-for-the-EdgeRouter-devic= es/fc1ff27c-f321-4344-8737-fcc755cae8a2 > > Given your 500kbit upload speed the minimum latency inflation under the s= tructure of this test (4 upload flows) should have been around 150ms (4 * 2= 5ms + ack overhead), but without looking at the *.gz files I don't know wha= t your baseline latency was to any of these sites. (it would be good if fle= nt reported on the text output what the minimum actually was) > > On Sun, Apr 17, 2022 at 7:35 AM wrote: > > > > Hi Dave, > > > > This isn't specifically cake related but I figured it could be interest= ing, especially as these are from a location where I've been investigating = setting up something with cake. These are statistics from a relatives locat= ion where they only have a DSL line from CenturyLink available to them. The= re's a little Debian based VM I setup for a couple random things so I was a= ble to run flent from that. They have a EdgeRouter X serving as the router = so they can at least take advantage of fq_codel through the routers Smart Q= ueue feature. I ran a quick speed test, then ran flent without sqm, and the= n with sqm. Sorry if this gets a bit long and hopefully it stays readable. > > > > ############# > > Speedtest by Ookla > > > > Server: 123.Net, Inc. - Southfield, MI (id =3D 15044) > > ISP: CenturyLink > > Latency: 45.45 ms (194.34 ms jitter) > > Download: 8.88 Mbps (data used: 12.6 MB ) > > Upload: 0.64 Mbps (data used: 962.7 kB ) > > Packet Loss: 5.1% > > Result URL: > > https://www.speedtest.net/result/c/799ba46b-d3d4-4d54-ac76-118fac847b7 > > 3 > > > > ########### > > No SQM or QOS > > > > flent -x --socket-stats --step-size=3D.05 -H de.starlink.taht.net -H > > london.starlink.taht.net -H singapore.starlink.taht.net -H fremont.star= link.taht.net -t nosqm_erx_rttfair4be_8Mbpsdown_500Kbpsup_centurylinkdsl rt= t_fair4be Started Flent 2.0.0 using Python 3.9.2. > > Starting rtt_fair4be test. Expected run time: 70 seconds. > > Data file written to > > ./rtt_fair4be-2022-04-17T145553.865990.nosqm_erx_rttfair4be_8Mbpsdown_ > > 500Kbpsup_centurylinkdsl.flent.gz > > > > Summary of rtt_fair4be test run from 2022-04-17 13:55:53.865990 > > Title: 'nosqm_erx_rttfair4be_8Mbpsdown_500Kbpsup_centurylinkdsl' > > > > = avg median # data pts > > Ping (ms) ICMP1 de.starlink.taht.net : 1= 026.06 920.00 ms 1383 > > Ping (ms) ICMP2 london.starlink.taht.net : 1= 009.28 887.00 ms 1383 > > Ping (ms) ICMP3 singapore.starlink.taht.net : 1= 121.09 1012.00 ms 1383 > > Ping (ms) ICMP4 fremont.starlink.taht.net : = 943.77 853.00 ms 1383 > > Ping (ms) avg : 1= 025.05 N/A ms 1383 > > TCP download BE1 de.starlink.taht.net : = 1.36 2.00 Mbits/s 1383 > > TCP download BE2 london.starlink.taht.net : = 0.72 0.65 Mbits/s 1383 > > TCP download BE3 singapore.starlink.taht.net : = 1.03 1.74 Mbits/s 1383 > > TCP download BE4 fremont.starlink.taht.net : = 2.11 2.88 Mbits/s 1383 > > TCP download avg : = 1.30 N/A Mbits/s 1383 > > TCP download fairness : = 0.86 N/A Mbits/s 1383 > > TCP download sum : = 5.22 N/A Mbits/s 1383 > > TCP upload BE1 de.starlink.taht.net : = 0.12 0.20 Mbits/s 1383 > > TCP upload BE1 de.starlink.taht.net::tcp_cwnd : = 16.85 14.00 1044 > > TCP upload BE1 de.starlink.taht.net::tcp_delivery_rate : = 0.12 0.12 1036 > > TCP upload BE1 de.starlink.taht.net::tcp_pacing_rate : = 0.35 0.26 1042 > > TCP upload BE1 de.starlink.taht.net::tcp_rtt : 1= 128.57 954.47 1020 > > TCP upload BE1 de.starlink.taht.net::tcp_rtt_var : = 68.22 50.08 1020 > > TCP upload BE2 london.starlink.taht.net : = 0.12 0.15 Mbits/s 1383 > > TCP upload BE2 london.starlink.taht.net::tcp_cwnd : = 17.33 11.00 1046 > > TCP upload BE2 london.starlink.taht.net::tcp_delivery_rate : = 0.12 0.11 1041 > > TCP upload BE2 london.starlink.taht.net::tcp_pacing_rate : = 0.38 0.26 1044 > > TCP upload BE2 london.starlink.taht.net::tcp_rtt : 1= 166.74 991.48 1043 > > TCP upload BE2 london.starlink.taht.net::tcp_rtt_var : = 79.99 68.27 1043 > > TCP upload BE3 singapore.starlink.taht.net : = 0.07 0.13 Mbits/s 1383 > > TCP upload BE3 singapore.starlink.taht.net::tcp_cwnd : = 13.60 9.00 1042 > > TCP upload BE3 singapore.starlink.taht.net::tcp_delivery_rate : = 0.07 0.08 1013 > > TCP upload BE3 singapore.starlink.taht.net::tcp_pacing_rate : = 0.23 0.20 1031 > > TCP upload BE3 singapore.starlink.taht.net::tcp_rtt : 1= 302.23 1129.07 967 > > TCP upload BE3 singapore.starlink.taht.net::tcp_rtt_var : = 113.67 98.84 967 > > TCP upload BE4 fremont.starlink.taht.net : = 0.11 0.20 Mbits/s 1383 > > TCP upload BE4 fremont.starlink.taht.net::tcp_cwnd : = 17.55 11.00 1038 > > TCP upload BE4 fremont.starlink.taht.net::tcp_delivery_rate : = 0.11 0.10 1035 > > TCP upload BE4 fremont.starlink.taht.net::tcp_pacing_rate : = 0.32 0.23 1036 > > TCP upload BE4 fremont.starlink.taht.net::tcp_rtt : 1= 094.64 887.54 1033 > > TCP upload BE4 fremont.starlink.taht.net::tcp_rtt_var : = 67.20 59.16 1033 > > TCP upload avg : = 0.10 N/A Mbits/s 1383 > > TCP upload fairness : = 0.96 N/A Mbits/s 1383 > > TCP upload sum : = 0.42 N/A Mbits/s 1383 > > > > ########## > > with smart queue (fq_codel) on the EdgeRouter > > > > flent -x --socket-stats --step-size=3D.05 -H de.starlink.taht.net -H > > london.starlink.taht.net -H singapore.starlink.taht.net -H fremont.star= link.taht.net -t fqcodel_erx_rttfair4be_8Mbpsdown_500Kbpsup_centurylinkdsl = rtt_fair4be Started Flent 2.0.0 using Python 3.9.2. > > Starting rtt_fair4be test. Expected run time: 70 seconds. > > Data file written to > > ./rtt_fair4be-2022-04-17T150339.549817.fqcodel_erx_rttfair4be_8Mbpsdow > > n_500Kbpsup_centurylinkdsl.flent.gz > > > > Summary of rtt_fair4be test run from 2022-04-17 14:03:39.549817 > > Title: 'fqcodel_erx_rttfair4be_8Mbpsdown_500Kbpsup_centurylinkdsl' > > > > = avg median # data pts > > Ping (ms) ICMP1 de.starlink.taht.net : = 392.93 208.00 ms 1384 > > Ping (ms) ICMP2 london.starlink.taht.net : = 373.94 196.00 ms 1384 > > Ping (ms) ICMP3 singapore.starlink.taht.net : = 508.15 319.00 ms 1384 > > Ping (ms) ICMP4 fremont.starlink.taht.net : = 346.23 152.00 ms 1384 > > Ping (ms) avg : = 405.31 N/A ms 1384 > > TCP download BE1 de.starlink.taht.net : = 1.36 1.62 Mbits/s 1384 > > TCP download BE2 london.starlink.taht.net : = 1.47 1.63 Mbits/s 1384 > > TCP download BE3 singapore.starlink.taht.net : = 1.55 1.60 Mbits/s 1384 > > TCP download BE4 fremont.starlink.taht.net : = 1.50 1.80 Mbits/s 1384 > > TCP download avg : = 1.47 N/A Mbits/s 1384 > > TCP download fairness : = 1.00 N/A Mbits/s 1384 > > TCP download sum : = 5.88 N/A Mbits/s 1384 > > TCP upload BE1 de.starlink.taht.net : = 0.09 0.85 Mbits/s 1384 > > TCP upload BE1 de.starlink.taht.net::tcp_cwnd : = 4.50 4.00 1035 > > TCP upload BE1 de.starlink.taht.net::tcp_delivery_rate : = 0.08 0.08 1030 > > TCP upload BE1 de.starlink.taht.net::tcp_pacing_rate : = 0.17 0.15 1032 > > TCP upload BE1 de.starlink.taht.net::tcp_rtt : = 458.11 318.82 1027 > > TCP upload BE1 de.starlink.taht.net::tcp_rtt_var : = 95.76 61.88 1027 > > TCP upload BE2 london.starlink.taht.net : = 0.09 0.93 Mbits/s 1384 > > TCP upload BE2 london.starlink.taht.net::tcp_cwnd : = 4.17 4.00 1038 > > TCP upload BE2 london.starlink.taht.net::tcp_delivery_rate : = 0.08 0.08 1033 > > TCP upload BE2 london.starlink.taht.net::tcp_pacing_rate : = 0.17 0.14 1036 > > TCP upload BE2 london.starlink.taht.net::tcp_rtt : = 452.67 322.49 1033 > > TCP upload BE2 london.starlink.taht.net::tcp_rtt_var : = 96.87 64.28 1033 > > TCP upload BE3 singapore.starlink.taht.net : = 0.06 0.44 Mbits/s 1384 > > TCP upload BE3 singapore.starlink.taht.net::tcp_cwnd : = 4.16 4.00 1040 > > TCP upload BE3 singapore.starlink.taht.net::tcp_delivery_rate : = 0.06 0.06 1031 > > TCP upload BE3 singapore.starlink.taht.net::tcp_pacing_rate : = 0.12 0.11 1036 > > TCP upload BE3 singapore.starlink.taht.net::tcp_rtt : = 577.75 438.61 1027 > > TCP upload BE3 singapore.starlink.taht.net::tcp_rtt_var : = 107.05 72.97 1027 > > TCP upload BE4 fremont.starlink.taht.net : = 0.10 1.48 Mbits/s 1384 > > TCP upload BE4 fremont.starlink.taht.net::tcp_cwnd : = 4.45 4.00 1020 > > TCP upload BE4 fremont.starlink.taht.net::tcp_delivery_rate : = 0.09 0.09 1017 > > TCP upload BE4 fremont.starlink.taht.net::tcp_pacing_rate : = 0.21 0.16 1019 > > TCP upload BE4 fremont.starlink.taht.net::tcp_rtt : = 439.05 285.86 1018 > > TCP upload BE4 fremont.starlink.taht.net::tcp_rtt_var : = 81.90 56.33 1018 > > TCP upload avg : = 0.08 N/A Mbits/s 1384 > > TCP upload fairness : = 0.97 N/A Mbits/s 1384 > > TCP upload sum : = 0.34 N/A Mbits/s 1384 > > > > > > EdgeRouter Settings > > > > smart-queue DSL { > > download { > > ecn enable > > flows 1024 > > fq-quantum 300 > > limit 10240 > > rate 8mbit > > } > > upload { > > ecn enable > > flows 1024 > > fq-quantum 300 > > limit 10240 > > rate 600kbit > > target 15ms > > } > > wan-interface pppoe0 > > > > Also, here's the VMs qdisc settings. Which I now realize probably > > aren't the best > > > > sysctl -a | grep qdisc > > net.core.default_qdisc =3D pfifo_fast > > > > tc -s qdisc > > qdisc noqueue 0: dev lo root refcnt 2 > > Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b > > 0p requeues 0 qdisc mq 0: dev eth0 root Sent 628201 bytes 8643 pkt > > (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc > > pfifo_fast 0: dev eth0 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 > > 1 1 1 1 1 Sent 628201 bytes 8643 pkt (dropped 0, overlimits 0 > > requeues 0) backlog 0b 0p requeues 0 > > > > > > > > -Jordan > > > > -----Original Message----- > > From: Starlink On Behalf Of > > Dave Taht > > Sent: Saturday, April 16, 2022 9:40 PM > > To: flent-users ; bloat > > ; starlink@lists.bufferbloat.net > > Subject: [Starlink] an rtt_fair test? > > > > if anyone has a bit of spare time this holiday, I'd like to collect a f= ew results from various networks around the web this week, not just on the = turris. Example command line and outputs here: > > > > https://forum.turris.cz/t/sqm-on-turris-flent-benchmarks/17048/ > > > > -- > > I tried to build a better future, a few times: > > https://wayforward.archive.org/?site=3Dhttps%3A%2F%2Fwww.icei.org > > > > Dave T=C3=A4ht CEO, TekLibre, LLC > > _______________________________________________ > > Starlink mailing list > > Starlink@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/starlink > > > > > -- > I tried to build a better future, a few times: > https://wayforward.archive.org/?site=3Dhttps%3A%2F%2Fwww.icei.org > > Dave T=C3=A4ht CEO, TekLibre, LLC > --=20 I tried to build a better future, a few times: https://wayforward.archive.org/?site=3Dhttps%3A%2F%2Fwww.icei.org Dave T=C3=A4ht CEO, TekLibre, LLC