From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) by mail.toke.dk (Postfix) with ESMTPS id 1669B7C6DED for ; Thu, 7 Jan 2021 23:22:42 +0100 (CET) Authentication-Results: mail.toke.dk; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=bOFvR0pQ Received: by mail-ed1-x529.google.com with SMTP id h16so9246847edt.7 for ; Thu, 07 Jan 2021 14:22:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=dwh1UI+U7sjy4NM4xF+ZniCagOSGmv5x7DaMOMUeqiM=; b=bOFvR0pQtDvPg4Y1SJbXCSbGVDIvLa/a5kYjPT5HbA6UAscw1s7ysMC8G4Xvr2J9y2 MyU7WSUVCAgJs3bgPCadlH0FnrwoacaQdkxVJLxhIkLlasge1ArnIQ7ZgaHafvp/pAff E+OVdijctHCPC2ZMD6JipSy95sWfy6bDTfyqw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=dwh1UI+U7sjy4NM4xF+ZniCagOSGmv5x7DaMOMUeqiM=; b=iN9CM4Oz1kP3wxoG+ElwwwI97kOVrMFFq/MEIKoISDgbnG6SRyajKN9aU1dEYQBK5T EPvuqyHv88lzXIfSebNkyZrHuCGQOu7jOjT9aCOtB+T0ld6S24yK/HcWZ70JAQGA2C6v pGMe1gjyaxaGuRjSEyYaAtearFS/nLr4P6TY/DSMvcY3sGwV1rcKrnWR4sapZOKZwnAl SS/V4Px3Gz8zxmlHyxTGId4JH1ldKot8tw7+R0Cpe3qJ1wTd6179obWSS9eMEdDVSmgv 2iG+eExHBp5oBIgHTVEwzMtP3PADqsTRSGmoA0eCZjjJJn5qtVjVnPdPQcaVh9wTnySn KN9w== X-Gm-Message-State: AOAM533TxsMGFP9tU8GxkpwHa93NPMgsS2DYhIc+QCSEJBpkRaK/sSvY aWeBrwC7ZDraxOpxuk77DGhKiyLup3ouJSylxOldiwOm6E7BkKFYXWleFCysA7LBbNR4AEiox49 M+HtjQdo2szCw2kT7EGp5sw== X-Google-Smtp-Source: ABdhPJzvJ7oSiTGmeAKRLU1tAQV4DdjUEjr53CBTcwrc8pGkfW3CIw04oh1904YgmOOmsloA1GA5R0ii2JrvpyIEbhw= X-Received: by 2002:a50:ed04:: with SMTP id j4mr3313241eds.84.1610058156287; Thu, 07 Jan 2021 14:22:36 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Bob McMahon Date: Thu, 7 Jan 2021 14:22:25 -0800 Message-ID: To: Dave Taht Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="0000000000008182d905b856e069" X-MailFrom: bob.mcmahon@broadcom.com X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation Message-ID-Hash: Z73Y7TNRJ7GVYTZ23VJH3F6OS43SFVPC X-Message-ID-Hash: Z73Y7TNRJ7GVYTZ23VJH3F6OS43SFVPC X-Mailman-Approved-At: Fri, 08 Jan 2021 12:34:35 +0100 CC: Taran Lynn , ghosal@cs.ucdavis.edu, BBR Development , Make-Wifi-fast , ECN-Sane , bloat , codel@lists.bufferbloat.net, flent-users X-Mailman-Version: 3.3.2 Precedence: list Subject: [Flent-users] Re: [Make-wifi-fast] D* tcp looks pretty good, on paper List-Id: Flent discussion list Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: --0000000000008182d905b856e069 Content-Type: multipart/alternative; boundary="0000000000007987bf05b856e0da" --0000000000007987bf05b856e0da Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Could be interesting to use an isochronous traffic profile as well as take histograms, using the actual end/end frame latency instead of RTT. This does require clock sync. CLIENT SPECIFIC OPTIONS*--isochronous[=3D**fps*:*mean*,*stdev**]*send isochronous traffic with frequency frames per second and load defined by mean and standard deviation using a log normal distribution, defaults to 60:20m,0. (Note: Here the suffixes indicate bytes/sec or bits/sec per use of uppercase or lowercase, respectively. Also the p suffix is supported to set the burst size in packets, e.g. isochronous=3D2:25p will send two 25 packet bursts every second, or one 25 packet burst every 0.5 seconds.)SERVE= R SPECIFIC OPTIONS*--histograms[=3D**binwidth*[u],*bincount*,[*lowerci*],[ *upperci*]]enable latency histograms for udp packets (-u), for tcp writes (with --trip-times), or for either udp or tcp with --isochronous clients. The binning can be modified. Bin widths (default 1 millisecond, append u for microseconds,) bincount is total bins (default 1000), ci is confidence interval between 0-100% (default lower 5%, upper 95%, 3 stdev 99.7%)Bob On Thu, Jan 7, 2021 at 1:41 PM Dave Taht wrote: > This is through one of the last remaining cerowrt boxes in the world, > running fq_codel. tcp-davis takes about a 20% single stream throughput > hit vs bbr. > > I note, that I don't care one whit about throughput anymore. I care > that nothing, NOTHING messes up my videoconference... > > and thus the tcp-rtt stats attached for davis are pleasing. > > On Thu, Jan 7, 2021 at 12:26 PM Bob McMahon > wrote: > > > > FYI, one can try this out using iperf 2.1 with --trip-times. This give= s > end/end delay at the application level. One can use --trip-times when clo= cks > > are sync'd to get the write to read latencies which are the latencies a= t > the application level. > > > > Note: I set up a Raspberry Pi 4 with a GPS hat from ubutronics for > solderless pulse per second. Then configured it as a PTP grandmaster. > This cost me around $200. > > > > I also added support for a very crude --near-congestion option that > paces the writes based upon the weight of the RTT. The tcp_info struct i= s > sampled and available > > for other experiments though one would have to modify the source a bit. > This current technique used by iperf 2.1 is designed for test networks on= ly > where all > > traffic is under script control. We've had too many people measuring > bloat as latency. We really need separate measurements between the two > phenomena, > > bloat vs latency, because they require different eng actions for a > semiconductor supplier. > > > > Below are examples over a 10G link, first with no write pacing then wit= h > it. The server output, shown first, has the latency data (as well as the > net power > > and little's law calculation.) (Note: use --histograms for to get full > distributions.) > > > > No write pacing > > > > [rjmcmahon@localhost iperf2-code]$ src/iperf -s -i 1 -e > > ------------------------------------------------------------ > > Server listening on TCP port 5001 with pid 24568 > > Read buffer size: 128 KByte (Dist bin width=3D16.0 KByte) > > TCP window size: 85.3 KByte (default) > > ------------------------------------------------------------ > > [ 1] local 192.168.1.10%enp2s0 port 5001 connected with 192.168.1.62 > port 50056 (MSS=3D1448) (trip-times) (sock=3D4) (peer 2.1.0-rc) on 2021-0= 1-07 > 12:11:04 (PST) > > [ ID] Interval Transfer Bandwidth Burst Latency > avg/min/max/stdev (cnt/size) inP NetPwr Reads=3DDist > > [ 1] 0.00-1.00 sec 1.09 GBytes 9.34 Gbits/sec > 2.959/1.180/3.681/0.388 ms (8905/131072) 3.31 MByte 394522 > 18480=3D2459:2580:2475:2354:2203:2192:1974:2243 > > [ 1] 1.00-2.00 sec 1.10 GBytes 9.41 Gbits/sec > 2.993/2.302/3.703/0.359 ms (8978/131072) 3.36 MByte 393209 > 19482=3D2526:2850:3102:2622:2344:2297:1867:1874 > > [ 1] 2.00-3.00 sec 1.10 GBytes 9.42 Gbits/sec > 3.010/2.302/3.692/0.347 ms (8978/131085) 3.38 MByte 391047 > 19387=3D2563:2757:2928:2708:2432:2244:1829:1926 > > [ 1] 3.00-4.00 sec 1.10 GBytes 9.41 Gbits/sec > 3.009/2.301/3.668/0.348 ms (8979/131060) 3.38 MByte 391094 > 18821=3D2456:2585:2660:2545:2270:2239:1906:2160 > > [ 1] 4.00-5.00 sec 1.10 GBytes 9.42 Gbits/sec > 2.985/2.299/3.696/0.359 ms (8979/131070) 3.35 MByte 394295 > 19441=3D2509:2886:2959:2728:2336:2200:1971:1852 > > [ 1] 5.00-6.00 sec 1.10 GBytes 9.41 Gbits/sec > 2.977/2.258/3.671/0.363 ms (8978/131082) 3.34 MByte 395352 > 18509=3D2352:2602:2464:2380:2263:2142:2095:2211 > > [ 1] 6.00-7.00 sec 1.10 GBytes 9.41 Gbits/sec > 2.980/2.290/3.680/0.363 ms (8978/131072) 3.34 MByte 394873 > 18522=3D2407:2499:2565:2334:2213:2268:1999:2237 > > [ 1] 7.00-8.00 sec 1.10 GBytes 9.42 Gbits/sec > 2.980/2.253/3.702/0.362 ms (8979/131073) 3.35 MByte 394972 > 18615=3D2427:2592:2493:2460:2281:2057:2062:2243 > > [ 1] 8.00-9.00 sec 1.10 GBytes 9.41 Gbits/sec > 2.976/2.277/3.663/0.364 ms (8979/131065) 3.34 MByte 395443 > 18632=3D2338:2615:2647:2351:2192:2317:2063:2109 > > [ 1] 9.00-10.00 sec 1.10 GBytes 9.41 Gbits/sec > 2.976/2.293/3.690/0.366 ms (8978/131076) 3.34 MByte 395416 > 18428=3D2281:2622:2497:2275:2178:2253:2129:2193 > > [ 1] 0.00-10.00 sec 11.0 GBytes 9.41 Gbits/sec > 2.984/1.180/3.703/0.362 ms (89736/131072) 3.35 MByte 394014 > 188367=3D24320:26609:26793:24757:22712:22211:19916:21049 > > > > > > [rjmcmahon@localhost iperf2-code]src/iperf -c 192.168.1.10 --trip-times > -i 1 -e > > ------------------------------------------------------------ > > Client connecting to 192.168.1.10, TCP port 5001 with pid 18961 (1 flow= s) > > Write buffer size: 131072 Byte > > TCP window size: 85.0 KByte (default) > > ------------------------------------------------------------ > > [ 1] local 192.168.1.62%enp2s0 port 50056 connected with 192.168.1.10 > port 5001 (MSS=3D1448) (trip-times) (sock=3D3) (ct=3D0.41 ms) on 2021-01-= 07 > 12:11:04 (PST) > > [ ID] Interval Transfer Bandwidth Write/Err Rtry > Cwnd/RTT NetPwr > > [ 1] 0.00-1.00 sec 1.09 GBytes 9.37 Gbits/sec 8937/0 0 > 1508K/1099 us 1065750 > > [ 1] 1.00-2.00 sec 1.10 GBytes 9.41 Gbits/sec 8975/0 0 > 1508K/1087 us 1082218 > > [ 1] 2.00-3.00 sec 1.10 GBytes 9.41 Gbits/sec 8975/0 0 > 1508K/1081 us 1088225 > > [ 1] 3.00-4.00 sec 1.10 GBytes 9.42 Gbits/sec 8984/0 0 > 1508K/1085 us 1085300 > > [ 1] 4.00-5.00 sec 1.10 GBytes 9.42 Gbits/sec 8980/0 0 > 1508K/1105 us 1065182 > > [ 1] 5.00-6.00 sec 1.10 GBytes 9.41 Gbits/sec 8975/0 0 > 1582K/1100 us 1069428 > > [ 1] 6.00-7.00 sec 1.10 GBytes 9.42 Gbits/sec 8979/0 0 > 1582K/1121 us 1049862 > > [ 1] 7.00-8.00 sec 1.10 GBytes 9.41 Gbits/sec 8976/0 0 > 1582K/1133 us 1038396 > > [ 1] 8.00-9.00 sec 1.10 GBytes 9.41 Gbits/sec 8978/0 0 > 1582K/1115 us 1055394 > > [ 1] 9.00-10.00 sec 1.10 GBytes 9.42 Gbits/sec 8986/0 0 > 1582K/1122 us 1049744 > > [ 1] 0.00-10.00 sec 11.0 GBytes 9.41 Gbits/sec 89748/0 0 > 1582K/1122 us 1048294 > > > > > > With write pacing > > > > [rjmcmahon@localhost iperf2-code]$ src/iperf -s -i 1 -e > > ------------------------------------------------------------ > > Server listening on TCP port 5001 with pid 24702 > > Read buffer size: 128 KByte (Dist bin width=3D16.0 KByte) > > TCP window size: 85.3 KByte (default) > > ------------------------------------------------------------ > > [ 1] local 192.168.1.10%enp2s0 port 5001 connected with 192.168.1.62 > port 50072 (MSS=3D1448) (trip-times) (sock=3D4) (peer 2.1.0-rc) on 2021-0= 1-07 > 12:14:59 (PST) > > [ ID] Interval Transfer Bandwidth Burst Latency > avg/min/max/stdev (cnt/size) inP NetPwr Reads=3DDist > > [ 1] 0.00-1.00 sec 1.08 GBytes 9.31 Gbits/sec > 0.401/0.193/2.682/0.168 ms (8876/131084) 456 KByte 2904347 > 19868=3D3296:2404:2508:2797:3559:1778:1551:1975 > > [ 1] 1.00-2.00 sec 1.10 GBytes 9.41 Gbits/sec > 0.400/0.219/0.627/0.053 ms (8971/131071) 460 KByte 2937822 > 19117=3D3069:2267:2307:2510:3029:1824:1683:2428 > > [ 1] 2.00-3.00 sec 1.09 GBytes 9.39 Gbits/sec > 0.374/0.193/0.541/0.055 ms (8958/131060) 428 KByte 3143030 > 18942=3D2846:2423:2304:2417:2927:1831:1856:2338 > > [ 1] 3.00-4.00 sec 1.09 GBytes 9.39 Gbits/sec > 0.385/0.190/0.664/0.070 ms (8952/131072) 441 KByte 3050401 > 19248=3D3041:2175:2343:2749:3320:1805:1526:2289 > > [ 1] 4.00-5.00 sec 1.09 GBytes 9.40 Gbits/sec > 0.380/0.197/0.546/0.057 ms (8965/131075) 436 KByte 3095915 > 19959=3D3321:2398:2551:2738:3500:1840:1532:2079 > > [ 1] 5.00-6.00 sec 1.09 GBytes 9.39 Gbits/sec > 0.369/0.198/0.536/0.051 ms (8956/131072) 423 KByte 3177431 > 21060=3D3627:2456:2886:3189:4246:1813:1190:1653 > > [ 1] 6.00-7.00 sec 1.09 GBytes 9.39 Gbits/sec > 0.380/0.202/0.562/0.054 ms (8959/131077) 436 KByte 3086914 > 19263=3D3044:2338:2424:2505:3155:1809:1636:2352 > > [ 1] 7.00-8.00 sec 1.09 GBytes 9.40 Gbits/sec > 0.376/0.198/0.541/0.053 ms (8965/131061) 432 KByte 3122495 > 19137=3D3079:2303:2340:2455:3017:1822:1683:2438 > > [ 1] 8.00-9.00 sec 1.10 GBytes 9.41 Gbits/sec > 0.381/0.208/0.576/0.054 ms (8974/131073) 438 KByte 3083767 > 19162=3D3050:2269:2392:2486:3019:1891:1667:2388 > > [ 1] 9.00-10.00 sec 1.09 GBytes 9.40 Gbits/sec > 0.371/0.194/0.582/0.057 ms (8964/131070) 425 KByte 3169244 > 19143=3D3006:2411:2303:2462:3067:1744:1760:2390 > > [ 1] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec > 0.382/0.190/2.682/0.076 ms (89544/131072) 437 KByte 3074913 > 194908=3D31380:23444:24362:26308:32839:18161:16084:22330 > > > > > > [rjmcmahon@localhost iperf2-code]$ src/iperf -c 192.168.1.10 > --near-congestion=3D0.05 --trip-times -i 1 -e > > ------------------------------------------------------------ > > Client connecting to 192.168.1.10, TCP port 5001 with pid 19320 (1 flow= s) > > Write buffer size: 131072 Byte > > TCP near-congestion delay weight set to 0.0500 > > TCP window size: 85.0 KByte (default) > > ------------------------------------------------------------ > > [ 1] local 192.168.1.62%enp2s0 port 50072 connected with 192.168.1.10 > port 5001 (MSS=3D1448) (trip-times) (sock=3D3) (ct=3D0.40 ms) on 2021-01-= 07 > 12:14:59 (PST) > > [ ID] Interval Transfer Bandwidth Write/Err Rtry > Cwnd/RTT NetPwr > > [ 1] 0.00-1.00 sec 1.08 GBytes 9.31 Gbits/sec 8881/0 0 > 1135K/373 us 3120427 > > [ 1] 1.00-2.00 sec 1.10 GBytes 9.41 Gbits/sec 8971/0 0 > 1135K/391 us 3007281 > > [ 1] 2.00-3.00 sec 1.09 GBytes 9.39 Gbits/sec 8958/0 0 > 1135K/331 us 3547260 > > [ 1] 3.00-4.00 sec 1.09 GBytes 9.39 Gbits/sec 8952/0 0 > 1135K/288 us 4074155 > > [ 1] 4.00-5.00 sec 1.09 GBytes 9.40 Gbits/sec 8965/0 0 > 1135K/301 us 3903855 > > [ 1] 5.00-6.00 sec 1.09 GBytes 9.39 Gbits/sec 8955/0 0 > 1135K/414 us 2835144 > > [ 1] 6.00-7.00 sec 1.09 GBytes 9.40 Gbits/sec 8961/0 0 > 1135K/470 us 2499013 > > [ 1] 7.00-8.00 sec 1.09 GBytes 9.40 Gbits/sec 8964/0 0 > 1135K/350 us 3356941 > > [ 1] 8.00-9.00 sec 1.10 GBytes 9.41 Gbits/sec 8973/0 0 > 1135K/472 us 2491756 > > [ 1] 9.00-10.00 sec 1.09 GBytes 9.40 Gbits/sec 8964/0 0 > 1135K/402 us 2922710 > > [ 1] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec 89547/0 0 > 1135K/402 us 2919642 > > > > Bob > > > > > > On Thu, Jan 7, 2021 at 11:22 AM Taran Lynn via Make-wifi-fast < > make-wifi-fast@lists.bufferbloat.net> wrote: > >> > >> The source can be found at https://github.com/lambda-11235/tcp_davis . > >> > >> The code mentioned in the paper can be found under the tag > "arxiv_2020". The current master branch has an additional stable mode tha= t > I was testing out. > >> > >> > >> On Thu, Jan 7, 2021 at 10:35 AM Dave Taht wrote: > >>> > >>> See: https://arxiv.org/pdf/2012.14996.pdf > >>> > >>> Things I really like: > >>> > >>> * they used flent > >>> * Using "variance" as the principal signal. This is essentially one o= f > >>> the great unpublished and unanalyzed improvements on the minstrel > >>> algorithm as well > >>> * Conventional ecn response > >>> * outperforms bbr on variable links > >>> > >>> Only negative so far is I haven't found any published source to it. := ( > >>> > >>> Otherwise a very promising start to a year. > >>> > >>> "The choice of feedback mechanism between delay and packet loss has > >>> long been a point of contention in TCP congestion control. This has > >>> partly been resolved, as it has become increasingly evident that dela= y > >>> based methods are needed to facilitate modern interactive web > >>> applications. However, what has not been resolved is what control > >>> should be used, with the two candidates being the congestion window > >>> and the pacing rate. BBR is a new delay based congestion control > >>> algorithm that uses a pacing rate as its primary control and the > >>> congestion window as a secondary control. We propose that a congestio= n > >>> window first algorithm might give more desirable performance > >>> characteristics in situations where latency must be minimized even at > >>> the expense of some loss in throughput. To evaluate this hypothesis w= e > >>> introduce a new congestion control algorithm called TCP D*, which is = a > >>> congestion window first algorithm that adopts BBR's approach of > >>> maximizing delivery rate while minimizing latency. In this paper, we > >>> discuss the key features of this algorithm, discuss the differences > >>> and similarity to BBR, and present some preliminary results based on = a > >>> real implementation." > >>> > >>> > >>> > >>> > >>> -- > >>> "For a successful technology, reality must take precedence over publi= c > >>> relations, for Mother Nature cannot be fooled" - Richard Feynman > >>> > >>> dave@taht.net CTO, TekLibre, LLC Tel: 1-831-435-0729 > >> > >> _______________________________________________ > >> Make-wifi-fast mailing list > >> Make-wifi-fast@lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/make-wifi-fast > > > > > > This electronic communication and the information and any files > transmitted with it, or attached to it, are confidential and are intended > solely for the use of the individual or entity to whom it is addressed an= d > may contain information that is confidential, legally privileged, protect= ed > by privacy laws, or otherwise restricted from disclosure to anyone else. = If > you are not the intended recipient or the person responsible for deliveri= ng > the e-mail to the intended recipient, you are hereby notified that any us= e, > copying, distributing, dissemination, forwarding, printing, or copying of > this e-mail is strictly prohibited. If you received this e-mail in error, > please return the e-mail to the sender, delete it from your computer, and > destroy any printed copy of it. > > > > -- > "For a successful technology, reality must take precedence over public > relations, for Mother Nature cannot be fooled" - Richard Feynman > > dave@taht.net CTO, TekLibre, LLC Tel: 1-831-435-0729 > --=20 This electronic communication and the information and any files transmitted= =20 with it, or attached to it, are confidential and are intended solely for=20 the use of the individual or entity to whom it is addressed and may contain= =20 information that is confidential, legally privileged, protected by privacy= =20 laws, or otherwise restricted from disclosure to anyone else. If you are=20 not the intended recipient or the person responsible for delivering the=20 e-mail to the intended recipient, you are hereby notified that any use,=20 copying, distributing, dissemination, forwarding, printing, or copying of= =20 this e-mail is strictly prohibited. If you received this e-mail in error,= =20 please return the e-mail to the sender, delete it from your computer, and= =20 destroy any printed copy of it. --0000000000007987bf05b856e0da Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Could be interesting to use an isochronous traffic profile= as well as take histograms, using the actual end/end frame latency instead= of RTT. This does require clock sync.

CLIENT SPECIFIC OPTIONS

--isochronous[=3Dfps:mean,stdev= ]
send isochronous traffic with frequency frames per second = and load defined by mean and standard deviation using a log normal distribu= tion, defaults to 60:20m,0. (Note: Here the suffixes indicate bytes/sec or = bits/sec per use of uppercase or lowercase, respectively. Also the p suffix= is supported to set the burst size in packets, e.g. isochronous=3D2:25p wi= ll send two 25 packet bursts every second, or one 25 packet burst every 0.5= seconds.)

SERVER SPECIFIC OPTIONS

--h= istograms[=3Dbinwidth[u],bincount,[lowerci],[up= perci]]
enable latency histograms for udp packets (-u), for tcp= writes (with --trip-times), or for either udp or tcp with --isochronous cl= ients. The binning can be modified. Bin widths (default 1 millisecond, appe= nd u for microseconds,) bincount is total bins (default 1000), ci is confid= ence interval between 0-100% (default lower 5%, upper 95%, 3 stdev 99.7%)
Bob

On Thu, Jan 7, 2021 at 1:41 PM Dave Taht <dave.taht@gmail.com> wrote:
This is through one of the last rema= ining cerowrt boxes in the world,
running fq_codel. tcp-davis takes about a 20% single stream throughput
hit vs bbr.

I note, that I don't care one whit about throughput anymore. I care
that nothing, NOTHING messes up my videoconference...

and thus the tcp-rtt stats attached for davis are pleasing.

On Thu, Jan 7, 2021 at 12:26 PM Bob McMahon <bob.mcmahon@broadcom.com> wrote:<= br> >
> FYI, one can try this out using iperf 2.1 with --trip-times.=C2=A0 Thi= s gives end/end delay at the application level. One can use --trip-times wh= en clocks
> are sync'd to get the write to read latencies which are the latenc= ies at the application level.
>
> Note: I set up a Raspberry Pi 4 with a GPS hat from ubutronics for sol= derless pulse per second.=C2=A0 Then configured it as a PTP grandmaster.=C2= =A0 This cost me around $200.
>
> I also added support for a very crude --near-congestion option that pa= ces the writes based upon the weight of the RTT.=C2=A0 The tcp_info struct = is sampled and available
> for other experiments though one would have to modify the source a bit= . This current technique used by iperf 2.1 is designed for test networks on= ly where all
> traffic is under script control. We've had too many people measuri= ng bloat as latency. We really need separate measurements between the two p= henomena,
> bloat vs latency, because they require different eng actions for a sem= iconductor supplier.
>
> Below are examples over a 10G link, first with no write pacing then wi= th it. The server output, shown first, has the latency data (as well as the= net power
> and little's law calculation.)=C2=A0 (Note: use --histograms for t= o get full distributions.)
>
> No write pacing
>
> [rjmcmahon@localhost iperf2-code]$ src/iperf -s -i 1 -e
> ------------------------------------------------------------
> Server listening on TCP port 5001 with pid 24568
> Read buffer size:=C2=A0 128 KByte (Dist bin width=3D16.0 KByte)
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> [=C2=A0 1] local 192.168.1.10%enp2s0 port 5001 connected with 192.168.= 1.62 port 50056 (MSS=3D1448) (trip-times) (sock=3D4) (peer 2.1.0-rc) on 202= 1-01-07 12:11:04 (PST)
> [ ID] Interval=C2=A0 =C2=A0 =C2=A0 =C2=A0 Transfer=C2=A0 =C2=A0 Bandwi= dth=C2=A0 =C2=A0 Burst Latency avg/min/max/stdev (cnt/size) inP NetPwr=C2= =A0 Reads=3DDist
> [=C2=A0 1] 0.00-1.00 sec=C2=A0 1.09 GBytes=C2=A0 9.34 Gbits/sec=C2=A0 = 2.959/1.180/3.681/0.388 ms (8905/131072) 3.31 MByte 394522=C2=A0 18480=3D24= 59:2580:2475:2354:2203:2192:1974:2243
> [=C2=A0 1] 1.00-2.00 sec=C2=A0 1.10 GBytes=C2=A0 9.41 Gbits/sec=C2=A0 = 2.993/2.302/3.703/0.359 ms (8978/131072) 3.36 MByte 393209=C2=A0 19482=3D25= 26:2850:3102:2622:2344:2297:1867:1874
> [=C2=A0 1] 2.00-3.00 sec=C2=A0 1.10 GBytes=C2=A0 9.42 Gbits/sec=C2=A0 = 3.010/2.302/3.692/0.347 ms (8978/131085) 3.38 MByte 391047=C2=A0 19387=3D25= 63:2757:2928:2708:2432:2244:1829:1926
> [=C2=A0 1] 3.00-4.00 sec=C2=A0 1.10 GBytes=C2=A0 9.41 Gbits/sec=C2=A0 = 3.009/2.301/3.668/0.348 ms (8979/131060) 3.38 MByte 391094=C2=A0 18821=3D24= 56:2585:2660:2545:2270:2239:1906:2160
> [=C2=A0 1] 4.00-5.00 sec=C2=A0 1.10 GBytes=C2=A0 9.42 Gbits/sec=C2=A0 = 2.985/2.299/3.696/0.359 ms (8979/131070) 3.35 MByte 394295=C2=A0 19441=3D25= 09:2886:2959:2728:2336:2200:1971:1852
> [=C2=A0 1] 5.00-6.00 sec=C2=A0 1.10 GBytes=C2=A0 9.41 Gbits/sec=C2=A0 = 2.977/2.258/3.671/0.363 ms (8978/131082) 3.34 MByte 395352=C2=A0 18509=3D23= 52:2602:2464:2380:2263:2142:2095:2211
> [=C2=A0 1] 6.00-7.00 sec=C2=A0 1.10 GBytes=C2=A0 9.41 Gbits/sec=C2=A0 = 2.980/2.290/3.680/0.363 ms (8978/131072) 3.34 MByte 394873=C2=A0 18522=3D24= 07:2499:2565:2334:2213:2268:1999:2237
> [=C2=A0 1] 7.00-8.00 sec=C2=A0 1.10 GBytes=C2=A0 9.42 Gbits/sec=C2=A0 = 2.980/2.253/3.702/0.362 ms (8979/131073) 3.35 MByte 394972=C2=A0 18615=3D24= 27:2592:2493:2460:2281:2057:2062:2243
> [=C2=A0 1] 8.00-9.00 sec=C2=A0 1.10 GBytes=C2=A0 9.41 Gbits/sec=C2=A0 = 2.976/2.277/3.663/0.364 ms (8979/131065) 3.34 MByte 395443=C2=A0 18632=3D23= 38:2615:2647:2351:2192:2317:2063:2109
> [=C2=A0 1] 9.00-10.00 sec=C2=A0 1.10 GBytes=C2=A0 9.41 Gbits/sec=C2=A0= 2.976/2.293/3.690/0.366 ms (8978/131076) 3.34 MByte 395416=C2=A0 18428=3D2= 281:2622:2497:2275:2178:2253:2129:2193
> [=C2=A0 1] 0.00-10.00 sec=C2=A0 11.0 GBytes=C2=A0 9.41 Gbits/sec=C2=A0= 2.984/1.180/3.703/0.362 ms (89736/131072) 3.35 MByte 394014=C2=A0 188367= =3D24320:26609:26793:24757:22712:22211:19916:21049
>
>
> [rjmcmahon@localhost iperf2-code]src/iperf -c 192.168.1.10 --trip-time= s -i 1 -e
> ------------------------------------------------------------
> Client connecting to 192.168.1.10, TCP port 5001 with pid 18961 (1 flo= ws)
> Write buffer size: 131072 Byte
> TCP window size: 85.0 KByte (default)
> ------------------------------------------------------------
> [=C2=A0 1] local 192.168.1.62%enp2s0 port 50056 connected with 192.168= .1.10 port 5001 (MSS=3D1448) (trip-times) (sock=3D3) (ct=3D0.41 ms) on 2021= -01-07 12:11:04 (PST)
> [ ID] Interval=C2=A0 =C2=A0 =C2=A0 =C2=A0 Transfer=C2=A0 =C2=A0 Bandwi= dth=C2=A0 =C2=A0 =C2=A0 =C2=A0Write/Err=C2=A0 Rtry=C2=A0 =C2=A0 =C2=A0Cwnd/= RTT=C2=A0 =C2=A0 =C2=A0 =C2=A0 NetPwr
> [=C2=A0 1] 0.00-1.00 sec=C2=A0 1.09 GBytes=C2=A0 9.37 Gbits/sec=C2=A0 = 8937/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01508K/1099 us= =C2=A0 1065750
> [=C2=A0 1] 1.00-2.00 sec=C2=A0 1.10 GBytes=C2=A0 9.41 Gbits/sec=C2=A0 = 8975/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01508K/1087 us= =C2=A0 1082218
> [=C2=A0 1] 2.00-3.00 sec=C2=A0 1.10 GBytes=C2=A0 9.41 Gbits/sec=C2=A0 = 8975/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01508K/1081 us= =C2=A0 1088225
> [=C2=A0 1] 3.00-4.00 sec=C2=A0 1.10 GBytes=C2=A0 9.42 Gbits/sec=C2=A0 = 8984/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01508K/1085 us= =C2=A0 1085300
> [=C2=A0 1] 4.00-5.00 sec=C2=A0 1.10 GBytes=C2=A0 9.42 Gbits/sec=C2=A0 = 8980/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01508K/1105 us= =C2=A0 1065182
> [=C2=A0 1] 5.00-6.00 sec=C2=A0 1.10 GBytes=C2=A0 9.41 Gbits/sec=C2=A0 = 8975/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01582K/1100 us= =C2=A0 1069428
> [=C2=A0 1] 6.00-7.00 sec=C2=A0 1.10 GBytes=C2=A0 9.42 Gbits/sec=C2=A0 = 8979/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01582K/1121 us= =C2=A0 1049862
> [=C2=A0 1] 7.00-8.00 sec=C2=A0 1.10 GBytes=C2=A0 9.41 Gbits/sec=C2=A0 = 8976/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01582K/1133 us= =C2=A0 1038396
> [=C2=A0 1] 8.00-9.00 sec=C2=A0 1.10 GBytes=C2=A0 9.41 Gbits/sec=C2=A0 = 8978/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01582K/1115 us= =C2=A0 1055394
> [=C2=A0 1] 9.00-10.00 sec=C2=A0 1.10 GBytes=C2=A0 9.42 Gbits/sec=C2=A0= 8986/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01582K/1122 u= s=C2=A0 1049744
> [=C2=A0 1] 0.00-10.00 sec=C2=A0 11.0 GBytes=C2=A0 9.41 Gbits/sec=C2=A0= 89748/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01582K/1122 = us=C2=A0 1048294
>
>
> With write pacing
>
> [rjmcmahon@localhost iperf2-code]$ src/iperf -s -i 1 -e
> ------------------------------------------------------------
> Server listening on TCP port 5001 with pid 24702
> Read buffer size:=C2=A0 128 KByte (Dist bin width=3D16.0 KByte)
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> [=C2=A0 1] local 192.168.1.10%enp2s0 port 5001 connected with 192.168.= 1.62 port 50072 (MSS=3D1448) (trip-times) (sock=3D4) (peer 2.1.0-rc) on 202= 1-01-07 12:14:59 (PST)
> [ ID] Interval=C2=A0 =C2=A0 =C2=A0 =C2=A0 Transfer=C2=A0 =C2=A0 Bandwi= dth=C2=A0 =C2=A0 Burst Latency avg/min/max/stdev (cnt/size) inP NetPwr=C2= =A0 Reads=3DDist
> [=C2=A0 1] 0.00-1.00 sec=C2=A0 1.08 GBytes=C2=A0 9.31 Gbits/sec=C2=A0 = 0.401/0.193/2.682/0.168 ms (8876/131084)=C2=A0 456 KByte 2904347=C2=A0 1986= 8=3D3296:2404:2508:2797:3559:1778:1551:1975
> [=C2=A0 1] 1.00-2.00 sec=C2=A0 1.10 GBytes=C2=A0 9.41 Gbits/sec=C2=A0 = 0.400/0.219/0.627/0.053 ms (8971/131071)=C2=A0 460 KByte 2937822=C2=A0 1911= 7=3D3069:2267:2307:2510:3029:1824:1683:2428
> [=C2=A0 1] 2.00-3.00 sec=C2=A0 1.09 GBytes=C2=A0 9.39 Gbits/sec=C2=A0 = 0.374/0.193/0.541/0.055 ms (8958/131060)=C2=A0 428 KByte 3143030=C2=A0 1894= 2=3D2846:2423:2304:2417:2927:1831:1856:2338
> [=C2=A0 1] 3.00-4.00 sec=C2=A0 1.09 GBytes=C2=A0 9.39 Gbits/sec=C2=A0 = 0.385/0.190/0.664/0.070 ms (8952/131072)=C2=A0 441 KByte 3050401=C2=A0 1924= 8=3D3041:2175:2343:2749:3320:1805:1526:2289
> [=C2=A0 1] 4.00-5.00 sec=C2=A0 1.09 GBytes=C2=A0 9.40 Gbits/sec=C2=A0 = 0.380/0.197/0.546/0.057 ms (8965/131075)=C2=A0 436 KByte 3095915=C2=A0 1995= 9=3D3321:2398:2551:2738:3500:1840:1532:2079
> [=C2=A0 1] 5.00-6.00 sec=C2=A0 1.09 GBytes=C2=A0 9.39 Gbits/sec=C2=A0 = 0.369/0.198/0.536/0.051 ms (8956/131072)=C2=A0 423 KByte 3177431=C2=A0 2106= 0=3D3627:2456:2886:3189:4246:1813:1190:1653
> [=C2=A0 1] 6.00-7.00 sec=C2=A0 1.09 GBytes=C2=A0 9.39 Gbits/sec=C2=A0 = 0.380/0.202/0.562/0.054 ms (8959/131077)=C2=A0 436 KByte 3086914=C2=A0 1926= 3=3D3044:2338:2424:2505:3155:1809:1636:2352
> [=C2=A0 1] 7.00-8.00 sec=C2=A0 1.09 GBytes=C2=A0 9.40 Gbits/sec=C2=A0 = 0.376/0.198/0.541/0.053 ms (8965/131061)=C2=A0 432 KByte 3122495=C2=A0 1913= 7=3D3079:2303:2340:2455:3017:1822:1683:2438
> [=C2=A0 1] 8.00-9.00 sec=C2=A0 1.10 GBytes=C2=A0 9.41 Gbits/sec=C2=A0 = 0.381/0.208/0.576/0.054 ms (8974/131073)=C2=A0 438 KByte 3083767=C2=A0 1916= 2=3D3050:2269:2392:2486:3019:1891:1667:2388
> [=C2=A0 1] 9.00-10.00 sec=C2=A0 1.09 GBytes=C2=A0 9.40 Gbits/sec=C2=A0= 0.371/0.194/0.582/0.057 ms (8964/131070)=C2=A0 425 KByte 3169244=C2=A0 191= 43=3D3006:2411:2303:2462:3067:1744:1760:2390
> [=C2=A0 1] 0.00-10.00 sec=C2=A0 10.9 GBytes=C2=A0 9.39 Gbits/sec=C2=A0= 0.382/0.190/2.682/0.076 ms (89544/131072)=C2=A0 437 KByte 3074913=C2=A0 19= 4908=3D31380:23444:24362:26308:32839:18161:16084:22330
>
>
> [rjmcmahon@localhost iperf2-code]$ src/iperf -c 192.168.1.10 --near-co= ngestion=3D0.05 --trip-times -i 1 -e
> ------------------------------------------------------------
> Client connecting to 192.168.1.10, TCP port 5001 with pid 19320 (1 flo= ws)
> Write buffer size: 131072 Byte
> TCP near-congestion delay weight set to 0.0500
> TCP window size: 85.0 KByte (default)
> ------------------------------------------------------------
> [=C2=A0 1] local 192.168.1.62%enp2s0 port 50072 connected with 192.168= .1.10 port 5001 (MSS=3D1448) (trip-times) (sock=3D3) (ct=3D0.40 ms) on 2021= -01-07 12:14:59 (PST)
> [ ID] Interval=C2=A0 =C2=A0 =C2=A0 =C2=A0 Transfer=C2=A0 =C2=A0 Bandwi= dth=C2=A0 =C2=A0 =C2=A0 =C2=A0Write/Err=C2=A0 Rtry=C2=A0 =C2=A0 =C2=A0Cwnd/= RTT=C2=A0 =C2=A0 =C2=A0 =C2=A0 NetPwr
> [=C2=A0 1] 0.00-1.00 sec=C2=A0 1.08 GBytes=C2=A0 9.31 Gbits/sec=C2=A0 = 8881/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01135K/373 us= =C2=A0 3120427
> [=C2=A0 1] 1.00-2.00 sec=C2=A0 1.10 GBytes=C2=A0 9.41 Gbits/sec=C2=A0 = 8971/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01135K/391 us= =C2=A0 3007281
> [=C2=A0 1] 2.00-3.00 sec=C2=A0 1.09 GBytes=C2=A0 9.39 Gbits/sec=C2=A0 = 8958/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01135K/331 us= =C2=A0 3547260
> [=C2=A0 1] 3.00-4.00 sec=C2=A0 1.09 GBytes=C2=A0 9.39 Gbits/sec=C2=A0 = 8952/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01135K/288 us= =C2=A0 4074155
> [=C2=A0 1] 4.00-5.00 sec=C2=A0 1.09 GBytes=C2=A0 9.40 Gbits/sec=C2=A0 = 8965/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01135K/301 us= =C2=A0 3903855
> [=C2=A0 1] 5.00-6.00 sec=C2=A0 1.09 GBytes=C2=A0 9.39 Gbits/sec=C2=A0 = 8955/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01135K/414 us= =C2=A0 2835144
> [=C2=A0 1] 6.00-7.00 sec=C2=A0 1.09 GBytes=C2=A0 9.40 Gbits/sec=C2=A0 = 8961/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01135K/470 us= =C2=A0 2499013
> [=C2=A0 1] 7.00-8.00 sec=C2=A0 1.09 GBytes=C2=A0 9.40 Gbits/sec=C2=A0 = 8964/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01135K/350 us= =C2=A0 3356941
> [=C2=A0 1] 8.00-9.00 sec=C2=A0 1.10 GBytes=C2=A0 9.41 Gbits/sec=C2=A0 = 8973/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01135K/472 us= =C2=A0 2491756
> [=C2=A0 1] 9.00-10.00 sec=C2=A0 1.09 GBytes=C2=A0 9.40 Gbits/sec=C2=A0= 8964/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01135K/402 us= =C2=A0 2922710
> [=C2=A0 1] 0.00-10.00 sec=C2=A0 10.9 GBytes=C2=A0 9.39 Gbits/sec=C2=A0= 89547/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A01135K/402 u= s=C2=A0 2919642
>
> Bob
>
>
> On Thu, Jan 7, 2021 at 11:22 AM Taran Lynn via Make-wifi-fast <make-= wifi-fast@lists.bufferbloat.net> wrote:
>>
>> The source can be found at https://github.com/lamb= da-11235/tcp_davis .
>>
>> The code mentioned in the paper can be found under the tag "a= rxiv_2020". The current master branch has an additional stable mode th= at I was testing out.
>>
>>
>> On Thu, Jan 7, 2021 at 10:35 AM Dave Taht <dave.taht@gmail.com> wrote:
>>>
>>> See: https://arxiv.org/pdf/2012.14996.pdf
>>>
>>> Things I really like:
>>>
>>> * they used flent
>>> * Using "variance" as the principal signal. This is = essentially one of
>>> the great unpublished and unanalyzed improvements on the minst= rel
>>> algorithm as well
>>> * Conventional ecn response
>>> * outperforms bbr on variable links
>>>
>>> Only negative so far is I haven't found any published sour= ce to it. :(
>>>
>>> Otherwise a very promising start to a year.
>>>
>>> "The choice of feedback mechanism between delay and packe= t loss has
>>> long been a point of contention in TCP congestion control. Thi= s has
>>> partly been resolved, as it has become increasingly evident th= at delay
>>> based methods are needed to facilitate modern interactive web<= br> >>> applications. However, what has not been resolved is what cont= rol
>>> should be used, with the two candidates being the congestion w= indow
>>> and the pacing rate. BBR is a new delay based congestion contr= ol
>>> algorithm that uses a pacing rate as its primary control and t= he
>>> congestion window as a secondary control. We propose that a co= ngestion
>>> window first algorithm might give more desirable performance >>> characteristics in situations where latency must be minimized = even at
>>> the expense of some loss in throughput. To evaluate this hypot= hesis we
>>> introduce a new congestion control algorithm called TCP D*, wh= ich is a
>>> congestion window first algorithm that adopts BBR's approa= ch of
>>> maximizing delivery rate while minimizing latency. In this pap= er, we
>>> discuss the key features of this algorithm, discuss the differ= ences
>>> and similarity to BBR, and present some preliminary results ba= sed on a
>>> real implementation."
>>>
>>>
>>>
>>>
>>> --
>>> "For a successful technology, reality must take precedenc= e over public
>>> relations, for Mother Nature cannot be fooled" - Richard = Feynman
>>>
>>> dave@taht.n= et <Dave T=C3=A4ht> CTO, TekLibre, LLC Tel: 1-831-435-0729
>>
>> _______________________________________________
>> Make-wifi-fast mailing list
>> Make-wifi-fast@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo= /make-wifi-fast
>
>
> This electronic communication and the information and any files transm= itted with it, or attached to it, are confidential and are intended solely = for the use of the individual or entity to whom it is addressed and may con= tain information that is confidential, legally privileged, protected by pri= vacy laws, or otherwise restricted from disclosure to anyone else. If you a= re not the intended recipient or the person responsible for delivering the = e-mail to the intended recipient, you are hereby notified that any use, cop= ying, distributing, dissemination, forwarding, printing, or copying of this= e-mail is strictly prohibited. If you received this e-mail in error, pleas= e return the e-mail to the sender, delete it from your computer, and destro= y any printed copy of it.



--
"For a successful technology, reality must take precedence over public=
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Da= ve T=C3=A4ht> CTO, TekLibre, LLC Tel: 1-831-435-0729

This ele= ctronic communication and the information and any files transmitted with it= , or attached to it, are confidential and are intended solely for the use o= f the individual or entity to whom it is addressed and may contain informat= ion that is confidential, legally privileged, protected by privacy laws, or= otherwise restricted from disclosure to anyone else. If you are not the in= tended recipient or the person responsible for delivering the e-mail to the= intended recipient, you are hereby notified that any use, copying, distrib= uting, dissemination, forwarding, printing, or copying of this e-mail is st= rictly prohibited. If you received this e-mail in error, please return the = e-mail to the sender, delete it from your computer, and destroy any printed= copy of it. --0000000000007987bf05b856e0da-- --0000000000008182d905b856e069 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIIQPwYJKoZIhvcNAQcCoIIQMDCCECwCAQExDzANBglghkgBZQMEAgEFADALBgkqhkiG9w0BBwGg gg2UMIIE6DCCA9CgAwIBAgIOSBtqCRO9gCTKXSLwFPMwDQYJKoZIhvcNAQELBQAwTDEgMB4GA1UE CxMXR2xvYmFsU2lnbiBSb290IENBIC0gUjMxEzARBgNVBAoTCkdsb2JhbFNpZ24xEzARBgNVBAMT Ckdsb2JhbFNpZ24wHhcNMTYwNjE1MDAwMDAwWhcNMjQwNjE1MDAwMDAwWjBdMQswCQYDVQQGEwJC RTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEzMDEGA1UEAxMqR2xvYmFsU2lnbiBQZXJzb25h bFNpZ24gMiBDQSAtIFNIQTI1NiAtIEczMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA tpZok2X9LAHsYqMNVL+Ly6RDkaKar7GD8rVtb9nw6tzPFnvXGeOEA4X5xh9wjx9sScVpGR5wkTg1 fgJIXTlrGESmaqXIdPRd9YQ+Yx9xRIIIPu3Jp/bpbiZBKYDJSbr/2Xago7sb9nnfSyjTSnucUcIP ZVChn6hKneVGBI2DT9yyyD3PmCEJmEzA8Y96qT83JmVH2GaPSSbCw0C+Zj1s/zqtKUbwE5zh8uuZ p4vC019QbaIOb8cGlzgvTqGORwK0gwDYpOO6QQdg5d03WvIHwTunnJdoLrfvqUg2vOlpqJmqR+nH 9lHS+bEstsVJtZieU1Pa+3LzfA/4cT7XA/pnwwIDAQABo4IBtTCCAbEwDgYDVR0PAQH/BAQDAgEG MGoGA1UdJQRjMGEGCCsGAQUFBwMCBggrBgEFBQcDBAYIKwYBBQUHAwkGCisGAQQBgjcUAgIGCisG AQQBgjcKAwQGCSsGAQQBgjcVBgYKKwYBBAGCNwoDDAYIKwYBBQUHAwcGCCsGAQUFBwMRMBIGA1Ud EwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFGlygmIxZ5VEhXeRgMQENkmdewthMB8GA1UdIwQYMBaA FI/wS3+oLkUkrk1Q+mOai97i3Ru8MD4GCCsGAQUFBwEBBDIwMDAuBggrBgEFBQcwAYYiaHR0cDov L29jc3AyLmdsb2JhbHNpZ24uY29tL3Jvb3RyMzA2BgNVHR8ELzAtMCugKaAnhiVodHRwOi8vY3Js Lmdsb2JhbHNpZ24uY29tL3Jvb3QtcjMuY3JsMGcGA1UdIARgMF4wCwYJKwYBBAGgMgEoMAwGCisG AQQBoDIBKAowQQYJKwYBBAGgMgFfMDQwMgYIKwYBBQUHAgEWJmh0dHBzOi8vd3d3Lmdsb2JhbHNp Z24uY29tL3JlcG9zaXRvcnkvMA0GCSqGSIb3DQEBCwUAA4IBAQConc0yzHxn4gtQ16VccKNm4iXv 6rS2UzBuhxI3XDPiwihW45O9RZXzWNgVcUzz5IKJFL7+pcxHvesGVII+5r++9eqI9XnEKCILjHr2 DgvjKq5Jmg6bwifybLYbVUoBthnhaFB0WLwSRRhPrt5eGxMw51UmNICi/hSKBKsHhGFSEaJQALZy 4HL0EWduE6ILYAjX6BSXRDtHFeUPddb46f5Hf5rzITGLsn9BIpoOVrgS878O4JnfUWQi29yBfn75 HajifFvPC+uqn+rcVnvrpLgsLOYG/64kWX/FRH8+mhVe+mcSX3xsUpcxK9q9vLTVtroU/yJUmEC4 OcH5dQsbHBqjMIIDXzCCAkegAwIBAgILBAAAAAABIVhTCKIwDQYJKoZIhvcNAQELBQAwTDEgMB4G A1UECxMXR2xvYmFsU2lnbiBSb290IENBIC0gUjMxEzARBgNVBAoTCkdsb2JhbFNpZ24xEzARBgNV BAMTCkdsb2JhbFNpZ24wHhcNMDkwMzE4MTAwMDAwWhcNMjkwMzE4MTAwMDAwWjBMMSAwHgYDVQQL ExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSMzETMBEGA1UEChMKR2xvYmFsU2lnbjETMBEGA1UEAxMK R2xvYmFsU2lnbjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMwldpB5BngiFvXAg7aE yiie/QV2EcWtiHL8RgJDx7KKnQRfJMsuS+FggkbhUqsMgUdwbN1k0ev1LKMPgj0MK66X17YUhhB5 uzsTgHeMCOFJ0mpiLx9e+pZo34knlTifBtc+ycsmWQ1z3rDI6SYOgxXG71uL0gRgykmmKPZpO/bL yCiR5Z2KYVc3rHQU3HTgOu5yLy6c+9C7v/U9AOEGM+iCK65TpjoWc4zdQQ4gOsC0p6Hpsk+QLjJg 6VfLuQSSaGjlOCZgdbKfd/+RFO+uIEn8rUAVSNECMWEZXriX7613t2Saer9fwRPvm2L7DWzgVGkW qQPabumDk3F2xmmFghcCAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8w HQYDVR0OBBYEFI/wS3+oLkUkrk1Q+mOai97i3Ru8MA0GCSqGSIb3DQEBCwUAA4IBAQBLQNvAUKr+ yAzv95ZURUm7lgAJQayzE4aGKAczymvmdLm6AC2upArT9fHxD4q/c2dKg8dEe3jgr25sbwMpjjM5 RcOO5LlXbKr8EpbsU8Yt5CRsuZRj+9xTaGdWPoO4zzUhw8lo/s7awlOqzJCK6fBdRoyV3XpYKBov Hd7NADdBj+1EbddTKJd+82cEHhXXipa0095MJ6RMG3NzdvQXmcIfeg7jLQitChws/zyrVQ4PkX42 68NXSb7hLi18YIvDQVETI53O9zJrlAGomecsMx86OyXShkDOOyyGeMlhLxS67ttVb9+E7gUJTb0o 2HLO02JQZR7rkpeDMdmztcpHWD9fMIIFQTCCBCmgAwIBAgIML7TfFWHfxluS5m0sMA0GCSqGSIb3 DQEBCwUAMF0xCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMTMwMQYDVQQD EypHbG9iYWxTaWduIFBlcnNvbmFsU2lnbiAyIENBIC0gU0hBMjU2IC0gRzMwHhcNMjAwODMxMDgw OTQ5WhcNMjIwOTAxMDgwOTQ5WjCBjDELMAkGA1UEBhMCSU4xEjAQBgNVBAgTCUthcm5hdGFrYTES MBAGA1UEBxMJQmFuZ2Fsb3JlMRYwFAYDVQQKEw1Ccm9hZGNvbSBJbmMuMRQwEgYDVQQDEwtCb2Ig TWNNYWhvbjEnMCUGCSqGSIb3DQEJARYYYm9iLm1jbWFob25AYnJvYWRjb20uY29tMIIBIjANBgkq hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzK1kauAQxAkeif97AJqv3QgJVDztfPjduswFJbcT0VbZ hr+E3gpqCnUm9TM4HZ2aviVSu9A9SkcpQBOaiOUPUkMrD6PoviodpwV2C5W+i7e/lexjsUdSuTZe h7GRKEOlhPnsW7RgKFR5+3Rrm1kAyEJ2x2ueyn3UhupxoNrYxZOss/+dPwQJ28kP7ICHBBJbcfmh doMU83sfJazBMsp+pMArApFdMwXOBL3dT4ZPkniaMilO+q2y7xQ212K+KlhaRWCWcImC/+pqZCyH /b1zx6vfBbdB/WD9+oi9rnXyd2ikVU2bvJ15VinB52kz7Kpj4A1e7zHgKxqNjFFycWXn2QIDAQAB o4IBzzCCAcswDgYDVR0PAQH/BAQDAgWgMIGeBggrBgEFBQcBAQSBkTCBjjBNBggrBgEFBQcwAoZB aHR0cDovL3NlY3VyZS5nbG9iYWxzaWduLmNvbS9jYWNlcnQvZ3NwZXJzb25hbHNpZ24yc2hhMmcz b2NzcC5jcnQwPQYIKwYBBQUHMAGGMWh0dHA6Ly9vY3NwMi5nbG9iYWxzaWduLmNvbS9nc3BlcnNv bmFsc2lnbjJzaGEyZzMwTQYDVR0gBEYwRDBCBgorBgEEAaAyASgKMDQwMgYIKwYBBQUHAgEWJmh0 dHBzOi8vd3d3Lmdsb2JhbHNpZ24uY29tL3JlcG9zaXRvcnkvMAkGA1UdEwQCMAAwRAYDVR0fBD0w OzA5oDegNYYzaHR0cDovL2NybC5nbG9iYWxzaWduLmNvbS9nc3BlcnNvbmFsc2lnbjJzaGEyZzMu Y3JsMCMGA1UdEQQcMBqBGGJvYi5tY21haG9uQGJyb2FkY29tLmNvbTATBgNVHSUEDDAKBggrBgEF BQcDBDAfBgNVHSMEGDAWgBRpcoJiMWeVRIV3kYDEBDZJnXsLYTAdBgNVHQ4EFgQUQ1s1Ocvee2Eh o0qY50aFmX9fHS0wDQYJKoZIhvcNAQELBQADggEBAFygcUiFPJ7uRXexxLYonpGCsztO0YcmuWCE jef/oS4oMYJyf+pv/dgdywaJ9U4FhYHg9wIPEcWwS1JpsUrEhx1zQ2JX8+HNHyuCiR11tgtzCr0z nxeMm3RAJPmM3cAy5mS7pDz1Ox081EqDQQzeesvMwwONjIpV1ORg17fpIHxz4KaKg8X5Yv6xYs0Z L+UVLQlmIJqoOHqzOelCjNTCSucYx/sYygzQcvUuX7qUgLOJq+o4JXhqu2dUJ6wu/NPApqyd8hSc YVBMvygG2Y3xBjxYC5Dr4+E1cmcOGEUQhkKo4cwv217+W4FXb4K4B/EM208EsUMXGTb7tV5zJ0rO IH4xggJvMIICawIBATBtMF0xCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNh MTMwMQYDVQQDEypHbG9iYWxTaWduIFBlcnNvbmFsU2lnbiAyIENBIC0gU0hBMjU2IC0gRzMCDC+0 3xVh38ZbkuZtLDANBglghkgBZQMEAgEFAKCB1DAvBgkqhkiG9w0BCQQxIgQgXqomsEoH+7C2Q/MD eYUcSu0N3oAazck0TsDAzut/fhcwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0B CQUxDxcNMjEwMTA3MjIyMjM2WjBpBgkqhkiG9w0BCQ8xXDBaMAsGCWCGSAFlAwQBKjALBglghkgB ZQMEARYwCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMAsGCSqGSIb3DQEBCjALBgkqhkiG9w0BAQcw CwYJYIZIAWUDBAIBMA0GCSqGSIb3DQEBAQUABIIBALTWj8GqxW5YhvKdAxBpvvoDHibWLXuchFv2 ZU1Yl4w2XXaxx0pXztiaU9ROiGpkEPhiZvklSF0gz/NrSuBDFf+3rrCaK1elQ9Y9x5q2zQHDyZJz l6N11L235fv+2Hla63Tt8lrQKC0lC4vY7UDykPS3qoBc1sGVnaBtsjaydwge4gad8jgQiPdIDPWW aTexNlnNG03PMpnU0iCjiCVY+z72SFekwvYaF9jlneRKt/XmXvH0RFweBhjWBMfKYvvnK2IoPnIU 229SnwqTWpKLlWlvvEShOHmuJJY8Z9p2S3Z+XB4oTuaHbVA3F5HOshYfbgZSWrXwg1dBSKDyLWoD DUM= --0000000000008182d905b856e069--