[Flent-users] ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner

[Flent-users] ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner

[Chriztoffer]

flent-users,

When I try to run the "bursts" test, I get an error message
complaining there is no command set for class SilentProcessRunner.
Which leaves me a bit clueless, after trying to dig something forth
from the github issue tracker. And looking at the source code for the
class. (What am I overlooking?)

(Pop!_OS 22.04 LTS)

$ flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30
--host=flent.example.com --title-extra="Flent Test
2023-03-28T11_03_30_688203731"
--output=flent_test_2023-03-28T11_03_30_688203731.png
--data-dir=/home/flent/flent/bursts/2023-03-28T11_03_30_688203731
Starting Flent 2.0.1 using Python 3.10.6.
Starting bursts test. Expected run time: 300 seconds.
ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner

(Debian GNU/Linux 11)

$ flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30
--host=flent.example.com --title-extra="Flent Test
2023-03-28T11_03_30_688203731"
--output=flent_test_2023-03-28T11_03_30_688203731.png
--data-dir=/home/flent/flent/bursts/2023-03-28T11_03_30_688203731
Started Flent 2.0.0 using Python 3.9.2.
Starting bursts test. Expected run time: 300 seconds.
ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner

/Chriztoffer

[Flent-users] Re: ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner

[Toke Høiland-Jørgensen]

Chriztoffer <ch@ntrv.dk> writes:

> flent-users,
>
> When I try to run the "bursts" test, I get an error message
> complaining there is no command set for class SilentProcessRunner.
> Which leaves me a bit clueless, after trying to dig something forth
> from the github issue tracker. And looking at the source code for the
> class. (What am I overlooking?)

Congratulations, you're the first person to try to use that test in a
least five years! (AFAICT it's been broken at least that long)

The immediate API issue you're seeing is fixable (pushed a fix to github
just now), but TBH I am not sure which utility that test is trying to
run. :/

It's calling a 'traffic-gen' utility, which is not really a googlable
name, and for the life of me I can't remember what that utility is
supposed to be. Do you have such a binary on your system?

-Toke

[Flent-users] Re: ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner

[ch]

Hi,

On 28/03/2023 14.39, Toke Høiland-Jørgensen wrote:
> The immediate API issue you're seeing is fixable (pushed a fix to github
> just now), but TBH I am not sure which utility that test is trying to
> run. :/
> 
> It's calling a 'traffic-gen' utility, which is not really a googlable
> name, and for the life of me I can't remember what that utility is
> supposed to be. Do you have such a binary on your system?

No executeable by that name. :(

% which traffic-gen
traffic-gen not found


Searching the local package index gives nothing. (Neither does the 
Debian+Ubuntu package indexes)

$ apt-cache search traffic-gen
<empty output>

https://packages.debian.org/index
https://packages.ubuntu.com/


Tried on Pypi to find something by that name.

1. https://pypi.org/search/?q=trafficgen
2. https://pypi.org/search/?q=traffic-gen

The second one gives so many results. It impossible for me to pick out 
where to start looking. 🤯


The commits adding tests/bursts.conf and flent/tests/bursts_11e.conf 
gives me little information to go by concerning the traffic-gen binary.

https://github.com/tohojo/flent/commit/d9aa3f223c4ae6ebafe1613fb7d980efe2cba23b?w=1
https://github.com/tohojo/flent/commit/056cba5eb7f0a74d39c5f1a00e145f03513ab10d?w=1

commit 056cba5eb7f0a74d39c5f1a00e145f03513ab10d refs/tags/v0.15.0
Date:   Wed Mar 16 13:01:51 2016 -0700

     Exercise all for 802.11e queues with bursts_11e.conf

     I note that given the structure of the test, we could get away
     from using ping and use netperf throughout.

commit d9aa3f223c4ae6ebafe1613fb7d980efe2cba23b refs/tags/v0.7.0
Date:   Tue Sep 2 15:37:17 2014 +0200

     Add 'bursts' test configuration for measuring latency under 
intermittent bursts.

-Chriztoffer

[Flent-users] Re: ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner

[ch]

[google foo]

https://www.google.com/search?q=%22traffic-gen%22+flent

The best that turned up are link to old 2016 posts,

https://lists.bufferbloat.net/pipermail/make-wifi-fast/2016-March/000413.html

http://lists.infradead.org/pipermail/ath10k/2016-March/007101.html

https://www.spinics.net/lists/linux-wireless/msg148958.html

[more google foo]

Looking for the binary name together with Git gives me confusing answers.

https://www.google.com/search?q=%22traffic-gen%22+%22git%22

Maybe Dave knows which traffic-gen binary the bursts tests uses?

[Flent-users] Re: ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner

[Toke Høiland-Jørgensen]

ch <ch@ntrv.dk> writes:

> [google foo]
>
> https://www.google.com/search?q=%22traffic-gen%22+flent
>
> The best that turned up are link to old 2016 posts,
>
> https://lists.bufferbloat.net/pipermail/make-wifi-fast/2016-March/000413.html
>
> http://lists.infradead.org/pipermail/ath10k/2016-March/007101.html
>
> https://www.spinics.net/lists/linux-wireless/msg148958.html
>
> [more google foo]
>
> Looking for the binary name together with Git gives me confusing answers.
>
> https://www.google.com/search?q=%22traffic-gen%22+%22git%22
>
> Maybe Dave knows which traffic-gen binary the bursts tests uses?

Ah, dug out some emails which set me on the right path. Seems to be this
utility which, apparently, I wrote myself a decade ago, and then,
evidently, forgot all about:

https://github.com/tohojo/traffic-gen

Seems to work well enough, so go ahead and try it, I guess? :)

-Toke

[Flent-users] Re: ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner

[ch]

Hi Toke,

On 28/03/2023 16.01, Toke Høiland-Jørgensen wrote:
> Ah, dug out some emails which set me on the right path. Seems to be this
> utility which, apparently, I wrote myself a decade ago, and then,
> evidently, forgot all about:
> 
> https://github.com/tohojo/traffic-gen
> 
> Seems to work well enough, so go ahead and try it, I guess? :)

Installed the latest flent and traffic-gen from source yesterday after 
our mail conversations.

Ran into an issue with the bursts test, where the flent test continues 
to run. Resulting in the node generating traffic for the full pipe for 
hours on end. My observation is the test does not terminate for unknown 
reasons.

Found this test running from yesterday afternoon this morning still 
actively running. Which was strange. As the test duration was set to 
begin => 30s => 300s => 30s => end.

> # ps aux | grep -P '(traffic-gen|flent)'
> avahi       5036  0.0  0.2   7344  3136 ?        Ss   Mar28   0:00 avahi-daemon: running [flent1.local]
> root       14476  0.0  0.3  12312  3688 ?        Ss   Mar28   0:00 zsh /usr/local/bin/flent-test.sh
> root       14502  0.0  3.8 908328 40248 ?        Sl   Mar28   0:03 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host1.example.com --title-extra=CPE IPv4 H:host1_example_com Flent Test 2023-03-28T20_45_38_799019106 --output=flent_test_2023-03-28T20_45_38_799019106.png --data-dir=/root/flent/bursts/2023-03-28T20_45_38_799019106
> root       14550  0.0  0.1   3920  2068 ?        R    Mar28 647:48 traffic-gen -f 1000 -p 0 -l 5 -s 500 -dD host1.example.com
> root       14551  0.0  0.1   3920  2080 ?        R    Mar28 647:06 traffic-gen -f 1000 -p 0 -l 5 -s 500 -dD host1.example.com
> root       14552  0.0  0.1   3920  2036 ?        R    Mar28 646:31 traffic-gen -f 1000 -p 0 -l 5 -s 500 -dD host1.example.com
> root       16188  0.0  0.0  10676   704 pts/3    S+   07:34   0:00 grep -P (traffic-gen|flent)

Could this happen if the node is not able to connect to the remote end 
during the test? I.e. the inband control channel is not possible (to 
little bandwidth available), due to the bandwidth just filling up the 
wired pipe... :think:

-Chriztoffer

[Flent-users] Re: ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner

[Toke Høiland-Jørgensen]

ch <ch@ntrv.dk> writes:

> Hi Toke,
>
> On 28/03/2023 16.01, Toke Høiland-Jørgensen wrote:
>> Ah, dug out some emails which set me on the right path. Seems to be this
>> utility which, apparently, I wrote myself a decade ago, and then,
>> evidently, forgot all about:
>> 
>> https://github.com/tohojo/traffic-gen
>> 
>> Seems to work well enough, so go ahead and try it, I guess? :)
>
> Installed the latest flent and traffic-gen from source yesterday after 
> our mail conversations.
>
> Ran into an issue with the bursts test, where the flent test continues 
> to run. Resulting in the node generating traffic for the full pipe for 
> hours on end. My observation is the test does not terminate for unknown 
> reasons.

Hmm, did you pull the latest version of traffic-gen? The version on
github had a bug where it would just go into an infinite loop and never
exit (and not produce any traffic either). Fixed that yesterday, so make
sure you pull the fixed version...

-Toke

[Flent-users] Re: ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner

[ch]

[-- Attachment #1: Type: text/plain, Size: 1317 bytes --]

On 29/03/2023 12.10, Toke Høiland-Jørgensen wrote:
>> to run. Resulting in the node generating traffic for the full pipe for
>> hours on end. My observation is the test does not terminate for unknown
>> reasons.
> Hmm, did you pull the latest version of traffic-gen? The version on
> github had a bug where it would just go into an infinite loop and never
> exit (and not produce any traffic either). Fixed that yesterday, so make
> sure you pull the fixed version...

Hmm. Good question

> # ls -la traffic-gen 
> -rwxr-xr-x 1 root root 22832 Mar 28 20:04 traffic-gen

I did compile the binary yesterday afternoon. So I can only assume it 
contains the tree commits from yesterday.

> # git log --oneline
> b316813 (HEAD -> master, origin/master, origin/HEAD) Use getrandom() instead of rand()/srand()
> eedf705 Fix infinite loop in scheduling logic
> 5c1007e Support setting the TOS byte on transmitted packets.

(The application is missing the option for outputting a version number 
or git hash using e.g. -v, -V, or --version)

> and not produce any traffic either

It did produce *lots* of traffic. Had to terminate flent process to 
terminate the child traffic-gen processes.
Attached is the graph of the network traffic levels (in MB/s) as 
reported by Proxmox.

-Chriztoffer

[-- Attachment #2: 2023-03-29_13-08.png --]
[-- Type: image/png, Size: 70191 bytes --]

[Flent-users] Re: ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner

[Toke Høiland-Jørgensen]

ch <ch@ntrv.dk> writes:

> On 29/03/2023 12.10, Toke Høiland-Jørgensen wrote:
>>> to run. Resulting in the node generating traffic for the full pipe for
>>> hours on end. My observation is the test does not terminate for unknown
>>> reasons.
>> Hmm, did you pull the latest version of traffic-gen? The version on
>> github had a bug where it would just go into an infinite loop and never
>> exit (and not produce any traffic either). Fixed that yesterday, so make
>> sure you pull the fixed version...
>
> Hmm. Good question
>
>> # ls -la traffic-gen 
>> -rwxr-xr-x 1 root root 22832 Mar 28 20:04 traffic-gen
>
> I did compile the binary yesterday afternoon. So I can only assume it 
> contains the tree commits from yesterday.
>
>> # git log --oneline
>> b316813 (HEAD -> master, origin/master, origin/HEAD) Use getrandom() instead of rand()/srand()
>> eedf705 Fix infinite loop in scheduling logic
>> 5c1007e Support setting the TOS byte on transmitted packets.
>
> (The application is missing the option for outputting a version number 
> or git hash using e.g. -v, -V, or --version)
>
>> and not produce any traffic either
>
> It did produce *lots* of traffic. Had to terminate flent process to 
> terminate the child traffic-gen processes.
> Attached is the graph of the network traffic levels (in MB/s) as 
> reported by Proxmox.

Ah! Seems I introduced another infinite loop while fixing the previous
one; pushed another fix, please try that...

-Toke

[Flent-users] Re: ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner

[ch]

On 29/03/2023 14.33, Toke Høiland-Jørgensen wrote:
> Ah! Seems I introduced another infinite loop while fixing the previous
> one; pushed another fix, please try that...

😅 Haha,

Compiled the new version (acf8c41) and setup my scheduled 
(flent-tests.sh) job to run the bursts test (again). 🙈

Created https://github.com/tohojo/traffic-gen/issues/1 for the version 
identifier request.

-ch

[Flent-users] Re: ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner

[ch]

Hi Toke,

On 29/03/2023 16.34, ch wrote:
> On 29/03/2023 14.33, Toke Høiland-Jørgensen wrote:
>> Ah! Seems I introduced another infinite loop while fixing the previous
>> one; pushed another fix, please try that...
> 
> 😅 Haha,
> 
> Compiled the new version (acf8c41) and setup my scheduled 
> (flent-tests.sh) job to run the bursts test (again). 🙈
> 
> Created https://github.com/tohojo/traffic-gen/issues/1 for the version 
> identifier request.

So the current summary is traffic-gen correctly terminates.

But the flent test itself never terminate. The process keeps running in 
the background. Resulting in the test results being saved to disk.

> # ps aux | grep -P '(flent|traffic-gen)'
> avahi       5036  0.0  0.2   7344  3136 ?        Ss   Mar28   0:00 avahi-daemon: running [flent1.local]
> root       20016  0.0  0.3  12312  3696 ?        Ss   15:00   0:00 zsh /usr/local/bin/flent-test.sh
> root       20043  0.0  3.9 892872 41040 ?        Sl   15:03 147:35 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host1.example.com --title-extra=CPE IPv6 H:host1_example_com Flent Test 2023-03-29T15_03_45_100731085 --output=flent_test_2023-03-29T15_03_45_100731085.png --data-dir=/root/flent/bursts/2023-03-29T15_03_45_100731085
> root       20126  0.0  2.9 884396 30512 ?        Sl   15:08   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host1.example.com --title-extra=CPE IPv6 H:host1_example_com Flent Test 2023-03-29T15_03_45_100731085 --output=flent_test_2023-03-29T15_03_45_100731085.png --data-dir=/root/flent/bursts/2023-03-29T15_03_45_100731085
> root       20132  0.0  2.8 884676 30112 ?        S    15:08   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host1.example.com --title-extra=CPE IPv6 H:host1_example_com Flent Test 2023-03-29T15_03_45_100731085 --output=flent_test_2023-03-29T15_03_45_100731085.png --data-dir=/root/flent/bursts/2023-03-29T15_03_45_100731085
> root       20133  0.0  2.8 884676 29568 ?        S    15:08   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host1.example.com --title-extra=CPE IPv6 H:host1_example_com Flent Test 2023-03-29T15_03_45_100731085 --output=flent_test_2023-03-29T15_03_45_100731085.png --data-dir=/root/flent/bursts/2023-03-29T15_03_45_100731085
> root       20135  0.0  2.8 884676 29568 ?        S    15:08   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host1.example.com --title-extra=CPE IPv6 H:host1_example_com Flent Test 2023-03-29T15_03_45_100731085 --output=flent_test_2023-03-29T15_03_45_100731085.png --data-dir=/root/flent/bursts/2023-03-29T15_03_45_100731085
> root       20302  0.0  0.3  12312  3688 ?        Ss   15:42   0:00 zsh /usr/local/bin/flent-test.sh
> root       20328  0.0  4.1 895176 43980 ?        Sl   15:46 104:36 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv4 H:host2_example_com Flent Test 2023-03-29T15_46_26_404412294 --output=flent_test_2023-03-29T15_46_26_404412294.png --data-dir=/root/flent/bursts/2023-03-29T15_46_26_404412294
> root       20423  0.0  2.9 884396 30680 ?        Sl   15:51   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv4 H:host2_example_com Flent Test 2023-03-29T15_46_26_404412294 --output=flent_test_2023-03-29T15_46_26_404412294.png --data-dir=/root/flent/bursts/2023-03-29T15_46_26_404412294
> root       20429  0.0  3.0 886064 31540 ?        S    15:51   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv4 H:host2_example_com Flent Test 2023-03-29T15_46_26_404412294 --output=flent_test_2023-03-29T15_46_26_404412294.png --data-dir=/root/flent/bursts/2023-03-29T15_46_26_404412294
> root       20430  0.0  3.4 890496 36156 ?        S    15:51   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv4 H:host2_example_com Flent Test 2023-03-29T15_46_26_404412294 --output=flent_test_2023-03-29T15_46_26_404412294.png --data-dir=/root/flent/bursts/2023-03-29T15_46_26_404412294
> root       20432  0.0  3.0 886356 31856 ?        S    15:51   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv4 H:host2_example_com Flent Test 2023-03-29T15_46_26_404412294 --output=flent_test_2023-03-29T15_46_26_404412294.png --data-dir=/root/flent/bursts/2023-03-29T15_46_26_404412294
> root       21062  0.0  0.3  12312  3632 ?        Ss   17:21   0:00 zsh /usr/local/bin/flent-test.sh
> root       21122  0.0  3.9 892872 41072 ?        Sl   17:25   6:03 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host1.example.com --title-extra=CPE IPv6 H:host1_example_com Flent Test 2023-03-29T17_25_09_411305033 --output=flent_test_2023-03-29T17_25_09_411305033.png --data-dir=/root/flent/bursts/2023-03-29T17_25_09_411305033
> root       21218  0.0  2.9 884396 30788 ?        Sl   17:30   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host1.example.com --title-extra=CPE IPv6 H:host1_example_com Flent Test 2023-03-29T17_25_09_411305033 --output=flent_test_2023-03-29T17_25_09_411305033.png --data-dir=/root/flent/bursts/2023-03-29T17_25_09_411305033
> root       21224  0.0  2.8 884676 30088 ?        S    17:30   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host1.example.com --title-extra=CPE IPv6 H:host1_example_com Flent Test 2023-03-29T17_25_09_411305033 --output=flent_test_2023-03-29T17_25_09_411305033.png --data-dir=/root/flent/bursts/2023-03-29T17_25_09_411305033
> root       21225  0.0  2.8 884676 30088 ?        S    17:30   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host1.example.com --title-extra=CPE IPv6 H:host1_example_com Flent Test 2023-03-29T17_25_09_411305033 --output=flent_test_2023-03-29T17_25_09_411305033.png --data-dir=/root/flent/bursts/2023-03-29T17_25_09_411305033
> root       21226  0.0  2.8 884676 30096 ?        S    17:30   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host1.example.com --title-extra=CPE IPv6 H:host1_example_com Flent Test 2023-03-29T17_25_09_411305033 --output=flent_test_2023-03-29T17_25_09_411305033.png --data-dir=/root/flent/bursts/2023-03-29T17_25_09_411305033
> 
> ~/flent/bursts # ls *
> 
> 	(empty output)

Any good ideas?

-ch

[Flent-users] Re: ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner

[ch]

On 29/03/2023 19.42, ch wrote:
> But the flent test itself never terminate. The process keeps running in 
> the background. Resulting in the test results being saved to disk.

Resulting in the test results *not* being saved to disk.

-ch

[Flent-users] Re: bursts test does not complete (WAS: ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner)

[ch]

Hi :wave:

On 29/03/2023 19.43, ch wrote:
> On 29/03/2023 19.42, ch wrote:
>> But the flent test itself never terminate. The process keeps running 
>> in the background. Resulting in the test results being saved to disk.
> 
> Resulting in the test results *not* being saved to disk.

I fell like I am getting more gray hairs by the hours. 🤕 I cannot 
figure out why this happens.

The burst test seems to run. Flent never "completes". Resulting in no 
data output. :|

> # tree $(pwd)
> /root/flent/bursts
> ├── 2023-03-29T19_04_13_233854739
> └── 2023-03-29T19_23_30_194006033

Ending up with multiple processes still running. Having not completed 
"correctly".

> # ps aux | grep -P '(flent|traffic-gen)'
> avahi       5036  0.0  0.2   7344  3136 ?        Ss   Mar28   0:00 avahi-daemon: running [flent1.local]
> root       20016  0.0  0.3  12312  3696 ?        Ss   15:00   0:00 zsh /usr/local/bin/flent-test.sh
> root       20043  0.0  3.9 892872 41040 ?        Sl   15:03 229:02 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv6 H:host2_example_com Flent Test 2023-03-29T15_03_45_100731085 --output=flent_test_2023-03-29T15_03_45_100731085.png --data-dir=/root/flent/bursts/2023-03-29T15_03_45_100731085
> root       20126  0.0  2.7 884396 29048 ?        Sl   15:08   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv6 H:host2_example_com Flent Test 2023-03-29T15_03_45_100731085 --output=flent_test_2023-03-29T15_03_45_100731085.png --data-dir=/root/flent/bursts/2023-03-29T15_03_45_100731085
> root       20132  0.0  2.7 884676 29232 ?        S    15:08   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv6 H:host2_example_com Flent Test 2023-03-29T15_03_45_100731085 --output=flent_test_2023-03-29T15_03_45_100731085.png --data-dir=/root/flent/bursts/2023-03-29T15_03_45_100731085
> root       20133  0.0  2.7 884676 28672 ?        S    15:08   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv6 H:host2_example_com Flent Test 2023-03-29T15_03_45_100731085 --output=flent_test_2023-03-29T15_03_45_100731085.png --data-dir=/root/flent/bursts/2023-03-29T15_03_45_100731085
> root       20135  0.0  2.7 884676 28672 ?        S    15:08   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv6 H:host2_example_com Flent Test 2023-03-29T15_03_45_100731085 --output=flent_test_2023-03-29T15_03_45_100731085.png --data-dir=/root/flent/bursts/2023-03-29T15_03_45_100731085
> root       20302  0.0  0.3  12312  3688 ?        Ss   15:42   0:00 zsh /usr/local/bin/flent-test.sh
> root       20328  0.0  4.1 895176 43980 ?        Sl   15:46 186:56 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host1.example.com --title-extra=CPE IPv4 H:host1_example_com Flent Test 2023-03-29T15_46_26_404412294 --output=flent_test_2023-03-29T15_46_26_404412294.png --data-dir=/root/flent/bursts/2023-03-29T15_46_26_404412294
> root       20423  0.0  2.8 884396 30064 ?        Sl   15:51   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host1.example.com --title-extra=CPE IPv4 H:host1_example_com Flent Test 2023-03-29T15_46_26_404412294 --output=flent_test_2023-03-29T15_46_26_404412294.png --data-dir=/root/flent/bursts/2023-03-29T15_46_26_404412294
> root       20429  0.0  2.9 886064 31404 ?        S    15:51   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host1.example.com --title-extra=CPE IPv4 H:host1_example_com Flent Test 2023-03-29T15_46_26_404412294 --output=flent_test_2023-03-29T15_46_26_404412294.png --data-dir=/root/flent/bursts/2023-03-29T15_46_26_404412294
> root       20430  0.0  3.4 890496 36020 ?        S    15:51   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host1.example.com --title-extra=CPE IPv4 H:host1_example_com Flent Test 2023-03-29T15_46_26_404412294 --output=flent_test_2023-03-29T15_46_26_404412294.png --data-dir=/root/flent/bursts/2023-03-29T15_46_26_404412294
> root       20432  0.0  3.0 886356 31856 ?        S    15:51   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host1.example.com --title-extra=CPE IPv4 H:host1_example_com Flent Test 2023-03-29T15_46_26_404412294 --output=flent_test_2023-03-29T15_46_26_404412294.png --data-dir=/root/flent/bursts/2023-03-29T15_46_26_404412294
> root       21062  0.0  0.3  12312  3632 ?        Ss   17:21   0:00 zsh /usr/local/bin/flent-test.sh
> root       21122  0.0  3.9 892872 41072 ?        Sl   17:25  87:55 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv6 H:host2_example_com Flent Test 2023-03-29T17_25_09_411305033 --output=flent_test_2023-03-29T17_25_09_411305033.png --data-dir=/root/flent/bursts/2023-03-29T17_25_09_411305033
> root       21218  0.0  2.9 884396 30788 ?        Sl   17:30   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv6 H:host2_example_com Flent Test 2023-03-29T17_25_09_411305033 --output=flent_test_2023-03-29T17_25_09_411305033.png --data-dir=/root/flent/bursts/2023-03-29T17_25_09_411305033
> root       21224  0.0  2.8 884676 30088 ?        S    17:30   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv6 H:host2_example_com Flent Test 2023-03-29T17_25_09_411305033 --output=flent_test_2023-03-29T17_25_09_411305033.png --data-dir=/root/flent/bursts/2023-03-29T17_25_09_411305033
> root       21225  0.0  2.8 884676 30088 ?        S    17:30   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv6 H:host2_example_com Flent Test 2023-03-29T17_25_09_411305033 --output=flent_test_2023-03-29T17_25_09_411305033.png --data-dir=/root/flent/bursts/2023-03-29T17_25_09_411305033
> root       21226  0.0  2.8 884676 30096 ?        S    17:30   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv6 H:host2_example_com Flent Test 2023-03-29T17_25_09_411305033 --output=flent_test_2023-03-29T17_25_09_411305033.png --data-dir=/root/flent/bursts/2023-03-29T17_25_09_411305033
> root       21848  0.0  0.3  12312  3500 ?        Ss   17:42   0:00 zsh /usr/local/bin/flent-test.sh
> root       21874  0.0  4.2 896440 45008 ?        Sl   17:44  69:08 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv4 H:host2_example_com Flent Test 2023-03-29T17_44_14_685857174 --output=flent_test_2023-03-29T17_44_14_685857174.png --data-dir=/root/flent/bursts/2023-03-29T17_44_14_685857174
> root       21970  0.0  2.9 884380 30772 ?        Sl   17:49   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv4 H:host2_example_com Flent Test 2023-03-29T17_44_14_685857174 --output=flent_test_2023-03-29T17_44_14_685857174.png --data-dir=/root/flent/bursts/2023-03-29T17_44_14_685857174
> root       21976  0.0  3.0 886088 31668 ?        S    17:49   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv4 H:host2_example_com Flent Test 2023-03-29T17_44_14_685857174 --output=flent_test_2023-03-29T17_44_14_685857174.png --data-dir=/root/flent/bursts/2023-03-29T17_44_14_685857174
> root       21977  0.0  3.0 886056 31640 ?        S    17:49   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv4 H:host2_example_com Flent Test 2023-03-29T17_44_14_685857174 --output=flent_test_2023-03-29T17_44_14_685857174.png --data-dir=/root/flent/bursts/2023-03-29T17_44_14_685857174
> root       21978  0.0  3.0 886068 31784 ?        S    17:49   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv4 H:host2_example_com Flent Test 2023-03-29T17_44_14_685857174 --output=flent_test_2023-03-29T17_44_14_685857174.png --data-dir=/root/flent/bursts/2023-03-29T17_44_14_685857174
> root       22143  0.0  0.3  12312  3560 ?        Ss   18:21   0:00 zsh /usr/local/bin/flent-test.sh
> root       22169  0.0  3.9 892872 41284 ?        Sl   18:22  40:04 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv6 H:host2_example_com Flent Test 2023-03-29T18_22_36_169860747 --output=flent_test_2023-03-29T18_22_36_169860747.png --data-dir=/root/flent/bursts/2023-03-29T18_22_36_169860747
> root       22257  0.0  2.9 892592 30836 ?        Sl   18:27   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv6 H:host2_example_com Flent Test 2023-03-29T18_22_36_169860747 --output=flent_test_2023-03-29T18_22_36_169860747.png --data-dir=/root/flent/bursts/2023-03-29T18_22_36_169860747
> root       22263  0.0  2.8 884676 30312 ?        S    18:27   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv6 H:host2_example_com Flent Test 2023-03-29T18_22_36_169860747 --output=flent_test_2023-03-29T18_22_36_169860747.png --data-dir=/root/flent/bursts/2023-03-29T18_22_36_169860747
> root       22264  0.0  2.8 884676 30312 ?        S    18:27   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv6 H:host2_example_com Flent Test 2023-03-29T18_22_36_169860747 --output=flent_test_2023-03-29T18_22_36_169860747.png --data-dir=/root/flent/bursts/2023-03-29T18_22_36_169860747
> root       22265  0.0  2.8 884676 30316 ?        S    18:27   0:00 /bin/python /usr/local/bin/flent bursts --ipv6 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv6 H:host2_example_com Flent Test 2023-03-29T18_22_36_169860747 --output=flent_test_2023-03-29T18_22_36_169860747.png --data-dir=/root/flent/bursts/2023-03-29T18_22_36_169860747
> root       22435  0.0  0.3  12312  3668 ?        Ss   19:00   0:00 zsh /usr/local/bin/flent-test.sh
> root       22461  0.0  4.3 896456 45168 ?        Sl   19:04  15:07 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host3.example.com --title-extra=CPE IPv4 H:host3_example_com Flent Test 2023-03-29T19_04_13_233854739 --output=flent_test_2023-03-29T19_04_13_233854739.png --data-dir=/root/flent/bursts/2023-03-29T19_04_13_233854739
> root       22556  0.0  2.9 884396 30776 ?        Sl   19:09   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host3.example.com --title-extra=CPE IPv4 H:host3_example_com Flent Test 2023-03-29T19_04_13_233854739 --output=flent_test_2023-03-29T19_04_13_233854739.png --data-dir=/root/flent/bursts/2023-03-29T19_04_13_233854739
> root       22562  0.0  3.0 885988 31728 ?        S    19:09   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host3.example.com --title-extra=CPE IPv4 H:host3_example_com Flent Test 2023-03-29T19_04_13_233854739 --output=flent_test_2023-03-29T19_04_13_233854739.png --data-dir=/root/flent/bursts/2023-03-29T19_04_13_233854739
> root       22563  0.0  3.0 885880 31592 ?        S    19:09   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host3.example.com --title-extra=CPE IPv4 H:host3_example_com Flent Test 2023-03-29T19_04_13_233854739 --output=flent_test_2023-03-29T19_04_13_233854739.png --data-dir=/root/flent/bursts/2023-03-29T19_04_13_233854739
> root       22564  0.0  3.0 886332 31864 ?        S    19:09   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host3.example.com --title-extra=CPE IPv4 H:host3_example_com Flent Test 2023-03-29T19_04_13_233854739 --output=flent_test_2023-03-29T19_04_13_233854739.png --data-dir=/root/flent/bursts/2023-03-29T19_04_13_233854739
> root       22575  0.0  0.3  12312  3564 ?        Ss   19:21   0:00 zsh /usr/local/bin/flent-test.sh
> root       22601  0.0  4.3 896456 45108 ?        Sl   19:23   5:36 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv4 H:host2_example_com Flent Test 2023-03-29T19_23_30_194006033 --output=flent_test_2023-03-29T19_23_30_194006033.png --data-dir=/root/flent/bursts/2023-03-29T19_23_30_194006033
> root       22704  0.0  2.9 884396 30600 ?        Sl   19:28   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv4 H:host2_example_com Flent Test 2023-03-29T19_23_30_194006033 --output=flent_test_2023-03-29T19_23_30_194006033.png --data-dir=/root/flent/bursts/2023-03-29T19_23_30_194006033
> root       22710  0.0  3.0 886088 31512 ?        S    19:28   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv4 H:host2_example_com Flent Test 2023-03-29T19_23_30_194006033 --output=flent_test_2023-03-29T19_23_30_194006033.png --data-dir=/root/flent/bursts/2023-03-29T19_23_30_194006033
> root       22711  0.0  3.0 886052 31832 ?        S    19:28   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv4 H:host2_example_com Flent Test 2023-03-29T19_23_30_194006033 --output=flent_test_2023-03-29T19_23_30_194006033.png --data-dir=/root/flent/bursts/2023-03-29T19_23_30_194006033
> root       22712  0.0  3.0 886088 31488 ?        S    19:28   0:00 /bin/python /usr/local/bin/flent bursts --ipv4 --plot=all_scaled --length=300 --delay=30 --host=host2.example.com --title-extra=CPE IPv4 H:host2_example_com Flent Test 2023-03-29T19_23_30_194006033 --output=flent_test_2023-03-29T19_23_30_194006033.png --data-dir=/root/flent/bursts/2023-03-29T19_23_30_194006033


Sidenotes,
- Distro info: Debian GNU/Linux 11 (bullseye)
- Kernel release (proxmox lxc): 5.15.102-1-pve
- Python 3.9.2
- lxc container is created as an unprivileged container, test process is
   running as root in the container.

-ch

[Flent-users] Re: bursts test does not complete (WAS: ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner)

[Toke Høiland-Jørgensen]

ch <ch@ntrv.dk> writes:

> Hi :wave:
>
> On 29/03/2023 19.43, ch wrote:
>> On 29/03/2023 19.42, ch wrote:
>>> But the flent test itself never terminate. The process keeps running 
>>> in the background. Resulting in the test results being saved to disk.
>> 
>> Resulting in the test results *not* being saved to disk.
>
> I fell like I am getting more gray hairs by the hours. 🤕 I cannot 
> figure out why this happens.
>
> The burst test seems to run. Flent never "completes". Resulting in no 
> data output. :|

It's crashing in the parsing step, leading to a hang; this is visible on
stdout, so I guess your runner script hides that?

$ ./run-flent bursts -l 10
Starting Flent 2.1.1+git.6959463f using Python 3.10.10.
Starting bursts test. Expected run time: 10 seconds.
Exception in thread Thread-8 (_handle_results):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 595, in _handle_results
    cache[job]._set(i, obj)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 779, in _set
    self._callback(self._value)
  File "/home/alrua/source/flent/flent/runners.py", line 625, in recv_result
    result, raw_values, metadata = res
TypeError: cannot unpack non-iterable NoneType object

Should be fixed in the latest master, please try that.

Also, BTW, not sure what your wrapper script does, but Flent has a
built-in batch run facility which may be helpful for such multi-run
tests...

-Toke

[Flent-users] Re: bursts test does not complete (WAS: ERROR: Runner Burst 0 failed check: No command set for SilentProcessRunner)

[ch]

On 30/03/2023 00.10, Toke Høiland-Jørgensen wrote:
>> I fell like I am getting more gray hairs by the hours. 🤕 I cannot
>> figure out why this happens.
>>
>> The burst test seems to run. Flent never "completes". Resulting in no
>> data output. :|
> It's crashing in the parsing step, leading to a hang; this is visible on
> stdout, so I guess your runner script hides that?

True. Switched to using systemd timers instead of crontab, saving logs 
from stdout/stderr going forwards. 💾 Had not initially thought about 
logging the output from the wrapper script. 🤦

> $ ./run-flent bursts -l 10
> Starting Flent 2.1.1+git.6959463f using Python 3.10.10.
> Starting bursts test. Expected run time: 10 seconds.
> Exception in thread Thread-8 (_handle_results):
> Traceback (most recent call last):
>    File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
>      self.run()
>    File "/usr/lib/python3.10/threading.py", line 953, in run
>      self._target(*self._args, **self._kwargs)
>    File "/usr/lib/python3.10/multiprocessing/pool.py", line 595, in _handle_results
>      cache[job]._set(i, obj)
>    File "/usr/lib/python3.10/multiprocessing/pool.py", line 779, in _set
>      self._callback(self._value)
>    File "/home/alrua/source/flent/flent/runners.py", line 625, in recv_result
>      result, raw_values, metadata = res
> TypeError: cannot unpack non-iterable NoneType object
> 
> Should be fixed in the latest master, please try that.

Installed the upgrade. 🤞

> Also, BTW, not sure what your wrapper script does, but Flent has a
> built-in batch run facility which may be helpful for such multi-run
> tests...

I wrote the wrapper script, due to not knowing about flent's batch 
capabilities initially. 🙄 Will look into the built-in batch 
capabilities instead of a wrapper script.

-ch