On 2020-12-02 20:23, George Kadianakis wrote:
Karsten Loesing karsten@torproject.org writes:
On 2020-11-03 15:01, George Kadianakis wrote:
Hello Karsten,
Hi George!
hope you are doing well!
I've been working on the S61 performance experiments [0] and I would appreciate some help with onionperf.
<snip>
Then for another onionperf run (attached as 'onionperf-44028.json.xz') it gave me a different error:
$ onionperf visualize --data onionperf.analysis.json.xz "Test Measurements" 2020-11-03 15:50:03 1604411403.946028 [onionperf] [INFO] loading analysis results from /user/tmp/onionperf/analysis/onionperf.analysis.json.xz 2020-11-03 15:50:03 1604411403.976088 [onionperf] [INFO] done! Traceback (most recent call last): File "/user/.local/bin/onionperf", line 4, in <module> __import__('pkg_resources').run_script('OnionPerf==0.8', 'onionperf') File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 650, in run_script self.require(requires)[0].run_script(script_name, ns) File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1453, in run_script exec(script_code, namespace, namespace) File "/user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/EGG-INFO/scripts/onionperf", line 622, in <module> File "/user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/EGG-INFO/scripts/onionperf", line 382, in main File "/user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/EGG-INFO/scripts/onionperf", line 522, in visualize File "/user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/onionperf/visualization.py", line 38, in plot_all File "/user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/onionperf/visualization.py", line 129, in __extract_data_frame KeyError: '44028'
Because of the two errors above I'm unable to proceed with producing any results. Any idea what I might be doing wrong? Is there any other data you might need to help me with this?
In this case it's failing even before writing a CSV file. Still, when looking at the .xz file I'm seeing only errors, too, so it's a related issue here.
Hello again Karsten,
Hi George!
your help was valuable last time and I managed to overcome those issues.
I've been doing more experiments recently (see https://gitlab.torproject.org/tpo/core/tor/-/issues/40157#note_2716591) and while the data collection is still ongoing, I decided to do some visualizations to make sure that I'm not collecting useless data all this time.
So I took collected onionperf data but when I tried to visualize them I got a similar error like the one from my previous email:
""" $ onionperf visualize --data onionperf.analysis.json.xz "bla" 2020-12-02 21:19:04 1606936744.475013 [onionperf] [INFO] loading analysis results from /home/user/tmp/onionperf/onionperf-data-2558933-close/onionperf.analysis.json.xz 2020-12-02 21:19:05 1606936745.245664 [onionperf] [INFO] done! Traceback (most recent call last): File "/home/user/.local/bin/onionperf", line 4, in <module> __import__('pkg_resources').run_script('OnionPerf==0.8', 'onionperf') File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 650, in run_script self.require(requires)[0].run_script(script_name, ns) File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1453, in run_script exec(script_code, namespace, namespace) File "/home/user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/EGG-INFO/scripts/onionperf", line 622, in <module> File "/home/user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/EGG-INFO/scripts/onionperf", line 382, in main File "/home/user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/EGG-INFO/scripts/onionperf", line 522, in visualize File "/home/user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/onionperf/visualization.py", line 38, in plot_all File "/home/user/.local/lib/python3.8/site-packages/OnionPerf-0.8-py3.8.egg/onionperf/visualization.py", line 129, in __extract_data_frame KeyError: '36178' """
However this time the onionperf.analysis.json.xz file looks more reasonable. I'm attaching it to this email just in case you have the time to check it out and tell me what might be going wrong.
This is related to the issues you saw before. Please try out the patch that I wrote for #40012 last week for the visualization part (no need to re-do the measurements):
$ git remote add karsten https://gitlab.torproject.org/karsten/onionperf.git
$ git fetch karsten
$ git checkout -b task-40012 karsten/task-40012
If you want, please comment on the ticket whether this worked for you!
All the best, Karsten
FWIW, the data was collected in a manner similar to this: $ onionperf measure -i --tgen ./tgen --tor ./tor --torclient-conf-file ./torrc --tgen-connect-ip 1.2.3.4 --tgen-listen-port 443 --tgen-connect-port 443 --tgen-pause-initial 20 --tgen-pause-between 2 --tgen-transfer-size '50 KiB'
Thanks a lot!