[metrics-bugs] #29787 [Metrics/Onionperf]: Enumerate possible failure cases and include failure information in .tpf output

[Tor Bug Tracker & Wiki]

#29787: Enumerate possible failure cases and include failure information in .tpf
output
-------------------------------+------------------------------
 Reporter:  karsten            |          Owner:  metrics-team
     Type:  enhancement        |         Status:  new
 Priority:  Medium             |      Milestone:
Component:  Metrics/Onionperf  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+------------------------------

Comment (by karsten):

 Now, when I remove those three lines and re-run the script, I get the
 following error:

 {{{
 $ python3 op_errors.py --path jsonfiles/
 Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/pandas/indexes/base.py", line 2134,
 in get_loc
     return self._engine.get_loc(key)
   File "pandas/index.pyx", line 132, in pandas.index.IndexEngine.get_loc
 (pandas/index.c:4433)
   File "pandas/index.pyx", line 154, in pandas.index.IndexEngine.get_loc
 (pandas/index.c:4279)
   File "pandas/src/hashtable_class_helper.pxi", line 732, in
 pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13742)
   File "pandas/src/hashtable_class_helper.pxi", line 740, in
 pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13696)
 KeyError: 'failure_reason_local_y'

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "op_errors.py", line 91, in <module>
     main()
   File "op_errors.py", line 85, in main
     all_csv_data["failure_reason_local"] =
 all_csv_data["failure_reason_local_y"]
   File "/usr/lib/python3/dist-packages/pandas/core/frame.py", line 2059,
 in __getitem__
     return self._getitem_column(key)
   File "/usr/lib/python3/dist-packages/pandas/core/frame.py", line 2066,
 in _getitem_column
     return self._get_item_cache(key)
   File "/usr/lib/python3/dist-packages/pandas/core/generic.py", line 1386,
 in _get_item_cache
     values = self._data.get(item)
   File "/usr/lib/python3/dist-packages/pandas/core/internals.py", line
 3543, in get
     loc = self.items.get_loc(item)
   File "/usr/lib/python3/dist-packages/pandas/indexes/base.py", line 2136,
 in get_loc
     return self._engine.get_loc(self._maybe_cast_indexer(key))
   File "pandas/index.pyx", line 132, in pandas.index.IndexEngine.get_loc
 (pandas/index.c:4433)
   File "pandas/index.pyx", line 154, in pandas.index.IndexEngine.get_loc
 (pandas/index.c:4279)
   File "pandas/src/hashtable_class_helper.pxi", line 732, in
 pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13742)
   File "pandas/src/hashtable_class_helper.pxi", line 740, in
 pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13696)
 KeyError: 'failure_reason_local_y'
 }}}

 This error seems to go away if I `unxz` the single file contained in my
 `jsonfiles/` directory. It would be nice if the script either handles xz-
 compressed files or ignores them (maybe after printing out a warning) and
 complains about not having any not-xz-compressed input files to process.

 However, after `unxz`-ing the input file, I get this error:

 {{{
 $ python3 op_errors.py --path jsonfiles/
 Traceback (most recent call last):
   File "op_errors.py", line 91, in <module>
     main()
   File "op_errors.py", line 53, in main
     tgen_dfObj = pd.DataFrame.from_dict(tgen_data, orient="index",
 columns= ["endpoint_local", "error_code", "endpoint_remote", "is_error",
 "total_bytes_read", "transfer_id", "unix_ts_end", "unix_ts_start",
 "elapsed_seconds"])
 TypeError: from_dict() got an unexpected keyword argument 'columns'
 }}}

 Which Git repository is that where you're developing this script? I'd like
 to pull from that rather than editing the script file I have here.

 Thanks!

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29787#comment:10>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online