[tor-bugs] #30879 [Metrics/Onionperf]: Log file reprocessing shouldn't abort at first encountered line parsing error

[Tor Bug Tracker & Wiki]

#30879: Log file reprocessing shouldn't abort at first encountered line parsing
error
-----------------------------------+--------------------------
     Reporter:  karsten            |      Owner:  metrics-team
         Type:  defect             |     Status:  new
     Priority:  Medium             |  Milestone:
    Component:  Metrics/Onionperf  |    Version:
     Severity:  Normal             |   Keywords:
Actual Points:                     |  Parent ID:
       Points:                     |   Reviewer:
      Sponsor:                     |
-----------------------------------+--------------------------
 I just reprocessed a bunch of log files, some of which apparently
 containing lines that cannot be parsed. Here's the output:

 {{{
 2019-06-13 15:06:47 1560431207.323223 [onionperf] [INFO] parsing log file
 at /home/karsten/op-logs/op-nl/onionperf-data/tgen-
 client/log_archive/onionperf_2019-01-03_23:59:59.tgen.log
 2019-06-13 15:06:47 1560431207.668200 [onionperf] [INFO] parsing log file
 at /home/karsten/op-logs/op-nl/onionperf-data/tor-
 client/log_archive/onionperf_2018-05-18_23:59:59.torctl.log
 2019-06-13 15:06:47 1560431207.929735 [onionperf] [INFO] parsing log file
 at /home/karsten/op-logs/op-nl/onionperf-data/tor-
 client/log_archive/onionperf_2018-10-25_23:59:59.torctl.log
 2019-06-13 15:06:47 1560431207.971164 [onionperf] [WARNING] TGenParser:
 skipping line due to parsing error: 2018-04-14 21:10:04 1523740204.809894
 [message] [shd-tgen-transfer.c:803] [_tgentransfer_log] [transfer-error]
 transport
 TCP,17,NULL:37.218.247.40:26006,NULL:0.0.0.0:0,146.0.73.4:146.0.73.4:1313,state=SUCCESS,error=NONE
 transfer (null),26847,op-nl,NONE,0,(null),0,state=ERROR,error=AUTH total-
 bytes-read=1 total-bytes-write=0 payload-bytes-write=0/0 (-nan%) usecs-to-
 socket-create=0 usecs-to-socket-connect=8053676879205 usecs-to-proxy-
 init=-1 usecs-to-proxy-choice=-1 usecs-to-proxy-request=-1 usecs-to-proxy-
 response=-1 usecs-to-command=-1 usecs-to-response=-1 usecs-to-first-
 byte=-1 usecs-to-last-byte=-1 usecs-to-checksum=-1

 2019-06-13 15:06:47 1560431207.971545 [onionperf] [INFO] Analysing pair
 for date 2018-01-27 00:00:00
 2019-06-13 15:06:47 1560431207.971635 [onionperf] [INFO] parsing log file
 at /home/karsten/op-logs/op-nl/onionperf-data/tgen-
 client/log_archive/onionperf_2018-01-27_23:59:59.tgen.log
 }}}

 It seems like commenting out a `raise` statement fixes this:

 {{{
 diff --git a/onionperf/analysis.py b/onionperf/analysis.py
 index 0bc1811..08f4e81 100644
 --- a/onionperf/analysis.py
 +++ b/onionperf/analysis.py
 @@ -532,7 +532,7 @@ class TGenParser(Parser):
                      break
              except:
                  logging.warning("TGenParser: skipping line due to parsing
 error: {0}".format(line))
 -                raise
 +                #raise
                  continue
          source.close()
 }}}

 Is this the correct fix? If so, would it make sense to add a unit test to
 test this behavior?

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/30879>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online