[tor-bugs] #5875 [Ooni]: Write progress report to date.

Tor Bug Tracker & Wiki blackhole at torproject.org
Tue Jan 22 11:17:04 UTC 2013 

#5875: Write progress report to date.
-----------------------+----------------------------------------------------
    Reporter:  phobos  |       Owner:  phobos        
        Type:  task    |      Status:  closed        
    Priority:  normal  |   Milestone:                
   Component:  Ooni    |     Version:                
  Resolution:  fixed   |    Keywords:  SponsorH201208
      Parent:          |      Points:                
Actualpoints:          |  
-----------------------+----------------------------------------------------
Changes (by hellais):

  * status:  new => closed
  * resolution:  => fixed


Comment:

 ## August

 The month of August was dedicated to laying out the foundations and the
 overall architecture of the ooniprobe backend and client system.

 * Work on daphn3

    * https://github.com/hellais/ooni-
 probe/commit/818e4603d4d2a91f033b013db2b3dfecc5c99114

    * https://github.com/hellais/ooni-
 probe/commit/14ade9b22b2d91dc3c85a7d246c76cc6bc935d22

 * Experiment with the idea of a eigenvector based classifier:
 https://github.com/hellais/ooni-
 probe/commit/346cd4e6cd1b028bb5f772b1a0bf15c2b2e12ae3
 https://github.com/hellais/ooni-
 probe/commit/78fe6dea0b46c4055fbc195cf9329001c062cb17

 * Start laying out the structure for OONIB reporting system:
 https://github.com/hellais/ooni-
 probe/commit/1d8240c746ec79381aafcb3d921568c90b70889b

 * Do research on existing censorship lists and in particular import the
 one from greatfire: https://github.com/hellais/ooni-
 probe/commit/190e24e5c3239322900c63f8a864b375cdd6320e


 ## September

 This month was mainly aimed at designing an API that would be flexible
 enough for supporting the writing of tests easily from third party
 developers.

 * Studied existing solutions to the problem of developing an API for
 developing tests. Of particular usefulness was nose
 (https://nose.readthedocs.org/en/latest/) and twisted trial
 (http://twistedmatrix.com/trac/wiki/TwistedTrial)

 * Implemented the ooniprobe client API based on twisted trial:

 https://github.com/hellais/ooni-
 probe/commit/98ebfe5e98824b4343a7a03f950f70e8f1c30faf
 https://github.com/hellais/ooni-
 probe/commit/2d31d5f14a0f3a7518d47fe748c1677cbd885828
 https://github.com/hellais/ooni-
 probe/commit/712f665423f24701c93889d32d040eb533065dfe
 https://github.com/hellais/ooni-
 probe/commit/cbfcbdd0344d8e9a80d565ba01b64c7a40c97352
 https://github.com/hellais/ooni-
 probe/commit/7dce07bfc58566d2fdec4eec8a831d4ecd545d5f

 * Write documentation for the API:

 https://github.com/hellais/ooni-
 probe/commit/839564d3620c24e0057c1dfd9d39ef15f8b202ac
 https://github.com/hellais/ooni-
 probe/commit/ffc44ce1cbc595ce8cbcd6522e8962f37b7e6869

 * Implement Scapy test template. This allows developers to write tests
 using the Scapy packet injection framework:

 https://github.com/hellais/ooni-
 probe/commit/36c8a8e149fa5d975fc0d39bd0e5bf72ab38086a

 * Find bugs in YAML, fix them:
 https://github.com/hellais/ooni-
 probe/commit/532e2e9853b4bdd61c951dc3073d8dc68d35f754

 * Add timeout support to scapy test template:
 https://github.com/hellais/ooni-
 probe/commit/b267f3bd882fb5da6aa20c9a106ff3a10630e4f9

 * Outline what is necessary for having database support in oonibackend:
 https://github.com/hellais/ooni-
 probe/commit/b1ca4257c13dff18664c8b175ed6b57acc8874b3

 * Add basic SSL support in our test helper:
 https://github.com/hellais/ooni-
 probe/commit/ae87c2b74844372586d9c863ab63bb5f5e6bec38

 ## October

 * Get the captive portal test to work:

 https://github.com/hellais/ooni-
 probe/commit/5cdc37ff80f1c8343e26f94fd7b7a29b3081ed03

 * Rewrite DNS tamper test to use twisted functions instead of default
 python ones:
 https://github.com/hellais/ooni-
 probe/commit/c904e684ac60f9c25c09d52251cb27966aaab065

 * Add support for getting the probes IP address via check.tpo:
 https://github.com/hellais/ooni-
 probe/commit/67b024a0361a0585ebc3bf9ed615ef8bf7c793fd

 * Work on removing dead code and refactoring:
 https://github.com/hellais/ooni-
 probe/commit/55ce0ae91ff67c8212f80db783a53842e7662198

 * Multiple iterations over getting the documentation ready and the tests
 working as they should


 ## November

 November was dedicated to making sure that the design layer out and
 experimented with in the past months worked properly. Focus was placed on
 the ooniprobe command line tool. The ooniprobe backend was made to do the
 basic set of tasks it needed to support the running of tests.

 Highlights:

 * Alpha support for logging to PCAP

 * Alpha support for starting and stopping of Tor

 * Alpha support for reporting to Tor Hidden Services

 * Revert to a state in which everything was not broken and attempt to port
 Bridget
 https://github.com/hellais/ooni-
 probe/commit/de0c47f7152b3a0a9dd2baf27ce2b028819e3f63

 * Make the DNS Tamper test use the a lower, more flexible twisted API
 call:
 https://github.com/hellais/ooni-
 probe/commit/769cd3cbbd8521af505f2b740af8468b1a347f2f

 * Work on the backend system database support:
 https://github.com/hellais/ooni-
 probe/commit/bf37eed44600ea9d307f958e35304ff4a7116032

 * Drop support for the old API:
 https://github.com/hellais/ooni-
 probe/commit/654147c3a92156c35fc1a8ecc0aac8e933f57e59

 * Add GeoIP Lookup support for determining the probes ASN number and
 country:
 https://github.com/hellais/ooni-
 probe/commit/e056fe9b886c3dfaf9546cd94656eec8117f70f8
 https://github.com/hellais/ooni-
 probe/commit/e10033c72b4cf3355095b3a0bbdfca3a5dbeac4a

 * Improve the scapy test template:
 https://github.com/hellais/ooni-
 probe/commit/f4a065da96af966e01126be4c3013780da1b7579

 * Iterations over existing tests:
 https://github.com/hellais/ooni-
 probe/commit/9bf7fc8d86901bca4061dd82f6ca6af311cabee4

 * Add basic basic capture support by spawning a thread:
 https://github.com/hellais/ooni-
 probe/commit/9e632c5d7c8d7f73add3aa38900898a86f6ca9fc
 https://github.com/hellais/ooni-
 probe/commit/6a3ee55b574adaa8740ccafe2e4a01719dc0e86e

 * Improve txscapy so that it uses a valid file descriptor instead of
 spawning a thread:
 https://github.com/hellais/ooni-
 probe/commit/12726ca463d9e68e93d49fcb418421648d054744

 * Config file support for oonib:
 https://github.com/hellais/ooni-
 probe/commit/d4adf3e0e177e80ce7bc825f1cb4e461e5551b2f

 * Engage the community in contributing code:
 https://github.com/hellais/ooni-
 probe/commit/28241124e6094b224f4a3b4f6c0a8e8a69a7eeb6
 https://github.com/hellais/ooni-
 probe/commit/c67d69ec432e32479daca2ed65b27529260c9ba5

 * Recuperate the UK mobile network test that was implemented some time ago
 (https://trac.torproject.org/projects/tor/ticket/6437) for the Open Rights
 Group:
 https://github.com/hellais/ooni-
 probe/commit/e115d4d0e6c063422fc35cbe1a63dcf0f5917a77

 * First iteration over Tor Hidden Service support for oonib and ooniprobe
 https://github.com/hellais/ooni-
 probe/commit/d0d9662b4cafe11979e8173e6cc03c548f99ecf0


 * First iteration over TCP Test template. This allows to specify a TCP
 payload from a flat file and it will get sent to the destination address
 and the response will be recorded.
 https://github.com/hellais/ooni-
 probe/commit/f873cd780a37451810b04526065fbe90d6250ba7

 * Work with somebody in Turkey to get them to run some tests.

 * Implement DNS Spoofing test:
 https://github.com/hellais/ooni-
 probe/commit/4208c6e963ae59380999fae45930656c287a627d

 * First iteration over resume support and test state tracking:
 https://github.com/hellais/ooni-
 probe/commit/659dda6b388ecc658f8060b7bef2aa7b4015dce2

 * Add support for starting Tor via txtorcon:
 https://github.com/hellais/ooni-
 probe/commit/a65b6ebb7e90a4350b1a214189196d11a8371eda

 * Make the scapy sniffer not run a separate thread, but be non blocking.
 Get inspiration from the work done by Paul Bohm for muxTCP
 (http://events.ccc.de/congress/2005/fahrplan/events/529.en.html)
 https://github.com/hellais/ooni-
 probe/commit/615ce75c47aec249b6b5a4c0b58fdf7a93f09582

 * Get aagbsn up to speed on ooniprobe.

 * Implement HTTP Request Line "fuzzer":
 https://github.com/hellais/ooni-
 probe/commit/1de07f659f1393d969a1b3766baffeecb111355d

 and document it:

 https://github.com/hellais/ooni-
 probe/commit/d67eecfd05f03002e3a9334db82ab2d8c2658368

 * Add support for ooniprobe test decks:
 https://github.com/hellais/ooni-probe/commit/263568f

 * Work on documentation for various tests.

 * Lot's of refactoring:
 https://github.com/hellais/ooni-
 probe/commit/10c63e0fceb0479c5893bfd2cb7e77af2af703cc
 https://github.com/hellais/ooni-
 probe/commit/7d6901f1552067bce9595db6a84f8f5245d8f28c


 ## December

 December has started as a month of craziness and me and aagbsn have been
 working very hard to make sure that we reach the 0.1 in time for the
 expected date.

 In particular we have started data collection from some machines we have
 rented.

 * Get aagbsn up to speed with ooniprobe and guide him through the things
 that need to be done

 * Buy some machines for running of tests

 * Coordinate the migration of ooni.nu to ooni.tpo
 https://trac.torproject.org/projects/tor/ticket/7644

 * Talk with the crate.io developer to get a Tor hidden service running to
 deliver us safe code via pip.

 * Iterations over the bugs that aaron encountered while running the code:
 https://github.com/hellais/ooni-
 probe/commit/18c2dbd0b0491b25e997214fef672a8ebe1b2312

 * Work with twisted developers of a better SOCKS5/4/4a implementation:
 https://github.com/hellais/txsocksx

 https://github.com/hellais/txsocksx/commit/72475c35db3441c65de243019615e85ac40a476a
 https://github.com/hellais/txsocksx/commit/fbd0a68be25acc0d6b891f558737320fe22ff375
 https://github.com/hellais/txsocksx/commit/9fb250f712a013fbafb843285795524b868c1688
 https://github.com/hellais/txsocksx/commit/f542f4d9b28f6690c0359a2d9a6b10d179d1872e
 https://github.com/hellais/txsocksx/commit/dcea1b008b57523e02d0eeecc4c750e671847be2

 Note: this is not yet bundled with ooniprobe and at this point will
 probably only be done post 0.1

 * Normalize the report format so that it follows ISO8601:
 https://github.com/hellais/ooni-
 probe/commit/fb477a0982247974bc5f5979239ed015a17ce3ec
 https://github.com/hellais/ooni-
 probe/commit/4648dd50aa44066eec9de1526f08a2e517ea14fb

 * Setting up of the machines for the collector and test helper

 * Documentation and quality assurance for the 0.1 release:
 https://github.com/hellais/ooni-probe/pull/9
 https://github.com/hellais/ooni-probe/pull/8
 https://github.com/hellais/ooni-probe/pull/10
 https://github.com/hellais/ooni-probe/pull/11
 https://github.com/hellais/ooni-probe/pull/13
 https://github.com/hellais/ooni-probe/pull/14
 https://github.com/hellais/ooni-probe/pull/15

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/5875#comment:3>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online

More information about the tor-bugs mailing list