Hello,
Happy new year, and happy new decade!
## OONI is an independent entity
As of 1st January 2020, OONI is an independent entity!
Back in 2011, OONI was born out of the Tor Project, which has since served as our fiscal sponsor. Thanks to Tor Project support, OONI was able to grow and become what it is today. We are immensely grateful to the Tor Project for all their support over the last 8 years, and we look forward to continuing to collaborate on the fight against internet censorship!
As part of our growth, OONI is transitioning to greater autonomy and independence. As of 1st January 2020, OONI is fiscally sponsored by the Hermes Center for Transparency and Digital Human Rights (https://www.hermescenter.org/), a non-profit, digital rights organization based in Italy, which was co-founded by Arturo Filastò (OONI’s co-founder) back in 2011. OONI’s Maria Xynou and Simone Basso are also members of the Hermes Center. We may eventually create a dedicated-to-OONI NGO, but for now, the current arrangement is sufficient.
## New project management methods
As of December 2019, we have changed how we do project management in our team.
We now work in 2-week sprint cycles, based on which we plan and track the progress of our activities. So far, this works really well!
We are using Zenhub boards to plan and estimate our activities, all of which are public, but require registration with the app.
Here is our Sprint Planning board: https://app.zenhub.com/workspaces/sprint-planning-5dfa4b7355438f7e86c80e5a/b...
Even if the internal planning is behind a registration wall, all of our activities and priorities are still tracked using native GitHub tooling, making it possible for anyone to see what our priorities are via the public GitHub interface.
We have also consolidated our GitHub issue repositories to only 9 in total:
* https://github.com/ooni/ooni.org * https://github.com/ooni/backend * https://github.com/ooni/probe * https://github.com/ooni/probe-engine * https://github.com/ooni/explorer * https://github.com/ooni/run * https://github.com/ooni/design-system * https://github.com/ooni/orchestra
This ticket shares more details: https://github.com/ooni/ooni.org/issues/317
To see what we are working on over the next 2 weeks for a given project, just select the next milestone of that repository on GitHub.
Each issue should also have a priority label assigned to it and an effort estimate label.
Each OONI team sprint has a marine-inspired name, and we encourage community members to propose names for our sprints here: https://github.com/ooni/ooni.org/issues/337
## OONI Probe support for running circumvention tool tests
During January 2020, we almost completed the implementation of all client-side logic that is required for testing Psiphon and Tor inside of OONI Probe.
To add support for running Tor, we:
* Wrote a first iteration of the Tor test: https://github.com/ooni/probe-engine/issues/226 * Added support for the Tor bridge reachability test in probe-engine: https://github.com/ooni/probe-engine/issues/2 * Added support for running obfs4proxy in probe-engine: https://github.com/ooni/probe-engine/issues/90 * Designed how the Tor test results should be presented in the OONI Probe desktop app: https://github.com/ooni/probe/issues/967 * Made a release of probe-cli with support for the Tor test experiment: https://github.com/ooni/probe/issues/974 * Exposed top level keys to expose top level statistics in the UI: https://github.com/ooni/probe-engine/issues/292 * Almost completed the UI work required to show the Tor test results in the OONI Probe app: https://github.com/ooni/probe/issues/975
To add support for running Psiphon, we:
* Finalized support for running Psiphon in probe-engine: https://github.com/ooni/probe-engine/issues/89 * Finished integrating the Psiphon test results with the correct Psiphon icon in the OONI Probe Desktop App * Designed the UI for the circumvention tests in general: https://github.com/ooni/probe/issues/804
## OONI Probe orchestration logic for circumvention tool testing
In order to support measuring Psiphon and Tor, OONI Probe clients need to know which configurations they should use to carry out the testing. We therefore added support for giving out such testing targets (Psiphon configuration, Tor default bridges, Tor directory authorities) via authenticated backend calls.
Specifically, we:
* Added support for giving out Psiphon configurations: https://github.com/ooni/orchestra/issues/78 * Added support for giving out addresses for Tor tests: https://github.com/ooni/orchestra/issues/82
## Making OONI Probe reporting logic more resilient to censorship
We made a bit of progress in understanding how we can make OONI Probe testing more resilient to blocking and we added extra robustness to the app to overcome potential blocking attempts.
Specifically, we:
* Improved how we display measurements which have failed due to a bouncer failure: https://github.com/ooni/probe/issues/855 * Discussed solutions for what we should do when the IP lookup of a measurement fails: https://github.com/ooni/probe/issues/879 * Investigated how to allow alternative methods of sending measurement results to us via email (https://github.com/ooni/probe/issues/992) or IM apps (https://github.com/ooni/probe/issues/976)
## Analysis of circumvention tool test data and presentation in OONI Explorer
We made a lot of progress with respect to presenting the new Psiphon and Tor test results inside of OONI Explorer and exposing them via the OONI API. We also worked on the scoring and analysis of the results in the data processing pipeline.
Specifically, we:
* Added support in the batch pipeline for scoring Tor test results: https://github.com/ooni/backend/issues/293 * Wrote data processing logic for processing Psiphon test results: https://github.com/ooni/backend/issues/209 * Re-wrote the OONI API list_measurements API call, adding support for listing and searching for Psiphon and the new Tor test results and boosting its performance: https://github.com/ooni/backend/issues/160 * Almost finished adding support for presenting Psiphon test results in OONI Explorer: https://github.com/ooni/explorer/issues/332 * Made considerable progress on presenting Tor test results in OONI Explorer: https://github.com/ooni/explorer/issues/332
To better tune the analysis engine, we need to ship the tests and increase the measurement coverage.
## OONI data processing pipeline
In November 2019, we started publishing OONI measurements from around the world in near real-time (i.e. within minutes). This, however, had an impact on the performance of the OONI API (which OONI Explorer relies on).
To boost the performance of the OONI API (and, by extension, the performance of OONI Explorer), we made a lot of progress on the OONI data processing pipeline, all of which can be viewed through the following tickets:
https://github.com/ooni/backend/issues/161 https://github.com/ooni/backend/issues/163 https://github.com/ooni/backend/issues/160
Throughout January 2020, we added many blockpage fingerprints to the fast-path pipeline, making it possible to automatically confirm many more cases of internet censorship. See: https://github.com/ooni/pipeline/pull/289
We also experimented with adding support for calculating some statistics of measurements and anomalies. See: https://github.com/ooni/backend/issues/292
Moreover, we added support in the fast-path pipeline for anomaly, failure, and confirmed columns. We also worked on adding support for reconnections during failing SSH conditions, as we had noticed issues with this.
We added support for ingesting the Citizen Lab category codes for URLs, in order to eventually support filtering URLs by category on OONI Explorer. This work is available here: https://github.com/ooni/backend/issues/231
We also discussed how we are going to move the development of the OONI data processing pipeline forward and came up with a plan for that, as documented in the following tickets:
https://github.com/ooni/backend/issues/195 https://github.com/ooni/backend/issues/286
## OONI Probe mobile app
During January 2020, we added support for setting no limit to the maximum runtime.
It is now possible for OONI Probe mobile app users to set the Websites test max_runtime to zero in order to test entire test lists.
See: https://github.com/ooni/probe/issues/893
We also made several important bug fixes related to how certain error conditions are being handled. See: https://github.com/ooni/probe/issues/865 & https://github.com/ooni/probe/issues/926
## OONI Probe measurement engine
We introduced timing and TLS in the testing (and other low-level data collection), to ensure that OONI Probe tests are as useful as possible. Essentially, OONI Probe tests that can make use of this low-level information do so. This is an important enhancement (and an essential building block) to the Tor test and other tests.
## Data analysis
We analyzed OONI measurements for the following:
* Azerbaijan Internet Watch (partner): https://github.com/ooni/ooni.org/issues/332 * OTF Information Controls Fellow (hosted under OONI): https://github.com/ooni/ooni.org/issues/331 * Research on the blocking of LGBTQI sites worldwide: https://github.com/ooni/backend/issues/224
## OONI backend
We only had one major new incident to deal with in January 2020, which is documented here: https://github.com/ooni/sysadmin/issues/422
We had a relapse of an old incident (https://github.com/ooni/sysadmin/issues/128) and we worked on coming up with a fix to this here: https://github.com/ooni/backend/issues/300
## Published OONI FAQ
We published a Frequently Asked Questions (FAQ) section on our website: https://ooni.org/support/faq/
Through the OONI FAQ, we aim to address the most frequently asked questions that we have received from community members over the last years.
The questions fall under the following areas:
* About OONI * OONI Probe * Testing websites * OONI data * OONI Explorer
We aim to continue to update the OONI FAQ on an ongoing basis, and we encourage further community feedback.
## Published OONI Glossary
We published an OONI Glossary on our website: https://ooni.org/support/glossary
This glossary is meant to cover terms that are commonly used in the OONI-verse, within OONI apps, documentation, and research reports. As a result, this glossary is quite OONI-specific.
That said, it also includes a variety of terms that are commonly used by many other projects and may therefore be useful to others in the broader internet freedom community.
Similarly to the OONI FAQ, we aim to continue to improve upon and update the OONI Glossary on an ongoing basis. We encourage further community feedback, and suggestions can be shared directly on GitHub: https://github.com/ooni/ooni.org
## Collaboration with Netalitica
We reviewed the test list updates by Netalitica researchers and opened pull requests for the following test lists: * Turkmenistan: https://github.com/citizenlab/test-lists/pull/552
* Uzbekistan: https://github.com/citizenlab/test-lists/pull/553
* Jordan: https://github.com/citizenlab/test-lists/pull/554 * China: https://github.com/citizenlab/test-lists/pull/555
We also reviewed Netalitica updates to the Russian (https://github.com/citizenlab/test-lists/pull/557), Turkish and Iranian (https://github.com/citizenlab/test-lists/pull/564) test lists.
## Organization of IFF Internet Measurement Village
This year OONI is co-organizing an Internet Measurement Village during the last 2 days of the Internet Freedom Festival (IFF) in Valencia, Spain.
We reached out to several other measurement projects, inviting them to co-host the Internet Measurement Village with us.
Throughout January 2020, we coordinated with the IFF organizers and with the other Internet Measurement Village members on logistics, and we organized and hosted two brainstorming calls. We also collected session ideas from community members, based on which we are creating the agenda for the Internet Measurement Village.
## Organization of OONI Partner Gathering
In July 2017, we organized and hosted the first OONI Partner Gathering: a two-day event that brought all of our partners from around the world together to share skills and knowledge around censorship measurement.
We plan to host the next OONI Partner Gathering in 2020! To this end, we wrote and submitted funding proposals, organized relevant logistics, and worked on updating relevant budgets.
## Community activities
### OONI presentation at the Nexa Center
On 22nd January 2020, Arturo presented OONI and discussed censorship findings from 2019 at the Nexa Center for Internet and Society in Turin, Italy.
Information about his presentation is available here: https://nexa.polito.it/lunch-75
### Community meeting
We facilitated the monthly OONI Community Meeting on 28th January 2020 on our Slack channel (https://slack.ooni.org/), during which we discussed: 1. Updates from the OONI team
2. Setting up a proxy for OONI Probe Android and iOS
3. Maxmind licensing problems
## Userbase
In January 2020, OONI Probe was run 8,142,429 times from 5,307 different vantage points in 206 countries around the world.
This information can also be found through our measurement stats on OONI Explorer (see chart on monthly coverage worldwide): http://explorer.ooni.org/
~ The OONI team.
tor-project@lists.torproject.org