Hi friends,
In light of the (unplanned) shutdown of some servers, we have had to work towards migrating our infrastructure (hosting the OONI data processing pipeline) and updating the OONI PostgreSQL metaDB to adapt it to the requirements of our new data processing pipeline.
As a result, we will have to stop updating the OONI PostgreSQL metaDB ( https://github.com/ooni/sysadmin/blob/master/docs/metadb-sharing.md) WAL snapshots currently available on S3.
If you are an OONI PostgreSQL metaDB user, we apologize for the short notice, and we encourage you to read the email below for further information.
# Migration impacting the OONI PostgreSQL metaDB
We are now replacing the batch OONI data processing pipeline with our new fast-path pipeline, which analyzes and publishes OONI measurements from around the world in near real-time (the measurements are published as soon as the report is closed).
To this end, we have built a new PostgreSQL metaDB, which is powered by the latest version of PostgreSQL (11) available in debian stable, and which uses a different set of tables (based on our improved, fast-path pipeline).
While we are excited to migrate over to a new and improved pipeline (particularly since this is expected to enhance the performance of OONI Explorer!), this unfortunately also means that the current OONI PostgreSQL MetaDB, which is powered by an older version ofPostgreSQL and which depends on the batch OONI data processing pipeline, will be affected.
Specifically: If you are an OONI PostgreSQL MetaDB user, you will still have access to all previous OONI data, but you will unfortunately not receive any updates once the migration has completed (by 31st October 2020 latest).
Going forward, we recommend removing the line in your PostgreSQL configuration that does sync (as we will not be updating the WAL snapshots).
We might eventually restart exporting database tables, but by using the more modern logical replication feature of PostgreSQL 11.
If you are an OONI PostgreSQL MetaDB user and unsure how to proceed, we encourage you to reach out and share details about your use case (so that we can also factor it into our future decisions).
We will be happy to help you. You can reach us via email ( team@openobservatory.org) or via the OONI Slack channel ( https://slack.ooni.org/).
In the meanwhile, please consider accessing OONI data via the ooni-data Amazon S3 bucket: https://ooni-data.s3.amazonaws.com/
Please refer to this blog post to see how to fetch the raw data on S3: https://ooni.org/post/mining-ooni-data/
We sincerely apologise for any inconvenience this may cause.
Thanks for using the OONI PostgreSQL metaDB, and thanks for your patience throughout this process.
Please contact us with any questions you may have.
Thanks,
~ OONI team.