<div dir="ltr"><div id="gmail-magicdomid8" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">Hi friends,</span></div><div id="gmail-magicdomid9" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid10" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">In light of the (unplanned) shutdown of some servers, we have had to work towards migrating our infrastructure (hosting the OONI data processing pipeline) and updating the OONI PostgreSQL metaDB to adapt it to the requirements of our new data processing pipeline.</span></div><div id="gmail-magicdomid11" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid12" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">As a result, we will have to </span><span class="gmail-b" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px"><span style="box-sizing:border-box;margin:0px;padding:0px;font-weight:700">stop updating the OONI PostgreSQL metaDB </span></span><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">(</span><span class="gmail-url" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px"><a href="https://github.com/ooni/sysadmin/blob/master/docs/metadb-sharing.md)" rel="noreferrer noopener" style="box-sizing:border-box;margin:0px;padding:0px;background-color:transparent">https://github.com/ooni/sysadmin/blob/master/docs/metadb-sharing.md)</a></span><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px"> WAL snapshots currently available on S3. </span></div><div id="gmail-magicdomid13" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid14" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">If you are an OONI PostgreSQL metaDB user, we apologize for the short notice, and we encourage you to read the email below for further information.</span></div><div id="gmail-magicdomid15" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid16" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-b" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px"><span style="box-sizing:border-box;margin:0px;padding:0px;font-weight:700"># Migration impacting the OONI PostgreSQL metaDB</span></span></div><div id="gmail-magicdomid17" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid48" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">We are now replacing the batch OONI data processing pipeline with our new fast-path pipeline, which analyzes and publishes OONI measurements from around the world in near real-time (the measurements are published as soon as the report is closed).</span></div><div id="gmail-magicdomid19" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid49" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">To this end, we have built a new PostgreSQL metaDB, which is powered by the latest version of PostgreSQL (11) available in debian stable, and which uses a different set of tables (based on our improved, fast-path pipeline). </span></div><div id="gmail-magicdomid21" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid22" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">While we are excited to migrate over to a new and improved pipeline (particularly since this is expected to enhance the performance of OONI Explorer!), this unfortunately also means that the current OONI PostgreSQL MetaDB, which is powered by an older version of</span><span class="gmail-author-a-z88zz76ziobz122zsz74z2z89zz122zitpfz67z" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px;background-color:rgb(217,169,169)"></span><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">PostgreSQL and which depends on the batch OONI data processing pipeline, will be affected.</span></div><div id="gmail-magicdomid23" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid55" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">Specifically: </span><span class="gmail-b" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px"><span style="box-sizing:border-box;margin:0px;padding:0px;font-weight:700">If you are an OONI PostgreSQL MetaDB user, you will still have access to all previous OONI data, but you will unfortunately not receive any updates once the migration has completed (by 31st October 2020 latest).</span></span></div><div id="gmail-magicdomid25" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid26" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">Going forward, we recommend </span><span class="gmail-b" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px"><span style="box-sizing:border-box;margin:0px;padding:0px;font-weight:700">removing the line in your PostgreSQL configuration that does sync</span></span><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px"> (as we will not be updating the WAL snapshots).</span></div><div id="gmail-magicdomid27" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid28" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">We might eventually restart exporting database tables, but by using the more modern logical replication feature of PostgreSQL 11.</span></div><div id="gmail-magicdomid29" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid30" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">If you are an OONI PostgreSQL MetaDB user and unsure how to proceed, we </span><span class="gmail-b" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px"><span style="box-sizing:border-box;margin:0px;padding:0px;font-weight:700">encourage you to reach out and share details about your use case</span></span><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px"> (so that we can also factor it into our future decisions). </span></div><div id="gmail-magicdomid31" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid32" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-b" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px"><span style="box-sizing:border-box;margin:0px;padding:0px;font-weight:700">We will be happy to help you.</span></span><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px"> You can reach us via email (<a href="mailto:team@openobservatory.org">team@openobservatory.org</a>) or via the OONI Slack channel (</span><span class="gmail-url" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px"><a href="https://slack.ooni.org/)" rel="noreferrer noopener" style="box-sizing:border-box;margin:0px;padding:0px;background-color:transparent">https://slack.ooni.org/)</a></span><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">.</span></div><div id="gmail-magicdomid33" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid34" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">In the meanwhile, please consider accessing OONI data via the ooni-data Amazon S3 bucket: </span><span class="gmail-url" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px"><a href="https://ooni-data.s3.amazonaws.com/" rel="noreferrer noopener" style="box-sizing:border-box;margin:0px;padding:0px;background-color:transparent">https://ooni-data.s3.amazonaws.com/</a></span><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px"> </span></div><div id="gmail-magicdomid35" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid36" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">Please refer to this blog post to see how to fetch the raw data on S3:</span><span class="gmail-url" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px"><a href="https://ooni.org/post/mining-ooni-data/" rel="noreferrer noopener" style="box-sizing:border-box;margin:0px;padding:0px;background-color:transparent">https://ooni.org/post/mining-ooni-data/</a></span></div><div id="gmail-magicdomid37" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid38" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">We sincerely apologise for any inconvenience this may cause.</span></div><div id="gmail-magicdomid39" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid40" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">Thanks for using the OONI PostgreSQL metaDB, and thanks for your patience throughout this process.</span></div><div id="gmail-magicdomid41" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid42" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">Please contact us with any questions you may have.</span></div><div id="gmail-magicdomid43" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid44" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">Thanks,</span></div><div id="gmail-magicdomid45" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px"></div><div id="gmail-magicdomid46" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><span class="gmail-" style="box-sizing:border-box;margin:0px;padding:3px 0px 4px">~ OONI team. </span></div><div id="gmail-magicdomid47" class="gmail-ace-line" style="box-sizing:border-box;margin:0px;padding:0px"><br style="box-sizing:border-box;margin:0px;padding:0px;color:rgb(72,83,101);font-family:monospace;font-size:15px"></div></div>