On Fri, Mar 11, 2016 at 06:10:16PM +0100, Arturo Filastò wrote:

On Mar 8, 2016, at 23:32, David Fifield david@bamsoftware.com wrote:

...

Did report_ids get regenerated when reports were converted from YAML to JSON? I think they did but I want to make sure.

For example, I have a copy of the old YAML report 20140428T232415Z-AS1241-http_requests_test-v1-probe.yaml.gz. It has a report_id ending in "zuj":

report_filename: 20140428T232415Z-AS1241-http_requests_test-v1-probe.yaml report_id: 2014-04-28aqfgmdfzxjwmreodmroptzeugvanvtznhclirzuj

The new corresponding JSON file seems to be https://ooni-public.s3.amazonaws.com/json/2014-02-28.json (requires an access key). Its report_id instead ends in "ois":

"report_filename": "20140428T232415Z-AS1241-http_requests_test-v1-probe.yaml", "report_id": "2014-04-28nltgjbivffrtkqsvpoaudhgsgyafkbfldqncrois”

Hi David,

Yes some of the reports have in fact changed their ID since not all the IDs were being generated in older versions of OONI and they were being added in the normalisation stage of the data pipeline.

If you need to know the mapping between the old IDs and the new ones I think I can come up with a way to do that, but can require a bit of time.

Can I ask what you were using the IDs for?

I was just checking to make sure that my programs were producing the same output from JSON input as they were producing from YAML input. So far the output is the same, except for the report_ids.