Feedback on OONI data collection, aggregation, and visualization
Hi everyone, I just subscribed to this list, because Arturo asked me to comment on two postings here. As a very quick introduction, and because I don't know how distinct the Tor community and the OONI community are: I'm the developer behind the Tor network data collector CollecTor [-1] and the Tor Metrics website that aggregates and visualizes Tor network data [0]. Here's what Arturo asked on another thread:
With OONI what we are currently focusing on is Bridge Reachability measurements. We have at this time 1 meter in China, 1 in Iran (a second one is going to be setup soon), 1 in Russia and 1 in Ukraine. We have some ideas of the sorts of information we would like to extract from this data, but it would also be very good to have some more feedback from you on what would be useful [1].
Long mail is long. Some random thoughts: - For Tor network data it has turned out to be quite useful to strictly separate data collection from data aggregation from data visualization. That is, don't worry too much about visualizing the right thing, but start with something, and if you don't like it, throw it away and do it differently. And if you're aggregating the wrong thing, then aggregate the previously collected data in a different way. Of course, if you figure out you collected the wrong thing, then you won't be able to go back in time and fix that. - I saw some discussion of "The pool from where the bridge has been extracted (private, tbb, BridgeDB https, BridgeDB email)". Note that isis and I are currently talking about removing sanitized bridge pool assignments from CollecTor. We're thinking about adding a new config line to tor that states the preferred bridge pool, which could be used here instead. Just as a heads-up, six months or so in advance. I can probably provide more details if this is relevant to you.
Another area that perhaps overlaps with the needs of the metrics is data storage. Currently we have around 16 GB of uncompressed raw report data that needs to be archived (currently it's being stored and published on staticiforme, but I have a feeling that is not ideal especially when the data will become much bigger) and indexed in some sort of database. Once we put the data (or a subset of it) in a database producing visualizations and exposing the data to end users will be much simpler. The question is if this is a need also for Metrics/BwAuth/ExitScanner/DocTor and if we can perhaps work out some shared infrastructure to fit both of our goals. Currently we have placed the data inside of MongoDB, but some concerns with it have been raised [2].
Again, some random thoughts: - For Metrics, the choice of database is entirely an internal decision, and no user would ever see that. It's part of the aggregation part. If we ever decide to pick something else (than PostgreSQL in this case), we'd have to rewrite the aggregation scripts, which would then produce the same or similar output (which is an .csv file in our case). That being said, trying out MongoDB or another NoSQL variant might be worthwhile, but don't rely on it too much. - Would you want to add bridge reachability statistics to Tor Metrics? I'm currently working on opening it up and making it easier for people to contribute metrics. Maybe take a look at the website prototype that I posted to tor-dev@ a week ago [3] (and if you want, comment there). I could very well imagine adding a new section "Reachability" right next to "Diversity" with one or more graphs/tables provided by you. Please see the new "Contributing to Tor Metrics" section on the About page for the various options for contributing data or metrics. - Please ask weasel for a VM to host those 16 GB of report data; having it on staticiforme is probably a bad idea. Also, do you have any plans to synchronize reports between hosts? I'm planning such a thing for CollecTor where two or more instances fetch relay descriptors from directory authorities and automatically exchange missing descriptors. - I could imagine extending CollecTor to also collect and archive OONI reports, as a long-term thing. Right now CollecTor does that for Tor relay and bridge descriptors, TORDNSEL exit lists, BridgeDB pool assignment files, and Torperf performance measurement results. But note that it's written in Java and that I hardly have development time to keep it afloat; so somebody else would have to extend it towards supporting OONI reports. I'd be willing to review and merge things. We should also keep CollecTor pure Java, because I want to make it easier for others to run their own mirror and help us make data more redundant. Anyway, I can also imagine keeping the OONI report collector distinct from CollecTor and only exchange design ideas and experiences if that's easier. Lots of ideas. What do you think? All the best, Karsten [-1] https://collector.torproject.org/ [0] https://metrics.torproject.org/ [1] https://lists.torproject.org/pipermail/ooni-dev/2014-October/000176.html [2] https://lists.torproject.org/pipermail/ooni-dev/2014-October/000178.html [3] https://kloesing.github.io/metrics-2.0/
Hi Karsten, Thanks for these thoughts and sorry for not replying sooner. On 10/19/14, 1:52 PM, Karsten Loesing wrote:
- For Tor network data it has turned out to be quite useful to strictly separate data collection from data aggregation from data visualization. That is, don't worry too much about visualizing the right thing, but start with something, and if you don't like it, throw it away and do it differently. And if you're aggregating the wrong thing, then aggregate the previously collected data in a different way. Of course, if you figure out you collected the wrong thing, then you won't be able to go back in time and fix that.
This is indeed the approach that we are now using in the case of the oonipipeline. Put all the data we collect into a NoSQL database, on which we can then run queries and present it in various ways. There are some ideas of how to present this data and you can learn more about this here: https://trac.torproject.org/projects/tor/ticket/13731
- I saw some discussion of "The pool from where the bridge has been extracted (private, tbb, BridgeDB https, BridgeDB email)". Note that isis and I are currently talking about removing sanitized bridge pool assignments from CollecTor. We're thinking about adding a new config line to tor that states the preferred bridge pool, which could be used here instead. Just as a heads-up, six months or so in advance. I can probably provide more details if this is relevant to you.
This is probably something that should be mentioned inside of this ticket: https://trac.torproject.org/projects/tor/ticket/13570 I like the idea that interaction with bridgeDB is opaque to us. All we care is that they give us a JSON dictionary that has some keys we expect.
Another area that perhaps overlaps with the needs of the metrics is data storage. Currently we have around 16 GB of uncompressed raw report data that needs to be archived (currently it's being stored and published on staticiforme, but I have a feeling that is not ideal especially when the data will become much bigger) and indexed in some sort of database. Once we put the data (or a subset of it) in a database producing visualizations and exposing the data to end users will be much simpler. The question is if this is a need also for Metrics/BwAuth/ExitScanner/DocTor and if we can perhaps work out some shared infrastructure to fit both of our goals. Currently we have placed the data inside of MongoDB, but some concerns with it have been raised [2].
Again, some random thoughts:
- For Metrics, the choice of database is entirely an internal decision, and no user would ever see that. It's part of the aggregation part. If we ever decide to pick something else (than PostgreSQL in this case), we'd have to rewrite the aggregation scripts, which would then produce the same or similar output (which is an .csv file in our case). That being said, trying out MongoDB or another NoSQL variant might be worthwhile, but don't rely on it too much.
At this point we have been using MongoDB for a couple of months and a part from a few initial issues (that had to do with me not being familiar with NoSQL document oriented databases) it works quite well. I also realized that doing JOINs with NoSQL on different collections (i.e. tables) is not something you want to do. If there is no (or minimal) duplication it's always best to just stick everything inside of one big fat document. To do this I need to re-process all the data, but it is the path that we are going to follow in the future.
- Would you want to add bridge reachability statistics to Tor Metrics? I'm currently working on opening it up and making it easier for people to contribute metrics. Maybe take a look at the website prototype that I posted to tor-dev@ a week ago [3] (and if you want, comment there). I could very well imagine adding a new section "Reachability" right next to "Diversity" with one or more graphs/tables provided by you. Please see the new "Contributing to Tor Metrics" section on the About page for the various options for contributing data or metrics.
Yes this would be awesome! Our timeline for shipping these visualizations is that we would like to have something ready by the end of this year (at this point 1 month). I think we should be able to get there also with the help of Choke Point Project. I will keep you posted and send a reply to that thread once we have something to be posted publicly ready.
- Please ask weasel for a VM to host those 16 GB of report data; having it on staticiforme is probably a bad idea. Also, do you have any plans to synchronize reports between hosts? I'm planning such a thing for CollecTor where two or more instances fetch relay descriptors from directory authorities and automatically exchange missing descriptors.
I ended up getting 1 box donated from GreenHost and renting another one, since this gives us more freedom to operate. We do have in mind a multi host sync protocol that follows a pub-sub paradigm, but for the moment it's implemented using just simple rsync based polling. I have a cronjob that runs a rsync task on every host that collects reports. There are hosts that receive reports (for archive purposes) and ones that just collect them from clients and then want them to be archived. For the later the cronjob will copy the reports that it has not already archived to all the hosts that should archive them and then deletes the copy on the collector. For how it is implemented see this code: https://github.com/TheTorProject/ooni-pipeline/blob/master/ooni/pipeline/tas...
- I could imagine extending CollecTor to also collect and archive OONI reports, as a long-term thing. Right now CollecTor does that for Tor relay and bridge descriptors, TORDNSEL exit lists, BridgeDB pool assignment files, and Torperf performance measurement results. But note that it's written in Java and that I hardly have development time to keep it afloat; so somebody else would have to extend it towards supporting OONI reports. I'd be willing to review and merge things. We should also keep CollecTor pure Java, because I want to make it easier for others to run their own mirror and help us make data more redundant. Anyway, I can also imagine keeping the OONI report collector distinct from CollecTor and only exchange design ideas and experiences if that's easier.
That would be awesome! Can you point me to relevant CollecTor code portions that would be helpful to implement this? It would be great if you could perhaps write a ticket giving some pointers to who may be interested in implementing this under the OONI component of trac.
Lots of ideas. What do you think?
Thanks for taking the time to compose this. ~ Arturo
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 01/12/14 15:42, Arturo Filastò wrote:
Hi Karsten,
Hi Arturo,
Thanks for these thoughts and sorry for not replying sooner.
Same here. Please find my reply inline, with parts that I don't have a good answer to removed.
On 10/19/14, 1:52 PM, Karsten Loesing wrote:
- I saw some discussion of "The pool from where the bridge has been extracted (private, tbb, BridgeDB https, BridgeDB email)". Note that isis and I are currently talking about removing sanitized bridge pool assignments from CollecTor. We're thinking about adding a new config line to tor that states the preferred bridge pool, which could be used here instead. Just as a heads-up, six months or so in advance. I can probably provide more details if this is relevant to you.
This is probably something that should be mentioned inside of this ticket:
https://trac.torproject.org/projects/tor/ticket/13570
I like the idea that interaction with bridgeDB is opaque to us. All we care is that they give us a JSON dictionary that has some keys we expect.
Oh, good, you're already talking to BridgeDB people about this. Note that I stopped collecting and sanitizing bridge pool assignments in CollecTor yesterday. There has been no discussion on the new config line yet.
- Would you want to add bridge reachability statistics to Tor Metrics? I'm currently working on opening it up and making it easier for people to contribute metrics. Maybe take a look at the website prototype that I posted to tor-dev@ a week ago [3] (and if you want, comment there). I could very well imagine adding a new section "Reachability" right next to "Diversity" with one or more graphs/tables provided by you. Please see the new "Contributing to Tor Metrics" section on the About page for the various options for contributing data or metrics.
Yes this would be awesome!
Our timeline for shipping these visualizations is that we would like to have something ready by the end of this year (at this point 1 month).
I think we should be able to get there also with the help of Choke Point Project.
I will keep you posted and send a reply to that thread once we have something to be posted publicly ready.
So, the redesign of Tor Metrics and its navigation is not done yet, but it's at a point where we can add new visualizations on bridge reachability quite easily. Just note that we should only add visualizations that are directly related to the Tor network, which is probably only a subset of what OONI produces. That's why I mentioned bridge reachability as an example. Given your deadline, how about we start with one or more "Link" pages like this one? https://metrics.torproject.org/oxford-anonymous-internet.html For each of these pages, I need a title ("Tor users as percentage of larger Internet population"), a permanent graph identifier ("oxford-anonymous-internet"), a short description ("The Oxford Internet Institute made..."), and the link ("http://geography.oii.ox.ac.uk/?page=tor"). Or, if you have visualizations that don't require server-side code, like d3.js, we can add that code directly to the website. For example: https://metrics.torproject.org/bubbles.html
We do have in mind a multi host sync protocol that follows a pub-sub paradigm, but for the moment it's implemented using just simple rsync based polling.
- I could imagine extending CollecTor to also collect and archive OONI reports, as a long-term thing. Right now CollecTor does that for Tor relay and bridge descriptors, TORDNSEL exit lists, BridgeDB pool assignment files, and Torperf performance measurement results. But note that it's written in Java and that I hardly have development time to keep it afloat; so somebody else would have to extend it towards supporting OONI reports. I'd be willing to review and merge things. We should also keep CollecTor pure Java, because I want to make it easier for others to run their own mirror and help us make data more redundant. Anyway, I can also imagine keeping the OONI report collector distinct from CollecTor and only exchange design ideas and experiences if that's easier.
That would be awesome!
Can you point me to relevant CollecTor code portions that would be helpful to implement this?
It would be great if you could perhaps write a ticket giving some pointers to who may be interested in implementing this under the OONI component of trac.
Or, before we talk about code, can you elaborate on the pub-sub paradigm that you mention above? Maybe we can combine my efforts to make CollecTor more redundant with your wish to do the same for OONI reports. I could imagine running two nodes that add Tor descriptors and mirror OONI reports, and you run nodes that add OONI reports and mirror Tor descriptors. And Java is not an issue for you? :) All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJUhs2kAAoJEJd5OEYhk8hIAWYH/34MSO7QuMj/bb4Bi3/cOG0p cXy35MXDCefAnDg0RnSV7MHs/uonTR/rkhWoCvdBGZn1ZsdFGizVuneNGZpdT1NN QP1l+Ass4t5L9oZT3PdAF0ZnAIdF/tA4KxpSCx+Q0pkko3yVXChpfzu3UipLG4/c pihD6z2FtRKuWl99EgypHBaQEgpxhfxI20HlSG+H44fM8RNsRmp6xQmdMllepuf+ qP7Mgn2fZEk837rYwQs+BhnVzAx4usqnBJRklT/0dsGklltt5ARQtFwl9TZBCj7Q yyzNnmWpPX6e4kq6PXHuEi7TCTZTnqllxD362yHl0cBTFdpCM5M2L9iBIrkf2gA= =V1Cx -----END PGP SIGNATURE-----
On 12/9/14, 11:23 AM, Karsten Loesing wrote:
I like the idea that interaction with bridgeDB is opaque to us. All we care is that they give us a JSON dictionary that has some keys we expect.
Oh, good, you're already talking to BridgeDB people about this.
Note that I stopped collecting and sanitizing bridge pool assignments in CollecTor yesterday. There has been no discussion on the new config line yet.
What could be the impact of this on our ability to produce the results we are currently producing? I am not very familiar with the BridgeDB component, so I am sort of expecting them to give us the data format described in the ticket #13570 and as long as that is the case we will not have worries. Will this make implemented #13570 more difficult for the BridgeDB team?
So, the redesign of Tor Metrics and its navigation is not done yet, but it's at a point where we can add new visualizations on bridge reachability quite easily.
Just note that we should only add visualizations that are directly related to the Tor network, which is probably only a subset of what OONI produces. That's why I mentioned bridge reachability as an example.
Given your deadline, how about we start with one or more "Link" pages like this one?
https://metrics.torproject.org/oxford-anonymous-internet.html
I am not yet fully certain we will hit the expected deadline as I believe Choke Point Project is working on it, but have not spoken to them recently so I am not sure how far along they are. I am adding them to cc to learn if it is probable that we will have implemented by the end of this year, the changes I have requested. However adding it to the link pages sounds like a good interim move once we are ready to go public with it.
For each of these pages, I need a title ("Tor users as percentage of larger Internet population"), a permanent graph identifier ("oxford-anonymous-internet"), a short description ("The Oxford Internet Institute made..."), and the link ("http://geography.oii.ox.ac.uk/?page=tor").
Title: Tor Bridge Reachability Timeline Description: The OONI is conducting a study on reachability of Tor bridges inside of countries that are known to block access to them. These visualizations show how many of the sampled bridges are working from the countries in question and which types of pluggable transports are more likely to work or not.
Or, if you have visualizations that don't require server-side code, like d3.js, we can add that code directly to the website. For example:
Our visualization don't require any server-side code, but I would need to periodically update some static files on that server with data from the latest measurements. I believe I should be able to do that running one of weasels magic scripts.
We do have in mind a multi host sync protocol that follows a pub-sub paradigm, but for the moment it's implemented using just simple rsync based polling.
- I could imagine extending CollecTor to also collect and archive OONI reports, as a long-term thing. Right now CollecTor does that for Tor relay and bridge descriptors, TORDNSEL exit lists, BridgeDB pool assignment files, and Torperf performance measurement results. But note that it's written in Java and that I hardly have development time to keep it afloat; so somebody else would have to extend it towards supporting OONI reports. I'd be willing to review and merge things. We should also keep CollecTor pure Java, because I want to make it easier for others to run their own mirror and help us make data more redundant. Anyway, I can also imagine keeping the OONI report collector distinct from CollecTor and only exchange design ideas and experiences if that's easier.
That would be awesome!
Can you point me to relevant CollecTor code portions that would be helpful to implement this?
It would be great if you could perhaps write a ticket giving some pointers to who may be interested in implementing this under the OONI component of trac.
Or, before we talk about code, can you elaborate on the pub-sub paradigm that you mention above?
Maybe we can combine my efforts to make CollecTor more redundant with your wish to do the same for OONI reports. I could imagine running two nodes that add Tor descriptors and mirror OONI reports, and you run nodes that add OONI reports and mirror Tor descriptors.
I just noticed that I did not write anywhere the ideas I had on this, but I did start writing (**very** little) code to implement this. I propose we start by writing a specification for the protocol and then discuss how we can implement it: https://trac.torproject.org/projects/tor/ticket/13964
And Java is not an issue for you? :)
I would prefer not to have to deal with that, but if that means running the software on shared infrastructure I would be down for it. I have written a bit of java and even have a university exam to prove it! ~ Arturo
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 15/12/14 14:29, Arturo Filastò wrote:
On 12/9/14, 11:23 AM, Karsten Loesing wrote:
I like the idea that interaction with bridgeDB is opaque to us. All we care is that they give us a JSON dictionary that has some keys we expect.
Oh, good, you're already talking to BridgeDB people about this.
Note that I stopped collecting and sanitizing bridge pool assignments in CollecTor yesterday. There has been no discussion on the new config line yet.
What could be the impact of this on our ability to produce the results we are currently producing? I am not very familiar with the BridgeDB component, so I am sort of expecting them to give us the data format described in the ticket #13570 and as long as that is the case we will not have worries.
Will this make implemented #13570 more difficult for the BridgeDB team?
No, it shouldn't. This would only affect you if you were downloading data from CollecTor or Onionoo. But if you receive data from BridgeDB in a new format, that is unrelated.
So, the redesign of Tor Metrics and its navigation is not done yet, but it's at a point where we can add new visualizations on bridge reachability quite easily.
Just note that we should only add visualizations that are directly related to the Tor network, which is probably only a subset of what OONI produces. That's why I mentioned bridge reachability as an example.
Given your deadline, how about we start with one or more "Link" pages like this one?
https://metrics.torproject.org/oxford-anonymous-internet.html
I am not yet fully certain we will hit the expected deadline as I believe Choke Point Project is working on it, but have not spoken to them recently so I am not sure how far along they are.
I am adding them to cc to learn if it is probable that we will have implemented by the end of this year, the changes I have requested.
However adding it to the link pages sounds like a good interim move once we are ready to go public with it.
Sounds good.
For each of these pages, I need a title ("Tor users as percentage of larger Internet population"), a permanent graph identifier ("oxford-anonymous-internet"), a short description ("The Oxford Internet Institute made..."), and the link ("http://geography.oii.ox.ac.uk/?page=tor").
Title: Tor Bridge Reachability Timeline
Description: The OONI is conducting a study on reachability of Tor bridges inside of countries that are known to block access to them. These visualizations show how many of the sampled bridges are working from the countries in question and which types of pluggable transports are more likely to work or not.
I need such a description for each graph you want to see on Tor Metrics. I can help write that as soon as I have the graphs.
Or, if you have visualizations that don't require server-side code, like d3.js, we can add that code directly to the website. For example:
Our visualization don't require any server-side code, but I would need to periodically update some static files on that server with data from the latest measurements. I believe I should be able to do that running one of weasels magic scripts.
You mean building graphs on a host and distributing them to Tor's various web servers? Sure, we just need a URL we can put on Tor Metrics.
We do have in mind a multi host sync protocol that follows a pub-sub paradigm, but for the moment it's implemented using just simple rsync based polling.
- I could imagine extending CollecTor to also collect and archive OONI reports, as a long-term thing. Right now CollecTor does that for Tor relay and bridge descriptors, TORDNSEL exit lists, BridgeDB pool assignment files, and Torperf performance measurement results. But note that it's written in Java and that I hardly have development time to keep it afloat; so somebody else would have to extend it towards supporting OONI reports. I'd be willing to review and merge things. We should also keep CollecTor pure Java, because I want to make it easier for others to run their own mirror and help us make data more redundant. Anyway, I can also imagine keeping the OONI report collector distinct from CollecTor and only exchange design ideas and experiences if that's easier.
That would be awesome!
Can you point me to relevant CollecTor code portions that would be helpful to implement this?
It would be great if you could perhaps write a ticket giving some pointers to who may be interested in implementing this under the OONI component of trac.
Or, before we talk about code, can you elaborate on the pub-sub paradigm that you mention above?
Maybe we can combine my efforts to make CollecTor more redundant with your wish to do the same for OONI reports. I could imagine running two nodes that add Tor descriptors and mirror OONI reports, and you run nodes that add OONI reports and mirror Tor descriptors.
I just noticed that I did not write anywhere the ideas I had on this, but I did start writing (**very** little) code to implement this.
I propose we start by writing a specification for the protocol and then discuss how we can implement it: https://trac.torproject.org/projects/tor/ticket/13964
Great, thanks! Will comment on the ticket once I have good ideas.
And Java is not an issue for you? :)
I would prefer not to have to deal with that, but if that means running the software on shared infrastructure I would be down for it.
I have written a bit of java and even have a university exam to prove it!
Okay. :) All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJUjueeAAoJEJd5OEYhk8hIvOoH+wRi4VlqRDWZVAf2UkwIuymq W/KGoRSTJMG7doC1ITnzyw9WZpXXa2K3koBNFk5I9Jmfw+jEG+UW/k0cMN84xsBj gGMGctQxq6PoRlT++gYlFAdD3C6GA2Ix7AaDFm6lSmDE9DxQQN2DkTPImYpxl6IX ukGOXLhNu9MCJE8nKkq50gsqQRV4FezQs9xoBJLxEJJnZA6X3vSEaiV2/LLY6+M4 2n25luU0/Il3kZoCPEMS9Rxoj/Jfb2J/gzjtiTjpkoXtcUG4Kxzk8IQWKhSa2SZe XC9YWbqYPaJiOoWqk1Moje25SxxejKf/JdcTqPoZkN0Srl2WaeMgb9CCtcOBKaM= =sqqa -----END PGP SIGNATURE-----
On 15/12/2014 14:29, Arturo Filastò wrote:
On 12/9/14, 11:23 AM, Karsten Loesing wrote:
I like the idea that interaction with bridgeDB is opaque to us. All we care is that they give us a JSON dictionary that has some keys we expect. Oh, good, you're already talking to BridgeDB people about this.
Note that I stopped collecting and sanitizing bridge pool assignments in CollecTor yesterday. There has been no discussion on the new config line yet.
What could be the impact of this on our ability to produce the results we are currently producing? I am not very familiar with the BridgeDB component, so I am sort of expecting them to give us the data format described in the ticket #13570 and as long as that is the case we will not have worries.
Will this make implemented #13570 more difficult for the BridgeDB team?
So, the redesign of Tor Metrics and its navigation is not done yet, but it's at a point where we can add new visualizations on bridge reachability quite easily.
Just note that we should only add visualizations that are directly related to the Tor network, which is probably only a subset of what OONI produces. That's why I mentioned bridge reachability as an example.
Given your deadline, how about we start with one or more "Link" pages like this one?
https://metrics.torproject.org/oxford-anonymous-internet.html
I am not yet fully certain we will hit the expected deadline as I believe Choke Point Project is working on it, but have not spoken to them recently so I am not sure how far along they are.
I am adding them to cc to learn if it is probable that we will have implemented by the end of this year, the changes I have requested.
However adding it to the link pages sounds like a good interim move once we are ready to go public with it. We are still aiming to deliver the changes before the end of the year. We're planning to throw time and attention at it starting the 19th. If something changes to that planning I will let you know asap.
For each of these pages, I need a title ("Tor users as percentage of larger Internet population"), a permanent graph identifier ("oxford-anonymous-internet"), a short description ("The Oxford Internet Institute made..."), and the link ("http://geography.oii.ox.ac.uk/?page=tor").
Title: Tor Bridge Reachability Timeline
Description: The OONI is conducting a study on reachability of Tor bridges inside of countries that are known to block access to them. These visualizations show how many of the sampled bridges are working from the countries in question and which types of pluggable transports are more likely to work or not.
Or, if you have visualizations that don't require server-side code, like d3.js, we can add that code directly to the website. For example:
Our visualization don't require any server-side code, but I would need to periodically update some static files on that server with data from the latest measurements. I believe I should be able to do that running one of weasels magic scripts.
We do have in mind a multi host sync protocol that follows a pub-sub paradigm, but for the moment it's implemented using just simple rsync based polling.
- I could imagine extending CollecTor to also collect and archive OONI reports, as a long-term thing. Right now CollecTor does that for Tor relay and bridge descriptors, TORDNSEL exit lists, BridgeDB pool assignment files, and Torperf performance measurement results. But note that it's written in Java and that I hardly have development time to keep it afloat; so somebody else would have to extend it towards supporting OONI reports. I'd be willing to review and merge things. We should also keep CollecTor pure Java, because I want to make it easier for others to run their own mirror and help us make data more redundant. Anyway, I can also imagine keeping the OONI report collector distinct from CollecTor and only exchange design ideas and experiences if that's easier. That would be awesome! Can you point me to relevant CollecTor code portions that would be helpful to implement this? It would be great if you could perhaps write a ticket giving some pointers to who may be interested in implementing this under the OONI component of trac. Or, before we talk about code, can you elaborate on the pub-sub paradigm that you mention above?
Maybe we can combine my efforts to make CollecTor more redundant with your wish to do the same for OONI reports. I could imagine running two nodes that add Tor descriptors and mirror OONI reports, and you run nodes that add OONI reports and mirror Tor descriptors.
I just noticed that I did not write anywhere the ideas I had on this, but I did start writing (**very** little) code to implement this.
I propose we start by writing a specification for the protocol and then discuss how we can implement it: https://trac.torproject.org/projects/tor/ticket/13964
And Java is not an issue for you? :) I would prefer not to have to deal with that, but if that means running the software on shared infrastructure I would be down for it.
I have written a bit of java and even have a university exam to prove it!
~ Arturo
participants (3)
-
Arturo Filastò -
Karsten Loesing -
Ruben Bloemgarten