August 2014 - tor-commits - lists.torproject.org

[spec/master] Spec for decks.
by art＠torproject.org 13 Aug '14

13 Aug '14

commit 12f882a6c530f3aba1479ff24b73abd4ec44d0bc Author: kudrom <kudrom(a)riseup.net> Date: Tue Aug 12 20:41:59 2014 +0200 Spec for decks. --- test-decks/td-spec.md | 138 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 138 insertions(+) diff --git a/test-decks/td-spec.md b/test-decks/td-spec.md new file mode 100644 index 0000000..c1ad49c --- /dev/null +++ b/test-decks/td-spec.md @@ -0,0 +1,138 @@ +# 0. Terminology + Analyst: The person who writes a deck. + + Collector: A machine running the ooni-backend that collects the reports generated by the execution of ooni-probe. + + Nettest: A test whose execution by ooniprobe creates a report sent later to the collector if provided. + + Ooni: Open observatory of network interference. + + Ooni-probe: The client side of ooni used to execute a set of nettests. + + Tester: The person who executes ooni-probe. + + YAML: A human-readable data serialization format. + +# 1. Rationale + + To ease the execution of nettests, the ooni developers came up with the idea of a container that would allow a tester to easily execute a bunch of configured nettests. That container was called a deck. + + This way, an analyst interested in a particular behaviour would write a deck; then, she would distribute that deck to every tester interested in the ongoing analysis. Finally, the tester would execute ooni-probe with that deck to properly create and send the nettests' reports. + + Unfortunately, right now both the deck format and ooni-probe don't allow that level of automation. The way we want to solve this situation is by writing a new spec for the deck format that would allow us to write, with great confidence, the given functionality to ooni-probe. The document that you're reading is that spec. + +# 2. Goals + + Allow an analyst to reuse well written and tested nettests in an easy way. + + Allow a tester to execute easily a battery of nettests with a complex configuration. + + Grant privacy of the tester. + +# 3. The data format + + Every deck is a yaml file composed of two major sections: the header and the body. + + The header is a dictionary that provides all the shared and global configuration of every nettest included in the deck. It's main purpose is to reduce boilerplate by letting the analyst express common behaviour in one section instead that in every nettest execution. The ooni-probe options allowed in this section are: + + 1. collector: Address of the collector of test results. + 2. bouncer: Address of the bouncer for test helpers. + 3. annotations: Annotate the report with a key:value[, key:value] format. + 4. no-collector: Disable the collector. (FLAG) + 5. no-geoip: Disable the geoip support. (FLAG) + + Every option of the header section is a dictionary's key except that labeled with a FLAG, which are members of a list called flags. So for example a valid header section is the following: + + header: + collector: 'http://localhost' + annotations: 'key1:val1, key2:val2' + flags: + - no-collector + + The body is a list composed of one element per nettest execution. Every nettest execution is a dictionary composed of the following four keys: + + 1. nettest: name of the nettest to execute (MANDATORY) + 2. local_options: local_options of the test execution (OPTIONAL) + 3. global_options: global options of the test execution (OPTIONAL) + 4. input_file: the input file of the net test (OPTIONAL) + + In the same way that with the header, every option can be a member of a flag list if that options doesn't have any arguments. A valid body can be: + + body: + - nettest: manipulation/http_request + local_options: + url: 'http://torproject.org' + global_options: + collector: 'http://localhost' + flags: + - no-geoip + - nettest: manipulation/captiveportal + +# 3. Implementation details + +## 3.1 Introduction + + Each execution of a nettest in ooniprobe needs four main inputs. + + 1. The global config file + 2. The global cmd options + 3. The nettest/deck + 4. The local options for each nettest + + The first three ones are mandatory to run a nettest, the last one depends on the nettest. + + When a single nettest is executed, all options except the first one are passed in the cmd line. + + The difference between the global config options and the global cmd options is that the second one has some shared options with the config file plus some additional subcommands to ooniprobe. + + When a deck is involved in the execution, the last three inputs are meant to be passed (at least partially) in the deck. In fact, the deck is no more than the three last inputs, with some subtleties that I expect to explain in the following sections. + +## 3.2 Who overwrites who + + In a single-nettest execution, the global config is parsed and a global object is built. Then the cmd line is parsed and it overwrites the global configs. This is done to allow the setting of the most dynamic options in the console without the need of writing them each time to the config file. + + All the ooni-probe's code base is allowed to access that global object, included the nettests. So, the global options affect the way ooni-probe behaves but also can affect the way the executing nettest behaves. That's the reason why the deck must provide the specification of some global options, but not all because there are some powerful options that would put in danger the tester. The allowed options are listed in the section [3. The data format]. + + In a deck-nettest execution, the header section of the deck is parsed first and a global object is built. Then the config file is parsed and the object is overwritten when it applies. Finally the cmd line of the ooni-probe is parsed and the object is overwritten. + + We read the header section first to avoid the analyst overwrite some sensitive options of the config file, which only should be modified by the tester. + +## 3.3 Copy-on-write + + To reduce the boilerplate in the deck, the file is splitted into a shared section for all nettests and a local one for each nettest. + + The idea is to allow the writer of the deck to express common and global behaviour of all the nettests in the header of the deck and to put specific and local options in each nettest element, as was already explained in [3. The data format]. + + What this means for the execution of the nettests is that when a nettest overwrites a global configuration of the header, this change is only visible to this nettest, not to every nettest who may ever execute in the same instance of ooni-probe. So what follows is that every change to the global options of ooni-probe in the local section of a deck should attend a copy-on-write policy regarding the global config object. + +## 3.4 The input file + + Ooni-probe invokes every nettest method with the information saved in a file called the input file. This file is part of the local configuration of every test, and therefore must be provided with the deck. + + So the deck should be a compressed container which includes both the deck file and every input file necessary to every nettest included in the deck. Otherwise, the analyst would have to send to the tester the input files separately, which is unacceptable. + + The container proposed is tar+gzip because it's well supported in python. + +# 5. Example deck + + The complete.deck provided with each installation of ooni-probe would be: + + header: + flags: + - no-collector + + body: + - nettest: blocking/http_request + input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1' + + - nettest: blocking/dns_consistency + input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1' + + - nettest: manipulation/http_invalid_request_line + + - nettest: manipulation/http_header_field_manipulation + + - nettest: manipulation/traceroute + + - nettest: blocking/http_host + input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1'

1 0

[spec/master] Merge pull request #29 from kudrom/master
by art＠torproject.org 13 Aug '14

13 Aug '14

commit 99fba9e6565d5f888e6c4624f569b15a5ef310eb Merge: ea366e9 12f882a Author: Arturo Filastò <hellais(a)users.noreply.github.com> Date: Wed Aug 13 16:54:40 2014 +0200 Merge pull request #29 from kudrom/master Spec for decks. test-decks/td-spec.md | 138 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 138 insertions(+)

1 0

[spec/master] Make the annotations be an actual dict and other misc fixes
by art＠torproject.org 13 Aug '14

13 Aug '14

commit 12a3670d8e579984d9cdafb0c21c62c603c41d37 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Aug 13 16:57:29 2014 +0200 Make the annotations be an actual dict and other misc fixes --- test-decks/td-spec.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/test-decks/td-spec.md b/test-decks/td-spec.md index c1ad49c..5e65f3d 100644 --- a/test-decks/td-spec.md +++ b/test-decks/td-spec.md @@ -33,7 +33,7 @@ Every deck is a yaml file composed of two major sections: the header and the body. - The header is a dictionary that provides all the shared and global configuration of every nettest included in the deck. It's main purpose is to reduce boilerplate by letting the analyst express common behaviour in one section instead that in every nettest execution. The ooni-probe options allowed in this section are: + The header is a dictionary that provides all the shared and global configuration of every nettest included in the deck. Its main purpose is to reduce boilerplate by letting the analyst express common behaviour in one section instead of in every nettest execution. The ooni-probe options allowed in this section are: 1. collector: Address of the collector of test results. 2. bouncer: Address of the bouncer for test helpers. @@ -45,16 +45,17 @@ header: collector: 'http://localhost' - annotations: 'key1:val1, key2:val2' + annotations: + key1: value1 + key2: value2 flags: - no-collector - The body is a list composed of one element per nettest execution. Every nettest execution is a dictionary composed of the following four keys: + The body is a list composed of one element per nettest execution. Every nettest execution is a dictionary composed of the following three keys: 1. nettest: name of the nettest to execute (MANDATORY) 2. local_options: local_options of the test execution (OPTIONAL) 3. global_options: global options of the test execution (OPTIONAL) - 4. input_file: the input file of the net test (OPTIONAL) In the same way that with the header, every option can be a member of a flag list if that options doesn't have any arguments. A valid body can be:

1 0

[spec/master] Fix the deck formatting so that they are valid.
by art＠torproject.org 13 Aug '14

13 Aug '14

commit 532484546119687a849dd7667093b8ef6b7bde29 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Aug 13 17:08:40 2014 +0200 Fix the deck formatting so that they are valid. --- test-decks/td-spec.md | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/test-decks/td-spec.md b/test-decks/td-spec.md index 83075b5..70e523a 100644 --- a/test-decks/td-spec.md +++ b/test-decks/td-spec.md @@ -64,14 +64,16 @@ header section is the following: ``` header: -collector: 'http://localhost' -annotations: -key1: value1 -key2: value2 -flags: -- no-collector + collector: 'http://localhost' + annotations: + key1: value1 + key2: value2 + flags: + - no-collector ``` +The header may also be omitted. + The body is a list composed of one element per nettest execution. Every nettest execution is a dictionary composed of the following three keys: @@ -85,13 +87,13 @@ list if that options doesn't have any arguments. A valid body can be: ``` body: - nettest: manipulation/http_request -local_options: -url: 'http://torproject.org' -global_options: -collector: 'http://localhost' -flags: -- no-geoip -- nettest: manipulation/captiveportal + local_options: + url: 'http://torproject.org' + global_options: + collector: 'http://localhost' + flags: + - no-geoip + - nettest: manipulation/captiveportal ``` # 3. Implementation details @@ -177,13 +179,12 @@ The container proposed is tar+gzip because it's well supported in python. The complete.deck provided with each installation of ooni-probe would be: ``` -header: body: - nettest: blocking/http_request -input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1' + input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1' - nettest: blocking/dns_consistency -input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1' + input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1' - nettest: manipulation/http_invalid_request_line @@ -192,5 +193,5 @@ input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371 - nettest: manipulation/traceroute - nettest: blocking/http_host -input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1' + input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1' ```

1 0

[spec/master] Re-indent and fix markup of spec
by art＠torproject.org 13 Aug '14

13 Aug '14

commit 4de9ae90859b2aeb35571ec349af98a08a94a0d1 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Aug 13 17:02:05 2014 +0200 Re-indent and fix markup of spec --- test-decks/td-spec.md | 293 +++++++++++++++++++++++++++++-------------------- 1 file changed, 175 insertions(+), 118 deletions(-) diff --git a/test-decks/td-spec.md b/test-decks/td-spec.md index 5e65f3d..83075b5 100644 --- a/test-decks/td-spec.md +++ b/test-decks/td-spec.md @@ -1,139 +1,196 @@ # 0. Terminology - Analyst: The person who writes a deck. - - Collector: A machine running the ooni-backend that collects the reports generated by the execution of ooni-probe. - - Nettest: A test whose execution by ooniprobe creates a report sent later to the collector if provided. - - Ooni: Open observatory of network interference. - - Ooni-probe: The client side of ooni used to execute a set of nettests. - - Tester: The person who executes ooni-probe. - - YAML: A human-readable data serialization format. +Analyst: The person who writes a deck. + +Collector: A machine running the ooni-backend that collects the reports +generated by the execution of ooni-probe. + +Nettest: A test whose execution by ooniprobe creates a report sent later to the +collector if provided. + +Ooni: Open observatory of network interference. + +Ooni-probe: The client side of ooni used to execute a set of nettests. + +Tester: The person who executes ooni-probe. + +YAML: A human-readable data serialization format. # 1. Rationale - To ease the execution of nettests, the ooni developers came up with the idea of a container that would allow a tester to easily execute a bunch of configured nettests. That container was called a deck. - - This way, an analyst interested in a particular behaviour would write a deck; then, she would distribute that deck to every tester interested in the ongoing analysis. Finally, the tester would execute ooni-probe with that deck to properly create and send the nettests' reports. - - Unfortunately, right now both the deck format and ooni-probe don't allow that level of automation. The way we want to solve this situation is by writing a new spec for the deck format that would allow us to write, with great confidence, the given functionality to ooni-probe. The document that you're reading is that spec. - +To ease the execution of nettests, the ooni developers came up with the idea of +a container that would allow a tester to easily execute a bunch of configured +nettests.That container was called a deck. + +This way, an analyst interested in a particular behaviour would write a deck; +then, she would distribute that deck to every tester interested in the ongoing +analysis. Finally, the tester would execute ooni-probe with that deck to +properly create and send the nettests' reports. + +Unfortunately, right now both the deck format and ooni-probe don't allow that +level of automation. The way we want to solve this situation is by writing a +new spec for the deck format that would allow us to write, with great +confidence, the given functionality to ooni-probe. The document that you're +reading is that spec. + # 2. Goals - Allow an analyst to reuse well written and tested nettests in an easy way. - - Allow a tester to execute easily a battery of nettests with a complex configuration. - - Grant privacy of the tester. +Allow an analyst to reuse well written and tested nettests in an easy way. + +Allow a tester to execute easily a battery of nettests with a complex +configuration. + +Grant privacy of the tester. # 3. The data format - Every deck is a yaml file composed of two major sections: the header and the body. - - The header is a dictionary that provides all the shared and global configuration of every nettest included in the deck. Its main purpose is to reduce boilerplate by letting the analyst express common behaviour in one section instead of in every nettest execution. The ooni-probe options allowed in this section are: - - 1. collector: Address of the collector of test results. - 2. bouncer: Address of the bouncer for test helpers. - 3. annotations: Annotate the report with a key:value[, key:value] format. - 4. no-collector: Disable the collector. (FLAG) - 5. no-geoip: Disable the geoip support. (FLAG) - - Every option of the header section is a dictionary's key except that labeled with a FLAG, which are members of a list called flags. So for example a valid header section is the following: - - header: - collector: 'http://localhost' - annotations: - key1: value1 - key2: value2 - flags: - - no-collector - - The body is a list composed of one element per nettest execution. Every nettest execution is a dictionary composed of the following three keys: - - 1. nettest: name of the nettest to execute (MANDATORY) - 2. local_options: local_options of the test execution (OPTIONAL) - 3. global_options: global options of the test execution (OPTIONAL) - - In the same way that with the header, every option can be a member of a flag list if that options doesn't have any arguments. A valid body can be: - - body: - - nettest: manipulation/http_request - local_options: - url: 'http://torproject.org' - global_options: - collector: 'http://localhost' - flags: - - no-geoip - - nettest: manipulation/captiveportal - +Every deck is a yaml file composed of two major sections: the header and the +body. + +The header is a dictionary that provides all the shared and global +configuration of every nettest included in the deck. Its main purpose is to +reduce boilerplate by letting the analyst express common behaviour in one +section instead of in every nettest execution. The ooni-probe options allowed +in this section are: + +1. collector: Address of the collector of test results. +2. bouncer: Address of the bouncer for test helpers. +3. annotations: Annotate the report with a key:value[, key:value] format. +4. no-collector: Disable the collector. (FLAG) +5. no-geoip: Disable the geoip support. (FLAG) + +Every option of the header section is a dictionary's key except that labeled +with a FLAG, which are members of a list called flags. So for example a valid +header section is the following: + +``` +header: +collector: 'http://localhost' +annotations: +key1: value1 +key2: value2 +flags: +- no-collector +``` + +The body is a list composed of one element per nettest execution. Every nettest +execution is a dictionary composed of the following three keys: + +1. nettest: name of the nettest to execute (MANDATORY) +2. local_options: local_options of the test execution (OPTIONAL) +3. global_options: global options of the test execution (OPTIONAL) + +In the same way that with the header, every option can be a member of a flag +list if that options doesn't have any arguments. A valid body can be: + +``` +body: +- nettest: manipulation/http_request +local_options: +url: 'http://torproject.org' +global_options: +collector: 'http://localhost' +flags: +- no-geoip +- nettest: manipulation/captiveportal +``` + # 3. Implementation details ## 3.1 Introduction - Each execution of a nettest in ooniprobe needs four main inputs. - - 1. The global config file - 2. The global cmd options - 3. The nettest/deck - 4. The local options for each nettest - - The first three ones are mandatory to run a nettest, the last one depends on the nettest. - - When a single nettest is executed, all options except the first one are passed in the cmd line. - - The difference between the global config options and the global cmd options is that the second one has some shared options with the config file plus some additional subcommands to ooniprobe. - - When a deck is involved in the execution, the last three inputs are meant to be passed (at least partially) in the deck. In fact, the deck is no more than the three last inputs, with some subtleties that I expect to explain in the following sections. - +Each execution of a nettest in ooniprobe needs four main inputs. + +1. The global config file +2. The global cmd options +3. The nettest/deck +4. The local options for each nettest + +The first three ones are mandatory to run a nettest, the last one depends on +the nettest. + +When a single nettest is executed, all options except the first one are +passed in the cmd line. + +The difference between the global config options and the global cmd options +is that the second one has some shared options with the config file plus some +additional subcommands to ooniprobe. + +When a deck is involved in the execution, the last three inputs are meant to +be passed (at least partially) in the deck. In fact, the deck is no more than +the three last inputs, with some subtleties that I expect to explain in the +following sections. + ## 3.2 Who overwrites who - - In a single-nettest execution, the global config is parsed and a global object is built. Then the cmd line is parsed and it overwrites the global configs. This is done to allow the setting of the most dynamic options in the console without the need of writing them each time to the config file. - - All the ooni-probe's code base is allowed to access that global object, included the nettests. So, the global options affect the way ooni-probe behaves but also can affect the way the executing nettest behaves. That's the reason why the deck must provide the specification of some global options, but not all because there are some powerful options that would put in danger the tester. The allowed options are listed in the section [3. The data format]. - - In a deck-nettest execution, the header section of the deck is parsed first and a global object is built. Then the config file is parsed and the object is overwritten when it applies. Finally the cmd line of the ooni-probe is parsed and the object is overwritten. - - We read the header section first to avoid the analyst overwrite some sensitive options of the config file, which only should be modified by the tester. - + +In a single-nettest execution, the global config is parsed and a global object +is built. Then the cmd line is parsed and it overwrites the global configs. +This is done to allow the setting of the most dynamic options in the console +without the need of writing them each time to the config file. + +All the ooni-probe's code base is allowed to access that global object, +included the nettests. So, the global options affect the way ooni-probe behaves +but also can affect the way the executing nettest behaves. That's the reason +why the deck must provide the specification of some global options, but not all +because there are some powerful options that would put in danger the tester. +The allowed options are listed in the section [3. The data format]. + +In a deck-nettest execution, the header section of the deck is parsed first and +a global object is built. Then the config file is parsed and the object is +overwritten when it applies. Finally the cmd line of the ooni-probe is parsed +and the object is overwritten. + +We read the header section first to avoid the analyst overwrite some sensitive +options of the config file, which only should be modified by the tester. + ## 3.3 Copy-on-write - To reduce the boilerplate in the deck, the file is splitted into a shared section for all nettests and a local one for each nettest. - - The idea is to allow the writer of the deck to express common and global behaviour of all the nettests in the header of the deck and to put specific and local options in each nettest element, as was already explained in [3. The data format]. - - What this means for the execution of the nettests is that when a nettest overwrites a global configuration of the header, this change is only visible to this nettest, not to every nettest who may ever execute in the same instance of ooni-probe. So what follows is that every change to the global options of ooni-probe in the local section of a deck should attend a copy-on-write policy regarding the global config object. - +To reduce the boilerplate in the deck, the file is splitted into a shared +section for all nettests and a local one for each nettest. + +The idea is to allow the writer of the deck to express common and global +behaviour of all the nettests in the header of the deck and to put specific and +local options in each nettest element, as was already explained in [3. The data +format]. + +What this means for the execution of the nettests is that when a nettest +overwrites a global configuration of the header, this change is only visible to +this nettest, not to every nettest who may ever execute in the same instance of +ooni-probe. So what follows is that every change to the global options of +ooni-probe in the local section of a deck should attend a copy-on-write policy +regarding the global config object. + ## 3.4 The input file - Ooni-probe invokes every nettest method with the information saved in a file called the input file. This file is part of the local configuration of every test, and therefore must be provided with the deck. - - So the deck should be a compressed container which includes both the deck file and every input file necessary to every nettest included in the deck. Otherwise, the analyst would have to send to the tester the input files separately, which is unacceptable. - - The container proposed is tar+gzip because it's well supported in python. +Ooni-probe invokes every nettest method with the information saved in a file +called the input file. This file is part of the local configuration of every +test, and therefore must be provided with the deck. + +So the deck should be a compressed container which includes both the deck file +and every input file necessary to every nettest included in the deck. +Otherwise, the analyst would have to send to the tester the input files +separately, which is unacceptable. + +The container proposed is tar+gzip because it's well supported in python. # 5. Example deck - The complete.deck provided with each installation of ooni-probe would be: - - header: - flags: - - no-collector - - body: - - nettest: blocking/http_request - input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1' - - - nettest: blocking/dns_consistency - input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1' - - - nettest: manipulation/http_invalid_request_line - - - nettest: manipulation/http_header_field_manipulation - - - nettest: manipulation/traceroute - - - nettest: blocking/http_host - input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1' +The complete.deck provided with each installation of ooni-probe would be: + +``` +header: +body: +- nettest: blocking/http_request +input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1' + +- nettest: blocking/dns_consistency +input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1' + +- nettest: manipulation/http_invalid_request_line + +- nettest: manipulation/http_header_field_manipulation + +- nettest: manipulation/traceroute + +- nettest: blocking/http_host +input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1' +```

1 0

[spec/master] Add header to example deck
by art＠torproject.org 13 Aug '14

13 Aug '14

commit c279e174c42b351e28d60173b3b2e9f1712e0c19 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Aug 13 17:45:39 2014 +0200 Add header to example deck --- test-decks/td-spec.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/test-decks/td-spec.md b/test-decks/td-spec.md index da85985..9d52487 100644 --- a/test-decks/td-spec.md +++ b/test-decks/td-spec.md @@ -203,6 +203,14 @@ This will then be compressed using tar+gzip. The complete.deck provided with each installation of ooni-probe would be: ``` +header: + name: Complete + description: Runs all the existing ooniprobe tests + author: 'OONI <ooni-dev(a)lists.torproject.org>' + version: 0.1.0 + flags: + - requires-root + - requires-tor body: - nettest: blocking/http_request input_file: 'httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1'

1 0

[spec/master] Add specification of the deck metadata
by art＠torproject.org 13 Aug '14

13 Aug '14

commit 645536d4bf847d1702c55e6839c040176407ed22 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Aug 13 17:44:17 2014 +0200 Add specification of the deck metadata --- test-decks/td-spec.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/test-decks/td-spec.md b/test-decks/td-spec.md index 70e523a..da85985 100644 --- a/test-decks/td-spec.md +++ b/test-decks/td-spec.md @@ -72,6 +72,16 @@ header: - no-collector ``` +The deck header can also contain metadata associated to the test deck. The +possible fields are: + +1. name: The name of the test deck. +2. description: A short description of the test deck. +3. author: The author of the test deck. +4. version: A version number for the test deck. +5. requires-root: A flag to indicate that the test deck requires root to run. (FLAG) +6. requires-tor: A flag to indicate that the test deck requires tor to run. (FLAG) + The header may also be omitted. The body is a list composed of one element per nettest execution. Every nettest @@ -174,6 +184,20 @@ separately, which is unacceptable. The container proposed is tar+gzip because it's well supported in python. +# 4. Container format + +The container proposed is tar+gzip because it's well supported in python. The +deck container will be composed of a directory named "deck" containing the deck +file and the inputs. + +The directory layout will be: + +deck/test.deck +deck/input-filename-1.txt +deck/input-filename-2.txt + +This will then be compressed using tar+gzip. + # 5. Example deck The complete.deck provided with each installation of ooni-probe would be:

1 0

[spec/master] Add header to test-deck specification.
by art＠torproject.org 13 Aug '14

13 Aug '14

commit e0268a8403d2b4e978c26c71b12827a258afac94 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Aug 13 17:47:29 2014 +0200 Add header to test-deck specification. --- test-decks/td-spec.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/test-decks/td-spec.md b/test-decks/td-spec.md index 9d52487..328c7a8 100644 --- a/test-decks/td-spec.md +++ b/test-decks/td-spec.md @@ -1,3 +1,9 @@ +# Test deck specification + +* version: 0.1.0 +* date: 2014-08-13 +* author: Alejandro (kudrom), Arturo Filastò + # 0. Terminology Analyst: The person who writes a deck.

1 0

[onionoo/master] Add Cache-Control header.
by karsten＠torproject.org 13 Aug '14

13 Aug '14

commit cbc03ebf6e0a3ad772440f62357ea3baa7c165e5 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Aug 13 15:54:16 2014 +0200 Add Cache-Control header. Set max-age to a value between 5 and 40 minutes, depending on how old the node index is. --- .../org/torproject/onionoo/server/ResourceServlet.java | 14 +++++++++++--- .../java/org/torproject/onionoo/util/DateTimeHelper.java | 1 + 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/src/main/java/org/torproject/onionoo/server/ResourceServlet.java b/src/main/java/org/torproject/onionoo/server/ResourceServlet.java index 481462e..ee71218 100644 --- a/src/main/java/org/torproject/onionoo/server/ResourceServlet.java +++ b/src/main/java/org/torproject/onionoo/server/ResourceServlet.java @@ -61,12 +61,18 @@ public class ResourceServlet extends HttpServlet { return; } - if (ApplicationFactory.getNodeIndexer().getLastIndexed( - DateTimeHelper.TEN_SECONDS) + DateTimeHelper.SIX_HOURS - < ApplicationFactory.getTime().currentTimeMillis()) { + long nowMillis = ApplicationFactory.getTime().currentTimeMillis(); + long indexWrittenMillis = + ApplicationFactory.getNodeIndexer().getLastIndexed( + DateTimeHelper.TEN_SECONDS); + long indexAgeMillis = nowMillis - indexWrittenMillis; + if (indexAgeMillis > DateTimeHelper.SIX_HOURS) { response.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR); return; } + long cacheMaxAgeMillis = Math.max(DateTimeHelper.FIVE_MINUTES, + ((DateTimeHelper.FOURTY_FIVE_MINUTES - indexAgeMillis) + / DateTimeHelper.FIVE_MINUTES) * DateTimeHelper.FIVE_MINUTES); NodeIndex nodeIndex = ApplicationFactory.getNodeIndexer(). getLatestNodeIndex(DateTimeHelper.TEN_SECONDS); @@ -304,6 +310,8 @@ public class ResourceServlet extends HttpServlet { response.setHeader("Access-Control-Allow-Origin", "*"); response.setContentType("application/json"); response.setCharacterEncoding("utf-8"); + response.setHeader("Cache-Control", "public, max-age=" + + (cacheMaxAgeMillis / DateTimeHelper.ONE_SECOND)); PrintWriter pw = response.getWriter(); rb.buildResponse(pw); int relayDocumentsWritten = rh.getOrderedRelays().size(); diff --git a/src/main/java/org/torproject/onionoo/util/DateTimeHelper.java b/src/main/java/org/torproject/onionoo/util/DateTimeHelper.java index 1fcf6e1..b350b9a 100644 --- a/src/main/java/org/torproject/onionoo/util/DateTimeHelper.java +++ b/src/main/java/org/torproject/onionoo/util/DateTimeHelper.java @@ -19,6 +19,7 @@ public class DateTimeHelper { ONE_MINUTE = 60L * ONE_SECOND, FIVE_MINUTES = 5L * ONE_MINUTE, FIFTEEN_MINUTES = 15L * ONE_MINUTE, + FOURTY_FIVE_MINUTES = 45L * ONE_MINUTE, ONE_HOUR = 60L * ONE_MINUTE, FOUR_HOURS = 4L * ONE_HOUR, SIX_HOURS = 6L * ONE_HOUR,

1 0

[translation/tails-greeter] Update translations for tails-greeter
by translation＠torproject.org 13 Aug '14

13 Aug '14

commit 9efb5960ede8d3605d4db1f8e0fd374d5c80d477 Author: Translation commit bot <translation(a)torproject.org> Date: Wed Aug 13 13:45:22 2014 +0000 Update translations for tails-greeter --- nn/nn.po | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nn/nn.po b/nn/nn.po index 8edb82e..e1abca0 100644 --- a/nn/nn.po +++ b/nn/nn.po @@ -8,7 +8,7 @@ msgstr "" "Project-Id-Version: The Tor Project\n" "Report-Msgid-Bugs-To: \n" "POT-Creation-Date: 2014-07-20 18:36+0200\n" -"PO-Revision-Date: 2014-08-13 13:11+0000\n" +"PO-Revision-Date: 2014-08-13 13:20+0000\n" "Last-Translator: runasand <runa.sandvik(a)gmail.com>\n" "Language-Team: Norwegian Nynorsk (http://www.transifex.com/projects/p/torproject/language/nn/)\n" "MIME-Version: 1.0\n"

1 0