May 2015 - tor-commits - lists.torproject.org

[tor-browser-spec/master] Update 'General Fingerprinting Defenses' with some ideas.
by mikeperry＠torproject.org 06 May '15

06 May '15

commit b8e7bb6abe402218e447f233a1083efaf217321e Author: Mike Perry <mikeperry-git(a)torproject.org> Date: Tue May 5 14:52:25 2015 -0700 Update 'General Fingerprinting Defenses' with some ideas. --- design-doc/design.xml | 67 +++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 56 insertions(+), 11 deletions(-) diff --git a/design-doc/design.xml b/design-doc/design.xml index 715e950..05d2e2b 100644 --- a/design-doc/design.xml +++ b/design-doc/design.xml @@ -1581,12 +1581,16 @@ url="https://amiunique.org/">Am I Unique</ulink>. </listitem> </orderedlist> </sect3> - <sect3 id="fingerprinting-defenses-general"> <title>General Fingerprinting Defenses</title> <para> + +XXX: Stategies vs approaches? Approaches will include things like +virtualization, spoofing, reimplementation, permissions, and disabling features.. + Without looking at a particular fingerprinting vector there are basically two strategies to thwart fingerprinting attacks in general: + <orderedlist> <listitem> Making users uniform: This would render fingerprinting moot as it only works @@ -1598,23 +1602,64 @@ strategies to thwart fingerprinting attacks in general: identify a user upon (re-)visit of a website which is rendering fingerprinting ineffective. </listitem> + <listitem>Virtualization..</listitem> + <listitem>Disabling features</listitem> </orderedlist> -Although there is some research <ulink url="http://research.microsoft.com/pubs/209989/tr1.pdf">suggesting</ulink> the second approach -we think the former is currently a better suited heuristic for Tor Browser -for a couple of reasons: + +Although there is some research <ulink +url="http://research.microsoft.com/pubs/209989/tr1.pdf">suggesting</ulink> the +second approach we think the former is currently a better suited heuristic for +Tor Browser for a couple of reasons: + <itemizedlist> <listitem> - It might not be possible to randomize all fingerprintable characteristics. + +It might not be possible to randomize all fingerprintable characteristics. +While it seems plausible that many end-user configuration details that the +browser currently exposes may be replaced by false information, this approach +seems to break down when it is applied to deeper issues. In particular, it is +not clear how to randomize the capabilities of hardware attached to a computer +in such a way that it convincingly behaves like other hardware, while still +providing a consistent experience to the user from site to site. Similarly, +concealing operating system version differences through randomization will +require an implementation of the underlying support code for every version +your randomization is trying to mimick. + +In both cases, randomizatin requires virtualization of many underlying +implementations, where as uniformity only requires virtualization of one +implementation. + +XXX Virtualization + + </listitem> + <listitem> +Usability. </listitem> <listitem> - It might not be easy to randomize values in a way that they are not - distinguishable from noise. + +It might not be easy to randomize values in a way that they are not +distinguishable from noise. In particular, naive randomization + + </listitem> + <listitem> + +Hard to measure success. + </listitem> <listitem> - Randomizing involves performance costs. This is especially true if the - fingerprinting surface is large (like in a modern browser) and one needs - more elaborate randomizing strategies to make the result - indistinguishable from noise. + +Completeness. Randomization may provide a false sense of security - any items +that are not randomized, or for which the randomization can be averaged away +will still be desirable targets. + + </listitem> + <listitem> + +Randomizing involves performance costs. This is especially true if the +fingerprinting surface is large (like in a modern browser) and one needs more +elaborate randomizing strategies (including randomized virtualization) to +ensure that the randomization fully conceals the true behavior. + </listitem> <listitem> Randomizing itself might introduce a new fingerprinting vector as the

1 0

[tor-browser-spec/master] Updating the fingerprinting section
by mikeperry＠torproject.org 06 May '15

06 May '15

commit 5578c09d4143b30b305204c0f0bd5f1eda4a377d Author: Georg Koppen <gk(a)torproject.org> Date: Tue May 5 10:45:50 2015 +0000 Updating the fingerprinting section --- design-doc/design.xml | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/design-doc/design.xml b/design-doc/design.xml index cf0d959..715e950 100644 --- a/design-doc/design.xml +++ b/design-doc/design.xml @@ -1581,6 +1581,55 @@ url="https://amiunique.org/">Am I Unique</ulink>. </listitem> </orderedlist> </sect3> + + <sect3 id="fingerprinting-defenses-general"> + <title>General Fingerprinting Defenses</title> + <para> +Without looking at a particular fingerprinting vector there are basically two +strategies to thwart fingerprinting attacks in general: +<orderedlist> + <listitem> + Making users uniform: This would render fingerprinting moot as it only works + if there are detectable differences between targets. + </listitem> + <listitem> + Giving randomized values back: This would bury the real device + characteristics within noise. That way a fingerprinter cannot be sure to + identify a user upon (re-)visit of a website which is rendering + fingerprinting ineffective. + </listitem> +</orderedlist> +Although there is some research <ulink url="http://research.microsoft.com/pubs/209989/tr1.pdf">suggesting</ulink> the second approach +we think the former is currently a better suited heuristic for Tor Browser +for a couple of reasons: + <itemizedlist> + <listitem> + It might not be possible to randomize all fingerprintable characteristics. + </listitem> + <listitem> + It might not be easy to randomize values in a way that they are not + distinguishable from noise. + </listitem> + <listitem> + Randomizing involves performance costs. This is especially true if the + fingerprinting surface is large (like in a modern browser) and one needs + more elaborate randomizing strategies to make the result + indistinguishable from noise. + </listitem> + <listitem> + Randomizing itself might introduce a new fingerprinting vector as the + process of generating the values for the fingerprintable attributes + could be susceptible to timing side-channel attacks. + </listitem> + </itemizedlist> + We'll see in the next section that the idea of making users uniform does not + work either in the general way expressed above mainly due to usability issues. + However, we believe that it avoids a lot of the complications involved in + randomization even if just used as a guiding principle. + </para> + </sect3> + + <sect3 id="fingerprinting-defenses"> <title>Fingerprinting Defenses in the Tor Browser</title> <para>

1 0

[tor-browser-spec/master] Separate general fingerprinting defesnes from randomiation discussion.
by mikeperry＠torproject.org 06 May '15

06 May '15

commit 646e0e732053c9e91b8194fb5f3f83babe115460 Author: Mike Perry <mikeperry-git(a)torproject.org> Date: Tue May 5 17:09:24 2015 -0700 Separate general fingerprinting defesnes from randomiation discussion. --- design-doc/design.xml | 217 ++++++++++++++++++++++++++++++++++--------------- 1 file changed, 153 insertions(+), 64 deletions(-) diff --git a/design-doc/design.xml b/design-doc/design.xml index 05d2e2b..47caa6e 100644 --- a/design-doc/design.xml +++ b/design-doc/design.xml @@ -1585,98 +1585,187 @@ url="https://amiunique.org/">Am I Unique</ulink>. <title>General Fingerprinting Defenses</title> <para> -XXX: Stategies vs approaches? Approaches will include things like -virtualization, spoofing, reimplementation, permissions, and disabling features.. +When implemented after an API or feature has been standardized and widely +deployed, defenses to fingerprinting issues tend to take one of the following +forms: value spoofing, subsystem reimplementation, virtualization, site +permissions, and feature removal. -Without looking at a particular fingerprinting vector there are basically two -strategies to thwart fingerprinting attacks in general: + </para> + <orderedlist> + <listitem><command>Value Spoofing</command> + <para> + +Value spoofing can be used for simple cases where the browser directly provides some +aspect of the user's configuration details, devices, hardware, or operating +system directly to a website. It becomes less useful when the fingerprinting +method is instead relying on API behavior. + + </para> + </listitem> + <listitem><command>Subsystem Reimplementation</command> + <para> + +In cases where simple spoofing is not enough to properly conceal underlying +device characteristics or operating system details, the underlying +susbsystem that provides the functionality for a feature or API may need +to be completely reimplemented. This is most common in cases where +customizable or version-specific aspects of the user's operating system are +visible through the browser's featureset or APIs, usually because the browser +directly exposes OS-provided implementations of underlying features. In these +cases, such OS-provided implementations must be replaced by a generic +implementation, or at least an implementation wrapper that makes effort to +conceal any user-customized aspects of the system. -<orderedlist> - <listitem> - Making users uniform: This would render fingerprinting moot as it only works - if there are detectable differences between targets. + </para> </listitem> - <listitem> - Giving randomized values back: This would bury the real device - characteristics within noise. That way a fingerprinter cannot be sure to - identify a user upon (re-)visit of a website which is rendering - fingerprinting ineffective. + <listitem><command>Virtualization</command> + <para> + +Virtualization is needed when simply reimplementing a feature in a different +way is insufficient to fully conceal the underlying behavior. This is most +common in instances of device and hardware fingerprinting, but since the +notion of time can also be virtualized, it also can apply to any instance +where an accurate measure of wallclock time is required for a fingerprinting +vector to attain high accuracy. + + </para> </listitem> - <listitem>Virtualization..</listitem> - <listitem>Disabling features</listitem> -</orderedlist> + <listitem><command>Site Permissions</command> + <para> -Although there is some research <ulink -url="http://research.microsoft.com/pubs/209989/tr1.pdf">suggesting</ulink> the -second approach we think the former is currently a better suited heuristic for -Tor Browser for a couple of reasons: +In the event that virtualization is too expensive in terms of performance or +engineering effort, and the relative expected usage of a feature is rare, site +permissions can be used to prevent the usage of a feature execpt in cases +where the user actually wishes to use it. Unfortunately, this mechanism +becomes less effective once a feature becomes widely overused and abused by +many websites, as warning fatigue quickly sets in for most users. - <itemizedlist> - <listitem> + </para> + </listitem> + <listitem><command>Feature/Functionality Removal</command> + <para> + +When extremely invasive features serve only a narrow domain or usecase, or +there are alternate ways of accomplishing the same task, features and/or +certain aspects of their functionality may be simply removed. -It might not be possible to randomize all fingerprintable characteristics. -While it seems plausible that many end-user configuration details that the -browser currently exposes may be replaced by false information, this approach -seems to break down when it is applied to deeper issues. In particular, it is -not clear how to randomize the capabilities of hardware attached to a computer -in such a way that it convincingly behaves like other hardware, while still -providing a consistent experience to the user from site to site. Similarly, -concealing operating system version differences through randomization will -require an implementation of the underlying support code for every version -your randomization is trying to mimick. + </para> + </listitem> + </orderedlist> + </sect3> + <sect3> + <title>Randomization or Uniformity?</title> + <para> -In both cases, randomizatin requires virtualization of many underlying -implementations, where as uniformity only requires virtualization of one -implementation. +When applying a form of defense to a specific fingerprinting vector or source, +there are two general strategies available. Either the implementation for all +users of a single browser implementation can be made to behave as uniformly as +possible, or the user agent can attempt to randomize its behavior, so that +each interaction between a user and a site provides a different fingerprint. -XXX Virtualization + </para> + <para> - </listitem> - <listitem> -Usability. - </listitem> - <listitem> +Although <ulink url="http://research.microsoft.com/pubs/209989/tr1.pdf">some +research suggests</ulink> that randomization can be effective, so far striving +for uniformity has generally proved to be a better strategy for Tor Browser +for the following reasons: -It might not be easy to randomize values in a way that they are not -distinguishable from noise. In particular, naive randomization + </para> + <orderedlist> + <listitem><command>Randomization is not a shortcut</command> + <para> + +While it appears that many end-user configuration details that the browser +currently exposes may be safely replaced by false information, randomization +of these details must be just as exhaustive as an approach that seeks to make +these behaviors uniform. In the face of either strategy, the adversary can +still make use of those features which have not been altered to be either +sufficiently uniform or sufficiently random. + + </para> + <para> + +Furthermore, the randomization approach seems to break down when it is applied +to deeper issues where underlying system functionality is directly exposed. In +particular, it is not clear how to randomize the capabilities of hardware +attached to a computer in such a way that it either convincingly behaves like +other hardware, or where the exact properties of the hardware that vary from +user to user are sufficiently randomized. Similarly, truly concealing operating +system version differences through randomization may require reimplementation +of the underlying operating system functionality to ensure that every version +that your randomization is trying to blend in with is covered by the range of +possible behaviors. + + </para> </listitem> - <listitem> + <listitem><command>Evaluation and measurement difficulties</command> + <para> + +The fact that randomization causes behaviors to differ slightly with every +visit makes it appealing at first glance, but this same property makes it very +difficult to objectively measure its effectiveness. By contrast, an +implementation that strives for uniformity is very simple to measure. Despite +their current flaws, a properly designed version of <ulink +url="https://panopticlick.eff.org/">Panopticlick</ulink> or <ulink +url="https://amiunique.org/">Am I Unique</ulink> could report the entropy and +uniqueness rates for all users of a single user agent version, without the +need for complicated statistics about the variance of the measured behaviors. + + </para> + <para> -Hard to measure success. +Randomization (especially incomplete randomization) may also provide a false +sense of security. When a fingerprinting attempt makes naive use of randomized +information, a fingerprint will appear unstable, but may not actually be +sufficiently randomized to prevent a dedicated adversary. Sophisticated +fingerprinting mechanisms may either ignore randomized information, or +incorportate knowledge of the distribution and range of randomized values into +the creation of a more stable fingerprint (by either removing the randomness, +modeling it, or averaging it). + </para> </listitem> - <listitem> + <listitem><command>Usability issues</command> + <para> -Completeness. Randomization may provide a false sense of security - any items -that are not randomized, or for which the randomization can be averaged away -will still be desirable targets. +When randomization is introduced to features that affect site behavior, it can +be very distracting for this behavior to change between visits of a given +site. For simple cases such as when this information affects layout behavior, +this will lead to visual nuisances. However, when this information affects +reported functionality or hardware characteristics, sometimes a site will +function one way on one visit, and another way on a subsequent visit. + </para> </listitem> - <listitem> + <listitem><command>Performance costs</command> + + <para> Randomizing involves performance costs. This is especially true if the fingerprinting surface is large (like in a modern browser) and one needs more elaborate randomizing strategies (including randomized virtualization) to -ensure that the randomization fully conceals the true behavior. +ensure that the randomization fully conceals the true behavior. Many calls to +a cryptographically secure random number generator during the course of a page +load will both serve to exhaust available entropy pools, as well as lead to +increased computation while loading a page. + </para> </listitem> - <listitem> - Randomizing itself might introduce a new fingerprinting vector as the - process of generating the values for the fingerprintable attributes - could be susceptible to timing side-channel attacks. - </listitem> - </itemizedlist> - We'll see in the next section that the idea of making users uniform does not - work either in the general way expressed above mainly due to usability issues. - However, we believe that it avoids a lot of the complications involved in - randomization even if just used as a guiding principle. - </para> - </sect3> + <listitem><command>Increased vulnerability surface</command> + <para> +Randomizing itself might introduce a new fingerprinting vector as the process +of generating the values for the fingerprintable attributes could be itself +susceptible to side-channel attacks, analysis, or exploitation. + </para> + </listitem> + </orderedlist> + </sect3> <sect3 id="fingerprinting-defenses"> - <title>Fingerprinting Defenses in the Tor Browser</title> + <title>Specific Fingerprinting Defenses in the Tor Browser</title> <para> The following defenses are listed roughly in order of most severe

1 0

[tor-browser-spec/master] Make all list headers bold.
by mikeperry＠torproject.org 06 May '15

06 May '15

commit f2867cf3dea9232e98069b578f3cbfafec9bc124 Author: Mike Perry <mikeperry-git(a)torproject.org> Date: Tue May 5 17:17:54 2015 -0700 Make all list headers bold. Docbook SGML is really showing its limitations here... :/ --- design-doc/design.xml | 82 ++++++++++++++++++++++++------------------------- 1 file changed, 41 insertions(+), 41 deletions(-) diff --git a/design-doc/design.xml b/design-doc/design.xml index 47caa6e..96a232b 100644 --- a/design-doc/design.xml +++ b/design-doc/design.xml @@ -887,7 +887,7 @@ are typically linked for these cases. Proxy obedience is assured through the following: </para> <orderedlist> - <listitem>Firefox proxy settings, patches, and build flags + <listitem><command>Firefox proxy settings, patches, and build flags</command> <para> Our <ulink @@ -941,7 +941,7 @@ as SMB URLs and other custom protocol handlers are all blocked. </para> </listitem> - <listitem>Disabling plugins + <listitem><command>Disabling plugins</command> <para>Plugins have the ability to make arbitrary OS system calls and <ulink url="http://decloak.net/">bypass proxy settings</ulink>. This includes @@ -973,7 +973,7 @@ address space</ulink> until they are explicitly enabled. </para> </listitem> - <listitem>External App Blocking and Drag Event Filtering + <listitem><command>External App Blocking and Drag Event Filtering</command> <para> External apps can be induced to load files that perform network activity. @@ -997,7 +997,7 @@ Torbutton</ulink> before the OS downloads the URLs the events contained. </para> </listitem> - <listitem>Disabling system extensions and clearing the addon whitelist + <listitem><command>Disabling system extensions and clearing the addon whitelist</command> <para> Firefox addons can perform arbitrary activity on your computer, including @@ -1164,7 +1164,7 @@ date: </para> <orderedlist> - <listitem>Cookies + <listitem><command>Cookies</command> <para><command>Design Goal:</command> All cookies MUST be double-keyed to the url bar origin and third-party @@ -1184,7 +1184,7 @@ unlinkability trumps that desire. </para> </listitem> - <listitem>Cache + <listitem><command>Cache</command> <para> In Firefox, there are actually two distinct caching mechanisms: One for @@ -1207,7 +1207,7 @@ this cache per url bar domain</ulink>. </para> </listitem> - <listitem>HTTP Auth + <listitem><command>HTTP Authentication</command> <para> HTTP Authorization headers can be used to encode <ulink @@ -1219,7 +1219,7 @@ to nsHTTPChannel</ulink>. </para> </listitem> - <listitem>DOM Storage + <listitem><command>DOM Storage</command> <para> DOM storage for third party domains MUST be isolated to the url bar origin, @@ -1230,7 +1230,7 @@ to Firefox</ulink>. </para> </listitem> - <listitem>Flash cookies + <listitem><command>Flash cookies</command> <para><command>Design Goal:</command> Users should be able to click-to-play flash objects from trusted sites. To @@ -1249,7 +1249,7 @@ file on Windows, so Flash remains difficult to enable. </para> </listitem> - <listitem>SSL+TLS session resumption + <listitem><command>SSL+TLS session resumption</command> <para><command>Design Goal:</command> TLS session resumption tickets and SSL Session IDs MUST be limited to the url @@ -1270,7 +1270,7 @@ False Start</ulink> via the Firefox Pref <command>security.ssl.enable_false_start</command>. </para> </listitem> - <listitem>Tor circuit and HTTP connection linkability + <listitem><command>Tor circuit and HTTP connection linkability</command> <para> Tor circuits and HTTP connections from a third party in one URL bar origin @@ -1293,7 +1293,7 @@ password authentication, providing us with HTTP Keep-Alive unlinkability. </para> </listitem> - <listitem>SharedWorkers + <listitem><command>SharedWorkers</command> <para> <ulink @@ -1315,7 +1315,7 @@ For now, we disable SharedWorkers via the pref </para> </listitem> - <listitem>blob: URIs (URL.createObjectURL) + <listitem><command>blob: URIs (URL.createObjectURL)</command> <para> The <ulink @@ -1340,7 +1340,7 @@ WebWorkers. </para> </listitem> - <listitem>SPDY + <listitem><command>SPDY</command> <para> Because SPDY can store identifiers, it is disabled through the @@ -1348,7 +1348,7 @@ Firefox preference <command>network.http.spdy.enabled</command>. </para> </listitem> - <listitem>Automated cross-origin redirects MUST NOT store identifiers + <listitem><command>Automated cross-origin redirects</command> <para><command>Design Goal:</command> To prevent attacks aimed at subverting the Cross-Origin Identifier @@ -1375,7 +1375,7 @@ open</ulink> to implement what we can. </para> </listitem> - <listitem>window.name + <listitem><command>window.name</command> <para> <ulink @@ -1397,7 +1397,7 @@ https/http schemes, the property is cleared. </para> </listitem> - <listitem>Auto form-fill + <listitem><command>Auto form-fill</command> <para> We disable the password saving functionality in the browser as part of our @@ -1411,7 +1411,7 @@ appear, setting this preference prevents automatic linkability from stored passw </para> </listitem> - <listitem>HSTS supercookies + <listitem><command>HSTS supercookies</command> <para> An extreme (but not impossible) attack to mount is the creation of <ulink @@ -1784,7 +1784,7 @@ Status</command>. </para> <orderedlist> - <listitem>Plugins + <listitem><command>Plugins</command> <para> Plugins add to fingerprinting risk via two main vectors: their mere presence @@ -1821,7 +1821,7 @@ leaking plugin installation information. </para> </listitem> - <listitem>HTML5 Canvas Image Extraction + <listitem><command>HTML5 Canvas Image Extraction</command> <para> After plugins and plugin-provided information, we believe that the <ulink @@ -1854,7 +1854,7 @@ image data, pure white image data is returned to the Javascript APIs. <para> </para> </listitem> - <listitem>Open TCP Port and Local Network Fingerprinting + <listitem><command>Open TCP Port and Local Network Fingerprinting</command> <para> In Firefox, by using either WebSockets or XHR, it is possible for remote @@ -1883,7 +1883,7 @@ mechanism. </para> </listitem> - <listitem>Invasive Authentication Mechanisms (NTLM and SPNEGO) + <listitem><command>Invasive Authentication Mechanisms (NTLM and SPNEGO)</command> <para> Both NTLM and SPNEGO authentication mechanisms can leak the hostname, and in @@ -1897,7 +1897,7 @@ placed behind a site permission before their use. We simply disable them. </para> </listitem> - <listitem>USB Device ID Enumeration + <listitem><command>USB Device ID Enumeration</command> <para> The <ulink @@ -1912,7 +1912,7 @@ We simply disable it via the pref <command>dom.gamepad.enabled</command>. </para> </listitem> - <listitem>Fonts + <listitem><command>Fonts</command> <para> According to the Panopticlick study, fonts provide the most linkability when @@ -1967,7 +1967,7 @@ font (in any order), we use that font instead of any of the named local fonts. </para> </listitem> - <listitem>Monitor, Widget, and OS Desktop Resolution + <listitem><command>Monitor, Widget, and OS Desktop Resolution</command> <para> Both CSS and Javascript have access to a lot of information about the screen @@ -2021,7 +2021,7 @@ maximized windows are detrimental to privacy in this mode. </para> </listitem> - <listitem>Display Media information + <listitem><command>Display Media information</command> <para> Beyond simple resolution information, a large amount of so-called "Media" @@ -2057,7 +2057,7 @@ landscape-primary</ulink> for the screen orientation. </para> </listitem> - <listitem>WebGL + <listitem><command>WebGL</command> <para> WebGL is fingerprintable both through information that is exposed about the @@ -2088,7 +2088,7 @@ such a library would avoid hardware-specific rendering differences. </para> </listitem> - <listitem>User Agent and HTTP Headers + <listitem><command>User Agent and HTTP Headers</command> <para><command>Design Goal:</command> All Tor Browser users MUST provide websites with an identical user agent and @@ -2109,7 +2109,7 @@ url="http://pseudo-flaw.net/tor/torbutton/fingerprint-firefox.html">can be used</ulink> to fingerprint OS, platform, and Firefox minor version. </para> </listitem> - <listitem>Locale Fingerprinting + <listitem><command>Locale Fingerprinting</command> <para> In Tor Browser, we provide non-English users the option of concealing their OS @@ -2129,7 +2129,7 @@ and exception handling. </para> </listitem> - <listitem>Timezone and Clock Offset + <listitem><command>Timezone and Clock Offset</command> <para> While the latency in Tor connections varies anywhere from milliseconds to @@ -2158,7 +2158,7 @@ all platforms. </para> </listitem> - <listitem>Javascript Performance Fingerprinting + <listitem><command>Javascript Performance Fingerprinting</command> <para> <ulink url="http://w2spconf.com/2011/papers/jspriv.pdf">Javascript performance @@ -2201,7 +2201,7 @@ Video Statistics</ulink> API extensions via the preference </para> </listitem> - <listitem>Keystroke Fingerprinting + <listitem><command>Keystroke Fingerprinting</command> <para> Keystroke fingerprinting is the act of measuring key strike time and key @@ -2218,7 +2218,7 @@ fingerprinting: timestamp quantization and jitter. We have no implementation as of yet. </para> </listitem> - <listitem>Operating System Type Fingerprinting + <listitem><command>Operating System Type Fingerprinting</command> <para> As we mentioned in the introduction of this section, OS type fingerprinting is @@ -2649,7 +2649,7 @@ address the following additional sources of non-determinism: </para> <orderedlist> - <listitem>Filesystem and archive reordering + <listitem><command>Filesystem and archive reordering</command> <para> The most prevalent source of non-determinism in the components of Tor Browser @@ -2677,7 +2677,7 @@ to aid in reproducible archive creation. </para> </listitem> - <listitem>Uninitialized memory in toolchain/archivers + <listitem><command>Uninitialized memory in toolchain/archivers</command> <para> We ran into difficulties with both binutils and the DMG archive script using @@ -2689,7 +2689,7 @@ patch</ulink>. </para> </listitem> - <listitem>Fine-grained timestamps and timezone leaks + <listitem><command>Fine-grained timestamps and timezone leaks</command> <para> The standard way of controlling timestamps in Gitian is to use libfaketime, @@ -2702,7 +2702,7 @@ script</ulink>. The timezone leaks were addressed by setting the </para> </listitem> - <listitem>Deliberately generated entropy + <listitem><command>Deliberately generated entropy</command> <para> In two circumstances, deliberately generated entropy was introduced in various @@ -2730,7 +2730,7 @@ course, but we handle those another way (as mentioned above). </para> </listitem> - <listitem>LXC-specific leaks + <listitem><command>LXC-specific leaks</command> <para> Gitian provides an option to use LXC containers instead of full qemu-kvm @@ -3158,7 +3158,7 @@ occurring. <sect1 id="deprecate"> <title>Deprecation Wishlist</title> <orderedlist> - <listitem>The Referer Header + <listitem><command>The Referer Header</command> <para> We haven't disabled or restricted the Referer ourselves because of the @@ -3198,7 +3198,7 @@ attribute. </para> </listitem> - <listitem>window.name + <listitem><command>window.name</command> <para> <ulink url="https://developer.mozilla.org/En/DOM/Window.name">window.name</ulink> is @@ -3215,7 +3215,7 @@ cross-origin navigation, but doing so may break federated login for some sites. </para> </listitem> - <listitem>Javascript link rewriting + <listitem><command>Javascript link rewriting</command> <para> In general, it should not be possible for onclick handlers to alter the

1 0

[tor-browser-spec/master] Full spell check.
by mikeperry＠torproject.org 06 May '15

06 May '15

commit 593110b026b1e6c823d1cf96af73aa2d87abb900 Author: Mike Perry <mikeperry-git(a)torproject.org> Date: Tue May 5 17:38:22 2015 -0700 Full spell check. --- design-doc/design.xml | 72 ++++++++++++++++++++++++------------------------- 1 file changed, 36 insertions(+), 36 deletions(-) diff --git a/design-doc/design.xml b/design-doc/design.xml index 96a232b..5765a51 100644 --- a/design-doc/design.xml +++ b/design-doc/design.xml @@ -23,7 +23,7 @@ <address><email>sjmurdoch#torproject org</email></address> </affiliation> </author> - <pubdate>April 30th, 2015</pubdate> + <pubdate>May 5th, 2015</pubdate> </articleinfo> <sect1> @@ -241,9 +241,9 @@ to prefer one browser over another. For the purposes of the unlinkability requirements of this section as well as the descriptions in the <link linkend="Implementation">implementation -section</link>, a <command>url bar origin</command> means at least the +section</link>, a <command>URL bar origin</command> means at least the second-level DNS name. For example, for mail.google.com, the origin would be -google.com. Implementations MAY, at their option, restrict the url bar origin +google.com. Implementations MAY, at their option, restrict the URL bar origin to be the entire fully qualified domain name. </para> @@ -253,8 +253,8 @@ to be the entire fully qualified domain name. Identifier Unlinkability</command></link> <para> -User activity on one url bar origin MUST NOT be linkable to their activity in -any other url bar origin by any third party automatically or without user +User activity on one URL bar origin MUST NOT be linkable to their activity in +any other URL bar origin by any third party automatically or without user interaction or approval. This requirement specifically applies to linkability from stored browser identifiers, authentication tokens, and shared state. The requirement does not apply to linkable information the user manually submits @@ -268,8 +268,8 @@ login in a substantial way. Fingerprinting Unlinkability</command></link> <para> -User activity on one url bar origin MUST NOT be linkable to their activity in -any other url bar origin by any third party. This property specifically applies to +User activity on one URL bar origin MUST NOT be linkable to their activity in +any other URL bar origin by any third party. This property specifically applies to linkability from fingerprinting browser behavior. </para> @@ -345,7 +345,7 @@ Therefore, if plugins are to be enabled in private browsing modes, they must be restricted from running automatically on every page (via click-to-play placeholders), and/or be sandboxed to restrict the types of system calls they can execute. If the user agent allows the user to craft an exemption to allow -a plugin to be used automatically, it must only apply to the top level url bar +a plugin to be used automatically, it must only apply to the top level URL bar domain, and not to all sites, to reduce cross-origin fingerprinting linkability. @@ -367,10 +367,10 @@ system-wide and/or operating system provided addons or plugins. Instead of global browser privacy options, privacy decisions should be made <ulink url="https://wiki.mozilla.org/Privacy/Features/Site-based_data_management_UI">per -url bar origin</ulink> to eliminate the possibility of linkability +URL bar origin</ulink> to eliminate the possibility of linkability between domains. For example, when a plugin object (or a Javascript access of window.plugins) is present in a page, the user should be given the choice of -allowing that plugin object for that url bar origin only. The same +allowing that plugin object for that URL bar origin only. The same goes for exemptions to third party cookie policy, geolocation, and any other privacy permissions. </para> @@ -397,7 +397,7 @@ third parties, rather than a list of specific URLs or hosts. <para> Filter-based addons can also introduce strange breakage and cause usability nightmares, and will also fail to do their job if an adversary simply -registers a new domain or creates a new url path. Worse still, the unique +registers a new domain or creates a new URL path. Worse still, the unique filter sets that each user creates or installs will provide a wealth of fingerprinting targets. </para> @@ -534,7 +534,7 @@ In some cases, the adversary may opt for a heavy-handed approach, such as seizing the computers of all Tor users in an area (especially after narrowing the field by the above two pieces of information). History records and cache data are the primary goals here. Secondary goals may include confirming -on-disk identifiers (such as hostname and disk-logged spoofed MAC adddress +on-disk identifiers (such as hostname and disk-logged spoofed MAC address history) obtained by other means. </para> @@ -699,7 +699,7 @@ AdBlock and other privacy filters can be used to fingerprint request patterns Javascript can reveal a lot of fingerprinting information. It provides DOM objects such as window.screen and window.navigator to extract information -about the useragent. +about the user agent. Also, Javascript can be used to query the user's timezone via the <function>Date()</function> object, <ulink @@ -931,7 +931,7 @@ activity in the source tree that did not use the browser proxy settings. We have verified that these settings and patches properly proxy HTTPS, OCSP, HTTP, FTP, gopher (now defunct), DNS, SafeBrowsing Queries, all JavaScript -activity, including HTML5 audio and video objects, addon updates, wifi +activity, including HTML5 audio and video objects, addon updates, WiFi geolocation queries, searchbox queries, XPCOM addon HTTPS/HTTP activity, WebSockets, and live bookmark updates. We have also verified that IPv6 connections are not attempted, through the proxy or otherwise (Tor does not @@ -1126,10 +1126,10 @@ referred to as "double-keying". The benefit of this approach comes not only in the form of reduced linkability, but also in terms of simplified privacy UI. If all stored browser -state and permissions become associated with the url bar origin, the six or +state and permissions become associated with the URL bar origin, the six or seven different pieces of privacy UI governing these identifiers and permissions can become just one piece of UI. For instance, a window that lists -the url bar origin for which browser state exists, possibly with a +the URL bar origin for which browser state exists, possibly with a context-menu option to drill down into specific types of state or permissions. An example of this simplification can be seen in Figure 1. @@ -1144,7 +1144,7 @@ An example of this simplification can be seen in Figure 1. This example UI is a mock-up of how isolating identifiers to the URL bar origin can simplify the privacy UI for all data - not just cookies. Once -browser identifiers and site permissions operate on a url bar basis, the same +browser identifiers and site permissions operate on a URL bar basis, the same privacy window can represent browsing history, DOM Storage, HTTP Auth, search form history, login values, and so on within a context menu for each site. @@ -1167,11 +1167,11 @@ date: <listitem><command>Cookies</command> <para><command>Design Goal:</command> -All cookies MUST be double-keyed to the url bar origin and third-party +All cookies MUST be double-keyed to the URL bar origin and third-party origin. There exists a <ulink url="https://bugzilla.mozilla.org/show_bug.cgi?id=565965">Mozilla bug</ulink> that contains a prototype patch, but it lacks UI, and does not apply to modern -Firefoxes. +Firefox versions. </para> <para><command>Implementation Status:</command> @@ -1203,7 +1203,7 @@ property prepended, which will list the FQDN that was used to source it. Additionally, because the image cache is a separate entity from the content cache, we had to patch Firefox to also <ulink url="https://gitweb.torproject.org/tor-browser.git/commit/?h=tor-browser-31.6.0e…">isolate -this cache per url bar domain</ulink>. +this cache per URL bar domain</ulink>. </para> </listitem> @@ -1222,7 +1222,7 @@ to nsHTTPChannel</ulink>. <listitem><command>DOM Storage</command> <para> -DOM storage for third party domains MUST be isolated to the url bar origin, +DOM storage for third party domains MUST be isolated to the URL bar origin, to prevent linkability between sites. This functionality is provided through a <ulink url="https://gitweb.torproject.org/tor-browser.git/commit/?h=tor-browser-31.6.0e…">patch @@ -1252,7 +1252,7 @@ file on Windows, so Flash remains difficult to enable. <listitem><command>SSL+TLS session resumption</command> <para><command>Design Goal:</command> -TLS session resumption tickets and SSL Session IDs MUST be limited to the url +TLS session resumption tickets and SSL Session IDs MUST be limited to the URL bar origin. </para> @@ -1355,8 +1355,8 @@ To prevent attacks aimed at subverting the Cross-Origin Identifier Unlinkability <link linkend="privacy">privacy requirement</link>, the browser MUST NOT store any identifiers (cookies, cache, DOM storage, HTTP auth, etc) for cross-origin redirect intermediaries that do not prompt for user input. -For example, if a user clicks on a bit.ly url that redirects to a -doubleclick.net url that finally redirects to a cnn.com url, only cookies from +For example, if a user clicks on a bit.ly URL that redirects to a +doubleclick.net URL that finally redirects to a cnn.com URL, only cookies from cnn.com should be retained after the redirect chain completes. </para> @@ -1393,7 +1393,7 @@ that utilize this property to function, we reset the window.name property of tabs in Torbutton every time we encounter a blank Referer. This behavior allows window.name to persist for the duration of a click-driven navigation session, but as soon as the user enters a new URL or navigates between -https/http schemes, the property is cleared. +HTTPS/HTTP schemes, the property is cleared. </para> </listitem> @@ -1425,7 +1425,7 @@ cookies based on stored HSTS state. There appears to be three options for us: 1. Disable HSTS entirely, and rely instead on HTTPS-Everywhere to crawl and ship rules for HSTS sites. 2. -Restrict the number of HSTS-enabled third parties allowed per url bar origin. +Restrict the number of HSTS-enabled third parties allowed per URL bar origin. 3. Prevent third parties from storing HSTS rules. We have not yet decided upon the best approach. @@ -1607,7 +1607,7 @@ method is instead relying on API behavior. In cases where simple spoofing is not enough to properly conceal underlying device characteristics or operating system details, the underlying -susbsystem that provides the functionality for a feature or API may need +subsystem that provides the functionality for a feature or API may need to be completely reimplemented. This is most common in cases where customizable or version-specific aspects of the user's operating system are visible through the browser's featureset or APIs, usually because the browser @@ -1625,7 +1625,7 @@ Virtualization is needed when simply reimplementing a feature in a different way is insufficient to fully conceal the underlying behavior. This is most common in instances of device and hardware fingerprinting, but since the notion of time can also be virtualized, it also can apply to any instance -where an accurate measure of wallclock time is required for a fingerprinting +where an accurate measure of wall clock time is required for a fingerprinting vector to attain high accuracy. </para> @@ -1635,7 +1635,7 @@ vector to attain high accuracy. In the event that virtualization is too expensive in terms of performance or engineering effort, and the relative expected usage of a feature is rare, site -permissions can be used to prevent the usage of a feature execpt in cases +permissions can be used to prevent the usage of a feature except in cases where the user actually wishes to use it. Unfortunately, this mechanism becomes less effective once a feature becomes widely overused and abused by many websites, as warning fatigue quickly sets in for most users. @@ -1645,7 +1645,7 @@ many websites, as warning fatigue quickly sets in for most users. <listitem><command>Feature/Functionality Removal</command> <para> -When extremely invasive features serve only a narrow domain or usecase, or +When extremely invasive features serve only a narrow domain or use case, or there are alternate ways of accomplishing the same task, features and/or certain aspects of their functionality may be simply removed. @@ -1721,7 +1721,7 @@ sense of security. When a fingerprinting attempt makes naive use of randomized information, a fingerprint will appear unstable, but may not actually be sufficiently randomized to prevent a dedicated adversary. Sophisticated fingerprinting mechanisms may either ignore randomized information, or -incorportate knowledge of the distribution and range of randomized values into +incorporate knowledge of the distribution and range of randomized values into the creation of a more stable fingerprint (by either removing the randomness, modeling it, or averaging it). @@ -1916,7 +1916,7 @@ We simply disable it via the pref <command>dom.gamepad.enabled</command>. <para> According to the Panopticlick study, fonts provide the most linkability when -they are provided as an enumerable list in filesystem order, via either the +they are provided as an enumerable list in file system order, via either the Flash or Java plugins. However, it is still possible to use CSS and/or Javascript to query for the existence of specific fonts. With a large enough pre-built list to query, a large amount of fingerprintable information may @@ -1936,7 +1936,7 @@ and <ulink url="https://fedorahosted.org/lohit/">Lohit fonts</ulink>. The Droid font set is fairly complete by itself, but Nanum and Lohit have smaller versions of many South Asian languages. When combined in a way that chooses the smallest font implementations for each locale, these three font sets provide -poverage for the all languages used on Wikipedia with more than +coverage for the all languages used on Wikipedia with more than 10,000 articles, and several others as well, in approximately 3MB of compressed overhead. The <ulink url="https://www.google.com/get/noto/">Noto font set</ulink> is another font set that aims for complete coverage, but is @@ -2461,7 +2461,7 @@ encrypted website activity. We want to deploy a mechanism that reduces the accuracy of <ulink url="https://en.wikipedia.org/wiki/Feature_selection">useful features</ulink> available for classification. This mechanism would either impact the true and false -positive accuracy rates, <emphasis>or</emphasis> reduce the number of webpages +positive accuracy rates, <emphasis>or</emphasis> reduce the number of web pages that could be classified at a given accuracy rate. </para> @@ -2655,7 +2655,7 @@ address the following additional sources of non-determinism: The most prevalent source of non-determinism in the components of Tor Browser by far was various ways that archives (such as zip, tar, jar/ja, DMG, and Firefox manifest lists) could be reordered. Many file archivers walk the -filesystem in inode structure order by default, which will result in ordering +file system in inode structure order by default, which will result in ordering differences between two different archive invocations, especially on machines of different disk and hardware configurations. @@ -3166,7 +3166,7 @@ non-trivial number of sites that rely on the Referer header to "authenticate" image requests and deep-link navigation on their sites. Furthermore, there seems to be no real privacy benefit to taking this action by itself in a vacuum, because many sites have begun encoding Referer URL information into -GET parameters when they need it to cross http to https scheme transitions. +GET parameters when they need it to cross HTTP to HTTPS scheme transitions. Google's +1 buttons are the best example of this activity. </para>

1 0

[tor/master] Fix a bug in format_changelog, in a silly way
by nickm＠torproject.org 06 May '15

06 May '15

commit 993b4b89183492c4cec916ee8ed514394c088dff Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue May 5 18:23:56 2015 -0400 Fix a bug in format_changelog, in a silly way --- scripts/maint/format_changelog.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/maint/format_changelog.py b/scripts/maint/format_changelog.py index d1b4a3d..a557fca 100755 --- a/scripts/maint/format_changelog.py +++ b/scripts/maint/format_changelog.py @@ -59,7 +59,7 @@ def generate_wrapping(words, divisions): w = words[last:i] last = i line = " ".join(w).replace("\xff ","-").replace("\xff","-") - lines.append(line) + lines.append(line.strip()) return lines def wrapping_quality(words, divisions, width1, width2):

1 0

[tor/master] Tweak changelog entries a bit
by nickm＠torproject.org 06 May '15

06 May '15

commit a2a6a19db165dd08b2dd6e7e493f21d2336a85d0 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue May 5 18:24:19 2015 -0400 Tweak changelog entries a bit --- ChangeLog | 31 ++++++++++++++----------------- 1 file changed, 14 insertions(+), 17 deletions(-) diff --git a/ChangeLog b/ChangeLog index f5ef3d7..1397382 100644 --- a/ChangeLog +++ b/ChangeLog @@ -2,14 +2,14 @@ Changes in version 0.2.7.1-alpha - 2015-05-?? Tor 0.2.7.1-alpha is the first alpha release in its series. o New system requirements: - - Tor no longer includes workarounds for Libevent versions before + - Tor no longer includes workarounds to support Libevent versions before 1.3e. Libevent 2.0 or later is recommended. Closes ticket 15248. o Major features (controller): - - Add the ADD_ONION and DEL_ONION commands that allows the creation + - Add the ADD_ONION and DEL_ONION commands that allow the creation and management of hidden services via the controller. Closes ticket 6411. - - New "GETINFO onions/current" and "GETINFO onions/detached" to get + - New "GETINFO onions/current" and "GETINFO onions/detached" commands to get information about hidden services created via the controller. Part of ticket 6411. - New HSFETCH command to launch a request for a hidden service @@ -18,33 +18,30 @@ Changes in version 0.2.7.1-alpha - 2015-05-?? ticket 3523. Patch by "DonnchaC". o Major bugfixes (hidden services): - - Revert commit that made directory authority assign the HSDir flag - to relay without a DirPort which is bad because relay can't handle + - Revert commit that made directory authorities assign the HSDir flag + to relay without a DirPort; this was bad because such relays can't handle BEGIN_DIR cells. Fixes bug 15850; bugfix on tor-0.2.6.3-alpha. o Minor features (clock-jump tolerance): - Recover better when our clock jumps back many hours, like might happen for Tails or Whonix users who start with a very wrong hardware clock, use Tor to discover a more accurate time, and then - fix their clock. Resolves part of ticket 8766. [I'd call this a - major feature if it actually fixed all of the issues.] + fix their clock. Resolves part of ticket 8766. o Minor features (command-line interface): - Make --hash-password imply --hush to prevent unnecessary noise. - Closes ticket 15542. + Closes ticket 15542. Patch from "cypherpunks". - Print a warning whenever we find a relative file path being used as torrc option. Resolves issue 14018. - - The "--hash-password" option now implies "--hush" to avoid - needless noise. Closes ticket 15542. Patch from "cypherpunks". o Minor features (controller): - - Add DirAuthority lines for default directory authorities to output - of the GETINFO config/defaults controller command if not already + - Add DirAuthority lines for default directory authorities to the output + of the "GETINFO config/defaults" command if not already present. Implements ticket 14840. - - Controllers can now use GETINFO hs/client/desc/id/... to retrieve + - Controllers can now use "GETINFO hs/client/desc/id/..." to retrieve items from the client's hidden service descriptor cache. Closes ticket 14845. - - Implement a new controller command "status/fresh-relay-descs" to + - Implement a new controller command "GETINFO status/fresh-relay-descs" to fetch a descriptor/extrainfo pair that was generated on demand just for the controller's use. Implements ticket 14784. @@ -61,9 +58,9 @@ Changes in version 0.2.7.1-alpha - 2015-05-?? o Minor features (HS popularity countermeasure): - To avoid leaking HS popularity, don't cycle the introduction point when we've handled a fixed number of INTRODUCE2 cells but instead - cycle it when a random value of introductions is reached thus + cycle it when a random number of introductions is reached, thus making it more difficult for an attacker to find out the amount of - clients that has passed through the introduction point for a + clients that have used the introduction point for a specific HS. Closes ticket 15745. o Minor features (logging): @@ -72,7 +69,7 @@ Changes in version 0.2.7.1-alpha - 2015-05-?? ticket 15026. o Minor features (pluggable transports): - - When launching managed pluggable transports on linux systems, + - When launching managed pluggable transports on Linux systems, attempt to have the kernel deliver a SIGTERM on tor exit if the pluggable transport process is still running. Resolves ticket 15471.

1 0

[tor-browser-spec/master] Typo fixes
by mikeperry＠torproject.org 06 May '15

06 May '15

commit 1d29509236ec28adbd4e90df5d55349aaf0d9dad Author: Georg Koppen <gk(a)torproject.org> Date: Tue May 5 08:17:56 2015 +0000 Typo fixes --- design-doc/design.xml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/design-doc/design.xml b/design-doc/design.xml index 88f6426..3a652be 100644 --- a/design-doc/design.xml +++ b/design-doc/design.xml @@ -1115,7 +1115,7 @@ $HOME environment variable to be the TBB extraction directory. The Cross-Origin Identifier Unlinkability design requirement is satisfied through first party isolation of all browser identifier sources. First party isolation means that all identifier sources and browser state are scoped -(isolated) using the the URL bar domain. This scoping is performed in +(isolated) using the URL bar domain. This scoping is performed in combination with any additional third party scope. When first party isolation is used with explicit identifier storage that already has a constrained third party scope (such as cookies, DOM storage, and cache), this approach is @@ -1544,7 +1544,7 @@ specific version of a system can be inferred. While somewhat outside the scope of browser fingerprinting, for completeness it is important to mention that users themselves theoretically might be fingerprinted through their behavior while interacting with a website. This -behavior includes as keystrokes, mouse movements, click speed, and writing +behavior includes e.g. keystrokes, mouse movements, click speed, and writing style. Basic vectors such as keystroke and mouse usage fingerprinting can be mitigated by altering Javascript's notion of time. More advanced issues like writing style fingerprinting are the domain of <ulink

1 0

[tor-browser-spec/master] Improve fingerprinting sources intro.
by mikeperry＠torproject.org 06 May '15

06 May '15

commit f875020671b624b849d9280e761ad7588812f945 Author: Mike Perry <mikeperry-git(a)torproject.org> Date: Tue May 5 13:58:32 2015 -0700 Improve fingerprinting sources intro. --- design-doc/design.xml | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/design-doc/design.xml b/design-doc/design.xml index 3a652be..cf0d959 100644 --- a/design-doc/design.xml +++ b/design-doc/design.xml @@ -1476,10 +1476,18 @@ severe, and how to study the efficacy of defenses properly. <title>Sources of Fingerprinting Issues</title> <para> -All fingerprinting issues arise from one of four primary sources in the -browser. Additionally, user behavior itself provides one more source of -potential fingerprinting. Listed in order from most severe to least severe in -terms of the amount of information they reveal, these sources are: +All browser fingerprinting issues arise from one of four primary sources: +end-user configuration details, device and hardware characteristics, operating +system vendor and version differences, and browser vendor and version +differences. Additionally, user behavior itself provides one more source of +potential fingerprinting. + + </para> + <para> + +In order to help prioritize and inform defenses, we now list these sources in +order from most severe to least severe in terms of the amount of information +they reveal, and describe them in more detail. </para> <orderedlist> @@ -1574,7 +1582,7 @@ url="https://amiunique.org/">Am I Unique</ulink>. </orderedlist> </sect3> <sect3 id="fingerprinting-defenses"> - <title>Fingerprinting defenses in the Tor Browser</title> + <title>Fingerprinting Defenses in the Tor Browser</title> <para> The following defenses are listed roughly in order of most severe

1 0

[tor/master] Merge remote-tracking branch 'public/bug15821_025'
by nickm＠torproject.org 05 May '15

05 May '15

commit b0ea36d779a0d9e5d94bdb056fb20b93240e404c Merge: caaaea2 2b27ce5 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue May 5 15:06:57 2015 -0400 Merge remote-tracking branch 'public/bug15821_025' changes/bug15823 | 4 ++++ src/or/rendservice.c | 18 ++++++++++++++---- 2 files changed, 18 insertions(+), 4 deletions(-)

1 0