May 2011 - tor-commits - lists.torproject.org

[tor/master] Initial draft changelog for 0.2.3.1-alpha
by nickm＠torproject.org 04 May '11

04 May '11

commit 5ee5103f63324d7cf5f4ff8bd0fc822f1d6507d2 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue May 3 21:33:10 2011 -0400 Initial draft changelog for 0.2.3.1-alpha --- ChangeLog | 132 +++++++++++++++++++++++++++++++++++++++ changes/automake_required | 6 -- changes/autoreconf | 5 -- changes/bufferevent-support | 12 ---- changes/bug1982 | 5 -- changes/bug2444 | 4 - changes/bug2559 | 6 -- changes/bug2573 | 3 - changes/bug2702 | 5 -- changes/bug507.event | 4 - changes/cpudetect | 3 - changes/dirreq-stats-default | 4 - changes/enhancement1668 | 4 - changes/enhancement1819 | 4 - changes/enhancement1883 | 4 - changes/eventdns_routerinfo_key | 4 - changes/feature1955 | 4 - changes/feature2345 | 4 - changes/getinfo_process | 3 - changes/heartbeat | 7 -- changes/install-fix | 3 - changes/microdesc_dl | 4 - changes/nodelist | 10 --- changes/prop174_server | 6 -- changes/tls-hash | 5 -- changes/tor-fw-helper | 14 ---- 26 files changed, 132 insertions(+), 133 deletions(-) diff --git a/ChangeLog b/ChangeLog index 7e95671..3d779c3 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,135 @@ +Changes in version 0.2.3.1-alpha - 2011-05-0? + + Tor 0.2.3.1-alpha adds some new experimental features, including support + for an improved network IO backend, IOCP networking on Windows, + microdescriptor caching, "fast-start" support for streams, and automatic + home router configuration. There are also numerous internal improvements + to try to make the code easier for developers to work with. + + o Major features + - Tor can now optionally build with the "bufferevents" buffered IO + backend provided by Libevent 2. To use this feature, make sure you + have the latest possible version of Libevent, and pass the + --enable-bufferevents flag to configure when building Tor from source. + Using this feature will make our networking code more flexible, let us + stack layers on each other, and let us use more efficient zero-copy + transports where available. + - As an experimental feature, Tor can use IOCP for networking on Windows. + Once this code is tuned and optimized, it promises much better + performance than the select-based backend we've used in the past. To + try this feature, you must build Tor with Libevent 2, configure Tor + with the "bufferevents" buffered IO backend, and add "DisableIOCP 0" to + your torrc. There are known bugs here: only try this if you can help + debug it as it breaks. + - The EntryNodes option can now include country codes like {de} or IP + addresses or network masks. Previously we had disallowed these options + because we didn't have an efficient way to keep the list up to + date. Fixes bug 1982. + - Exit nodes now accept and queue data on not-yet-connected streams. + Previously, the client wasn't allowed to send data until the stream was + connected, which slowed down all connections. This change will enable + clients to perform a "fast-start" on streams and send data without + having to wait for a confirmation that the stream has opened. (Patch + from Ian Goldberg; implements the server side of Proposal 174.) + - Tor now has initial support for automatic port mapping on on the many + home routers that support NAT-PMP or UPnP. (Not yet supported on + Windows). To build the support code, you'll need to have libnatpnp + library and/or the libminiupnpc library, and you'll need to enable the + feature specifically by passing "--enable-upnp" and/or + "--enable-natpnp" to configure. To turn it on, use the new + PortForwarding option. + - Caches now download, cache, and sever multiple "flavors" of the + consensus, including a flavor that describes microdescriptors. + - Caches now download, cache, and serve microdescriptors -- small + summaries of router descriptors that are authenticated by all of the + directory authorities. Once enough caches are running this code, + clients will be able to save significant amounts of directory bandwidth + by downloading microdescriptors instead of router descriptors. + + o Minor features: + - Make logging resolution configurable and change default from 1 + millisecond to 1 second. Implements enhancement 1668. + - We log which torrc file we're using on startup. Implements ticket 2444. + - Ordinarily, Tor does not count traffic from private addresses (like + 127.0.0.1 or 10.0.0.1) when calculating rate limits or accounting. + There is now a new option, CountPrivateBandwidth, to disable this + behavior. Patch from Daniel Cagara. + - New --enable-static-tor configure option for building Tor as statically + as possible. Idea, general hackery and thoughts from Alexei Czeskis, + John Gilmore, Jacob Appelbaum. Implements ticket 2702. + - If you set the NumCPUs option to 0, Tor will now try to detect how many + CPUs you have. This is the new default behavior. + - Turn on directory request statistics by default and include them in + extra-info descriptors. Don't break if we have no GeoIP database. + - Relays that set "ConnDirectionStatistics 1" write statistics on the + bidirectional use of connections to disk every 24 hours. + - Add a GeoIP file digest to the extra-info descriptor. Implements + enhancement 1883. + - Add a new 'Heartbeat' log message type to periodically log a message + describing Tor's status at level Notice. This feature is meant for + operators who log at notice, adn want to make sure that their Tor + server is still working. Implementation by George Kadianakis. + + o Minor bugfixes (on 0.2.2.25-alpha): + - When loading the microdesc journal, remember its current size. + In 0.2.2, this helps prevent the microdesc journal from growing + without limit on authorities (who are the only ones to use it in + 0.2.2). Fixes a part of bug 2230; bugfix on 0.2.2.6-alpha. + Fix posted by "cypherpunks." + - The microdesc journal is supposed to get rebuilt only if it is + at least _half_ the length of the store, not _twice_ the length + of the store. Bugfix on 0.2.2.6-alpha; fixes part of bug 2230. + - Authorities now clean their microdesc cache periodically and when + reading from disk initially, not only when adding new descriptors. + This prevents a bug where we could lose microdescriptors. Bugfix + on 0.2.2.6-alpha. + + o Minor features (controller) + - Add a new SIGNAL event to the controller interface so that + controllers can be notified when Tor handles a signal. Resolves + issue 1955. Patch by John Brooks. + - Add a new GETINFO option to get total bytes read and written. Patch + from pipe, revised by atagar. Resolves ticket 2345. + - Implement some GETINFO controller fields to provide information about + the Tor process's pid, euid, username, and resource limits. + + o Build changes + - Our build system requires automake 1.6 or later to create the + Makefile.in files. Previously, you could have used 1.4. + This only affects developers and people building Tor from git; + people who build Tor from the source distribution without changing + the Makefile.am files should be fine. + - Our autogen.sh script uses autoreconf to launch autoconf, automake, and + so on. This is more robust against some of the failure modes + associated with running the autotools pieces on their own. + + o Minor packaging issues: + - On OpenSUSE, create the /var/run/tor directory on startup if it is not + already created. Patch from Andreas Stieger. Fixes bug 2573. + + o Code simplifications and refactoring: + - A major revision to our internal node-selecting and listing logic. + Tor already had at least two major ways to look at the question of + "which Tor servers do we know about": a list of router descriptors, + and a list of entries in the current consensus. With + microdescriptors, we're adding a third. Having so many systems + without an abstraction layer over them was hurting the codebase. + Now, we have a new "node_t" abstraction that presents a consistent + interface to a client's view of a Tor node, and holds (nearly) all + of the mutable state formerly in routerinfo_t and routerstatus_t. + - The helper programs tor-gencert, tor-resolve, and tor-checkkey + no longer link against Libevent: they never used it, but + our library structure used to force them to link it. + + o Removed features: + - Removed some old code to work around even older versions of Tor that + used forked processes to handle DNS requests. Such versions of Tor + are no longer in use as servers. + + o Documentation fixes: + - Correct a broken faq link in the INSTALL file. Fixes bug 2307. + + Changes in version 0.2.2.25-alpha - 2011-04-29 Tor 0.2.2.25-alpha fixes many bugs: hidden service clients are more robust, routers no longer overreport their bandwidth, Win7 should crash diff --git a/changes/automake_required b/changes/automake_required deleted file mode 100644 index bb50355..0000000 --- a/changes/automake_required +++ /dev/null @@ -1,6 +0,0 @@ - o Build changes - - Our build system requires automake 1.6 or later to create the - Makefile.in files. Previously, you could have used 1.4. - This only affects developers and people building Tor from git; - people who build Tor from the source distribution without changing - the Makefile.am files should be fine. diff --git a/changes/autoreconf b/changes/autoreconf deleted file mode 100644 index c08cc7a..0000000 --- a/changes/autoreconf +++ /dev/null @@ -1,5 +0,0 @@ - o Minor build changes: - - Use autoreconf to launch autoconf, automake, etc from autogen.sh. - This is more robust against some of the failure modes associated - with running the autotools chain on its own. - diff --git a/changes/bufferevent-support b/changes/bufferevent-support deleted file mode 100644 index e39c36f..0000000 --- a/changes/bufferevent-support +++ /dev/null @@ -1,12 +0,0 @@ - o Major features - - Tor can now optionally build with the "bufferevents" buffered IO - backend provided by Libevent, when building with Libevent 2.0.7-rc - or later. To use this feature, make sure you have the latest possible - version of Libevent, and run autoconf with the --enable-bufferevents - flag. Using this feature will make our networking code more flexible, - lets us stack layers on each other, and let us use more efficient - zero-copy transports where available. - - As an experimental feature, when using the "bufferevents" buffered - IO backend, Tor can try to use Windows's IOCP networking API. This - is off by default. To turn it on, add "DisableIOCP 0" to your torrc. - diff --git a/changes/bug1982 b/changes/bug1982 deleted file mode 100644 index 30ce2df..0000000 --- a/changes/bug1982 +++ /dev/null @@ -1,5 +0,0 @@ - o Major features: - - Allow EntryNodes to include country codes like {de} or IP addresses - or network masks. Previously we had disallowed these options - because we didn't have an efficient way to keep the list up to - date. Resolves bug 1982. diff --git a/changes/bug2444 b/changes/bug2444 deleted file mode 100644 index 405e8dc..0000000 --- a/changes/bug2444 +++ /dev/null @@ -1,4 +0,0 @@ - o Minor features: - - If we did find a configuration file, log a message saying where we found - it during startup. Implements ticket 2444. - diff --git a/changes/bug2559 b/changes/bug2559 deleted file mode 100644 index 9715eaf..0000000 --- a/changes/bug2559 +++ /dev/null @@ -1,6 +0,0 @@ - o Minor features: - - Ordinarily, Tor does not count traffic from private addresses - (like 127.0.0.1 or 10.0.0.1) when calculating rate limits or - accounting. There is now a new option, CountPrivateBandwidth, to - disable this behavior. Patch from Daniel Cagara. - diff --git a/changes/bug2573 b/changes/bug2573 deleted file mode 100644 index 7a2a802..0000000 --- a/changes/bug2573 +++ /dev/null @@ -1,3 +0,0 @@ - o Minor packaging issues - - Create the /var/run/tor directory on startup on OpenSUSE if it is - not already created. Patch from Andreas Stieger. Fixes bug 2573. diff --git a/changes/bug2702 b/changes/bug2702 deleted file mode 100644 index b119f69..0000000 --- a/changes/bug2702 +++ /dev/null @@ -1,5 +0,0 @@ - o Minor features: - - Implements --enable-static-tor for configure time. Implements ticket - 2702. Idea, general hackery and thoughts from Alexei Czeskis, John - Gilmore, Jacob Appelbaum. - diff --git a/changes/bug507.event b/changes/bug507.event deleted file mode 100644 index 68ef4b1..0000000 --- a/changes/bug507.event +++ /dev/null @@ -1,4 +0,0 @@ - o Code simplifications - - We no longer need link tor-gencert, tor-resolve, or tor-checkkey - against libevent; they don't use them, and our code was already - refactored not to force them to use them. diff --git a/changes/cpudetect b/changes/cpudetect deleted file mode 100644 index eeaa492..0000000 --- a/changes/cpudetect +++ /dev/null @@ -1,3 +0,0 @@ - o Minor features - - If you set the NumCPUs option to 0, Tor will try to detect how many - CPUs you have. This is the new default behavior. diff --git a/changes/dirreq-stats-default b/changes/dirreq-stats-default deleted file mode 100644 index 673be6a..0000000 --- a/changes/dirreq-stats-default +++ /dev/null @@ -1,4 +0,0 @@ - o Minor features: - - Turn on directory request statistics by default and include them in - extra-info descriptors. Don't break if we have no GeoIP database. - diff --git a/changes/enhancement1668 b/changes/enhancement1668 deleted file mode 100644 index 0d9f88f..0000000 --- a/changes/enhancement1668 +++ /dev/null @@ -1,4 +0,0 @@ - o Minor features: - - Make logging resolution configurable and change default from 1 - millisecond to 1 second. Implements enhancement 1668. - diff --git a/changes/enhancement1819 b/changes/enhancement1819 deleted file mode 100644 index 9a3f2af..0000000 --- a/changes/enhancement1819 +++ /dev/null @@ -1,4 +0,0 @@ - o Minor features: - - Relays that set "ConnDirectionStatistics 1" write statistics on the - birectional use of connections to disk every 24 hours. - diff --git a/changes/enhancement1883 b/changes/enhancement1883 deleted file mode 100644 index fe6bc1a..0000000 --- a/changes/enhancement1883 +++ /dev/null @@ -1,4 +0,0 @@ - o Minor features: - - Add GeoIP file digest to extra-info descriptor. Implements - enhancement 1883. - diff --git a/changes/eventdns_routerinfo_key b/changes/eventdns_routerinfo_key deleted file mode 100644 index 46dcb52..0000000 --- a/changes/eventdns_routerinfo_key +++ /dev/null @@ -1,4 +0,0 @@ - o Minor feature removals - - Removed old code to work around versions of Tor so old that they - used multiple forked processes to handle DNS requests. Such - versions of Tor are no longer in use as servers. diff --git a/changes/feature1955 b/changes/feature1955 deleted file mode 100644 index 169b7fd..0000000 --- a/changes/feature1955 +++ /dev/null @@ -1,4 +0,0 @@ - o Minor features (controller) - - Add a new SIGNAL event to the controller interface so that - controllers can be notified when Tor handles a signal. Resolves - issue 1955. Patch by John Brooks. diff --git a/changes/feature2345 b/changes/feature2345 deleted file mode 100644 index 5ab6a0f..0000000 --- a/changes/feature2345 +++ /dev/null @@ -1,4 +0,0 @@ - o Minor features (controller) - - Add GETINFO options to get total bytes read and written. Patch - from pipe, revised by atagar. Resolves ticket 2345. - diff --git a/changes/getinfo_process b/changes/getinfo_process deleted file mode 100644 index c6eb6c0..0000000 --- a/changes/getinfo_process +++ /dev/null @@ -1,3 +0,0 @@ - o Minor features - - Implement some GETINFO controller fields to provide information about - the Tor process's pid, euid, username, and resource limits. diff --git a/changes/heartbeat b/changes/heartbeat deleted file mode 100644 index 008d915..0000000 --- a/changes/heartbeat +++ /dev/null @@ -1,7 +0,0 @@ - o Minor features - - Add a new 'Heartbeat' log message type to periodically log a - message describing Tor's status at level Notice. This feature - is meant for operators who log at notice, adn want to make sure - that their Tor server is still working. Implementation by - George Kadianakis. - diff --git a/changes/install-fix b/changes/install-fix deleted file mode 100644 index fb921eb..0000000 --- a/changes/install-fix +++ /dev/null @@ -1,3 +0,0 @@ - o Minor bugfixes: - - Correct a broken faq link in the INSTALL file. Fixes bug 2307. - diff --git a/changes/microdesc_dl b/changes/microdesc_dl deleted file mode 100644 index aca634c..0000000 --- a/changes/microdesc_dl +++ /dev/null @@ -1,4 +0,0 @@ - o Major features: - - Caches now download and cache all the consensus flavors that - they know about. This allows them to assess which microdescriptors - they need to fetch. diff --git a/changes/nodelist b/changes/nodelist deleted file mode 100644 index 033a6c1..0000000 --- a/changes/nodelist +++ /dev/null @@ -1,10 +0,0 @@ - o Code refactorings - - Unified our node-listing and selecting logic. We had at least - two major ways to look at the question of "which Tor servers do - we know about": our list of router descriptors, and the current - consensus. We're adding a third in microdescriptors. Having - so many systems without an abstraction layer over them was - hurting the codebase. Now, we have a new "node_t" abstraction - that presents a consistent interface to a client's view of - a Tor node, and holds (nearly) all of the mutable state - formerly in routerinfo_t and routerstatus_t. diff --git a/changes/prop174_server b/changes/prop174_server deleted file mode 100644 index a2fb39f..0000000 --- a/changes/prop174_server +++ /dev/null @@ -1,6 +0,0 @@ - o Features: - - Servers now accept and queue data on not-yet-connected - streams. Previously, such data wasn't allowed. This forced - clients to wait for a CONNECTED cell before sending their - data, thus adding a round trip to stream setup. (Patch from - Ian Goldberg; implements the server side of Proposal 174.) diff --git a/changes/tls-hash b/changes/tls-hash deleted file mode 100644 index 884ff91..0000000 --- a/changes/tls-hash +++ /dev/null @@ -1,5 +0,0 @@ - o Code simplifications and refactoring: - - Use SSL_get_ex_data to map SSL objects to tor_tls_t objects: there's - no need for a hash table. - - diff --git a/changes/tor-fw-helper b/changes/tor-fw-helper deleted file mode 100644 index f87d828..0000000 --- a/changes/tor-fw-helper +++ /dev/null @@ -1,14 +0,0 @@ - o Major features: - - Tor now has the ability to wrangle NAT devices like a good network cowbot - with the tor-fw-helper tool. The tor-fw-helper tool supports Apple's - NAT-PMP protocol and the UPnP standard for TCP port mapping. This - optional tool may be enabled at compile time by configuring with - '--enable-upnp' or '--enable-natpmp' or with both. This tool may be - called by hand or by Tor. By configuring the PortForwarding option, Tor - will launch the helper on a regular basis to ensure that the NAT mapping - is regularly updated. Additionally, a user may also specify an - alternative helper by using the PortForwardingHelper option. The helper - may be specified by name or with the full path to the helper. The default - helper is named 'tor-fw-helper' and any alternative helper must take the - tor-fw-helper-spec.txt into account. -

1 0

[tor/master] Remove changes files already merged into release-0.2.2
by nickm＠torproject.org 04 May '11

04 May '11

commit 993536e78e07a36a97b37cfd22f8d109dfa80e4c Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue May 3 20:54:37 2011 -0400 Remove changes files already merged into release-0.2.2 --- changes/bug2230_part1 | 7 ------- changes/bug2230_part2 | 5 ----- changes/bug2230_part4 | 6 ------ 3 files changed, 0 insertions(+), 18 deletions(-) diff --git a/changes/bug2230_part1 b/changes/bug2230_part1 deleted file mode 100644 index 79f7254..0000000 --- a/changes/bug2230_part1 +++ /dev/null @@ -1,7 +0,0 @@ - o Minor bugfixes - - When loading the microdesc journal, remember its current size. - In 0.2.2, this helps prevent the microdesc journal from growing - without limit on authorities (who are the only ones to use it in - 0.2.2). Fixes a part of bug 2230; bugfix on 0.2.2.6-alpha. - Fix posted by "cypherpunks." - diff --git a/changes/bug2230_part2 b/changes/bug2230_part2 deleted file mode 100644 index 2664ecc..0000000 --- a/changes/bug2230_part2 +++ /dev/null @@ -1,5 +0,0 @@ - o Minor bugfixes - - The microdesc journal is supposed to get rebuilt only if it is - at least _half_ the length of the store, not _twice_ the length - of the store. Bugfix on 0.2.2.6-alpha; fixes part of bug 2230. - diff --git a/changes/bug2230_part4 b/changes/bug2230_part4 deleted file mode 100644 index f7721fa..0000000 --- a/changes/bug2230_part4 +++ /dev/null @@ -1,6 +0,0 @@ - o Minor bugfixes: - - Authorities now clean their microdesc cache periodically and when - reading from disk initially, not only when adding new descriptors. - This prevents a bug where we could lose microdescriptors. Bugfix - on 0.2.2.6-alpha. -

1 0

[tor/release-0.2.2] Add missing code to set cache->journal_len when reading microdesc journal
by nickm＠torproject.org 04 May '11

04 May '11

commit 698fa0fc67118bea2ea7720455ebbdee705c8c1a Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue May 3 16:22:31 2011 -0400 Add missing code to set cache->journal_len when reading microdesc journal This could be one reason that authorities' journals would grow without bound; related to bug 2230. Bugfix on 0.2.2.6-alpha. Fix by "cypherpunks". --- changes/bug2230_part1 | 7 +++++++ src/or/microdesc.c | 1 + 2 files changed, 8 insertions(+), 0 deletions(-) diff --git a/changes/bug2230_part1 b/changes/bug2230_part1 new file mode 100644 index 0000000..79f7254 --- /dev/null +++ b/changes/bug2230_part1 @@ -0,0 +1,7 @@ + o Minor bugfixes + - When loading the microdesc journal, remember its current size. + In 0.2.2, this helps prevent the microdesc journal from growing + without limit on authorities (who are the only ones to use it in + 0.2.2). Fixes a part of bug 2230; bugfix on 0.2.2.6-alpha. + Fix posted by "cypherpunks." + diff --git a/src/or/microdesc.c b/src/or/microdesc.c index 2c4b343..3566277 100644 --- a/src/or/microdesc.c +++ b/src/or/microdesc.c @@ -261,6 +261,7 @@ microdesc_cache_reload(microdesc_cache_t *cache) journal_content = read_file_to_str(cache->journal_fname, RFTS_IGNORE_MISSING, &st); if (journal_content) { + cache->journal_len = (size_t) st.st_size; added = microdescs_add_to_cache(cache, journal_content, journal_content+st.st_size, SAVED_IN_JOURNAL, 0);

1 0

[tor/release-0.2.2] Merge branch 'maint-0.2.2' into release-0.2.2
by nickm＠torproject.org 04 May '11

04 May '11

commit e68f9cc9b863b22d4ccdcb711b35965445d0acca Merge: 10d45f8 68ae5af Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue May 3 20:53:16 2011 -0400 Merge branch 'maint-0.2.2' into release-0.2.2 changes/bug2230_clean_1 | 4 ++ changes/bug2230_part1 | 7 +++ changes/bug2230_part2 | 5 ++ changes/bug2230_part4 | 6 +++ doc/HACKING | 42 ++++++++++++++++++++ src/or/config.c | 7 +-- src/or/main.c | 2 + src/or/microdesc.c | 100 +++++++++++++++++++++++++++++++++++++++++----- src/or/microdesc.h | 3 +- 9 files changed, 160 insertions(+), 16 deletions(-)

1 0

[tor/release-0.2.2] Change who calls microdesc_cache_rebuild().
by nickm＠torproject.org 04 May '11

04 May '11

commit 68ae5afa5a172fbc485eda7e6bfd4750fa6a69f5 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue May 3 17:28:28 2011 -0400 Change who calls microdesc_cache_rebuild(). Previously we ensured that it would get called periodically by doing it from inside the code that added microdescriptors. That won't work though: it would interfere with our code that tried to read microdescs from disk initially. Instead, we should consider rebuilding the cache periodically, and on startup. --- changes/bug2230_part4 | 6 ++++++ src/or/main.c | 2 ++ src/or/microdesc.c | 11 ++++++++--- 3 files changed, 16 insertions(+), 3 deletions(-) diff --git a/changes/bug2230_part4 b/changes/bug2230_part4 new file mode 100644 index 0000000..f7721fa --- /dev/null +++ b/changes/bug2230_part4 @@ -0,0 +1,6 @@ + o Minor bugfixes: + - Authorities now clean their microdesc cache periodically and when + reading from disk initially, not only when adding new descriptors. + This prevents a bug where we could lose microdescriptors. Bugfix + on 0.2.2.6-alpha. + diff --git a/src/or/main.c b/src/or/main.c index a26be39..462b51e 100644 --- a/src/or/main.c +++ b/src/or/main.c @@ -1075,6 +1075,8 @@ run_scheduled_events(time_t now) rep_history_clean(now - options->RephistTrackTime); rend_cache_clean(); rend_cache_clean_v2_descs_as_dir(); + if (authdir_mode_v3(options)) + microdesc_cache_rebuild(NULL, 0); #define CLEAN_CACHES_INTERVAL (30*60) time_to_clean_caches = now + CLEAN_CACHES_INTERVAL; } diff --git a/src/or/microdesc.c b/src/or/microdesc.c index c6bb3c6..73d2285 100644 --- a/src/or/microdesc.c +++ b/src/or/microdesc.c @@ -208,8 +208,6 @@ microdescs_add_list_to_cache(microdesc_cache_t *cache, if (f) finish_writing_to_file(open_file); /*XXX Check me.*/ - microdesc_cache_rebuild(cache, 0/* only as needed */); - return added; } @@ -230,6 +228,7 @@ microdesc_cache_clear(microdesc_cache_t *cache) } cache->total_len_seen = 0; cache->n_seen = 0; + cache->bytes_dropped = 0; } /** Reload the contents of cache from disk. If it is empty, load it @@ -271,7 +270,7 @@ microdesc_cache_reload(microdesc_cache_t *cache) log_notice(LD_DIR, "Reloaded microdescriptor cache. Found %d descriptors.", total); - microdesc_cache_clean(cache, 0, 0); + microdesc_cache_rebuild(cache, 0 /* don't force */); return 0; } @@ -354,6 +353,12 @@ microdesc_cache_rebuild(microdesc_cache_t *cache, int force) off_t off = 0; int orig_size, new_size; + if (cache == NULL) { + cache = the_microdesc_cache; + if (cache == NULL) + return 0; + } + /* Remove dead descriptors */ microdesc_cache_clean(cache, 0/*cutoff*/, 0/*force*/);

1 0

[tor/release-0.2.2] Rebuild the microdesc cache when a sufficient number of bytes are dropped
by nickm＠torproject.org 04 May '11

04 May '11

commit cb6c909664f97f751fe78c0aa3205a9042760c53 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue May 3 17:03:49 2011 -0400 Rebuild the microdesc cache when a sufficient number of bytes are dropped Previously on 0.2.2, we'd never clean the cache. Now that we can clean it, we want to add a condition to rebuild it: that should happen whenever we have dropped enough microdescriptors that we could save a lot of space. No changes file, since 0.2.3 doesn't need one and 0.2.2 already has some changes files for the backport of the microdesc_clean_cahce() function. --- src/or/microdesc.c | 45 +++++++++++++++++++++++++++++++++------------ src/or/microdesc.h | 2 +- 2 files changed, 34 insertions(+), 13 deletions(-) diff --git a/src/or/microdesc.c b/src/or/microdesc.c index 58fbe37..c6bb3c6 100644 --- a/src/or/microdesc.c +++ b/src/or/microdesc.c @@ -177,6 +177,8 @@ microdescs_add_list_to_cache(microdesc_cache_t *cache, if (md2->last_listed < md->last_listed) md2->last_listed = md->last_listed; microdesc_free(md); + if (where != SAVED_NOWHERE) + cache->bytes_dropped += size; continue; } @@ -206,14 +208,7 @@ microdescs_add_list_to_cache(microdesc_cache_t *cache, if (f) finish_writing_to_file(open_file); /*XXX Check me.*/ - { - size_t old_content_len = - cache->cache_content ? cache->cache_content->size : 0; - if (cache->journal_len > 16384 + old_content_len && - cache->journal_len > old_content_len / 2) { - microdesc_cache_rebuild(cache); - } - } + microdesc_cache_rebuild(cache, 0/* only as needed */); return added; } @@ -275,6 +270,9 @@ microdesc_cache_reload(microdesc_cache_t *cache) } log_notice(LD_DIR, "Reloaded microdescriptor cache. Found %d descriptors.", total); + + microdesc_cache_clean(cache, 0, 0); + return 0; } @@ -323,11 +321,30 @@ microdesc_cache_clean(microdesc_cache_t *cache, time_t cutoff, int force) } } +static int +should_rebuild_md_cache(microdesc_cache_t *cache) +{ + const size_t old_len = + cache->cache_content ? cache->cache_content->size : 0; + const size_t journal_len = cache->journal_len; + const size_t dropped = cache->bytes_dropped; + + if (journal_len < 16384) + return 0; /* Don't bother, not enough has happened yet. */ + if (dropped > (journal_len + old_len) / 3) + return 1; /* We could save 1/3 or more of the currently used space. */ + if (journal_len > old_len / 2) + return 1; /* We should append to the regular file */ + + return 0; +} + /** Regenerate the main cache file for cache, clear the journal file, * and update every microdesc_t in the cache with pointers to its new - * location. */ + * location. If force is true, do this unconditionally. If + * force is false, do it only if we expect to save space on disk. */ int -microdesc_cache_rebuild(microdesc_cache_t *cache) +microdesc_cache_rebuild(microdesc_cache_t *cache, int force) { open_file_t *open_file; FILE *f; @@ -337,11 +354,14 @@ microdesc_cache_rebuild(microdesc_cache_t *cache) off_t off = 0; int orig_size, new_size; - log_info(LD_DIR, "Rebuilding the microdescriptor cache..."); - /* Remove dead descriptors */ microdesc_cache_clean(cache, 0/*cutoff*/, 0/*force*/); + if (!force && !should_rebuild_md_cache(cache)) + return 0; + + log_info(LD_DIR, "Rebuilding the microdescriptor cache..."); + orig_size = (int)(cache->cache_content ? cache->cache_content->size : 0); orig_size += (int)cache->journal_len; @@ -396,6 +416,7 @@ microdesc_cache_rebuild(microdesc_cache_t *cache) write_str_to_file(cache->journal_fname, "", 1); cache->journal_len = 0; + cache->bytes_dropped = 0; new_size = (int)cache->cache_content->size; log_info(LD_DIR, "Done rebuilding microdesc cache. " diff --git a/src/or/microdesc.h b/src/or/microdesc.h index 30cb25d..77ce853 100644 --- a/src/or/microdesc.h +++ b/src/or/microdesc.h @@ -22,7 +22,7 @@ smartlist_t *microdescs_add_list_to_cache(microdesc_cache_t *cache, int no_save); void microdesc_cache_clean(microdesc_cache_t *cache, time_t cutoff, int force); -int microdesc_cache_rebuild(microdesc_cache_t *cache); +int microdesc_cache_rebuild(microdesc_cache_t *cache, int force); int microdesc_cache_reload(microdesc_cache_t *cache); void microdesc_cache_clear(microdesc_cache_t *cache);

1 0

[tor/release-0.2.2] Backport microdesc_cache_clean to 0.2.2
by nickm＠torproject.org 04 May '11

04 May '11

commit 56fbd728c2df7f189390b9a417b4c3ecc3690ede Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue May 3 16:45:15 2011 -0400 Backport microdesc_cache_clean to 0.2.2 Otherwise we have no way to keep authorities' microdesc caches in 0.2.2 from growing without bound. --- changes/bug2230_clean_1 | 4 +++ src/or/microdesc.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++ src/or/microdesc.h | 1 + 3 files changed, 56 insertions(+), 0 deletions(-) diff --git a/changes/bug2230_clean_1 b/changes/bug2230_clean_1 new file mode 100644 index 0000000..a4edf94 --- /dev/null +++ b/changes/bug2230_clean_1 @@ -0,0 +1,4 @@ + o Minor features + - Backport code from 0.2.3.x to allow directory authorities to clean + their microdescriptor caches. + diff --git a/src/or/microdesc.c b/src/or/microdesc.c index 0ceb134..58fbe37 100644 --- a/src/or/microdesc.c +++ b/src/or/microdesc.c @@ -23,6 +23,8 @@ struct microdesc_cache_t { tor_mmap_t *cache_content; /** Number of bytes used in the journal file. */ size_t journal_len; + /** Number of bytes in descriptors removed as too old. */ + size_t bytes_dropped; /** Total bytes of microdescriptor bodies we have added to this cache */ uint64_t total_len_seen; @@ -276,6 +278,51 @@ microdesc_cache_reload(microdesc_cache_t *cache) return 0; } +/** By default, we remove any microdescriptors that have gone at least this + * long without appearing in a current consensus. */ +#define TOLERATE_MICRODESC_AGE (7*24*60*60) + +/** Remove all microdescriptors from cache that haven't been listed for + * a long time. Does not rebuild the cache on disk. If cutoff is + * positive, specifically remove microdescriptors that have been unlisted + * since cutoff. If force is true, remove microdescriptors even + * if we have no current live microdescriptor consensus. + */ +void +microdesc_cache_clean(microdesc_cache_t *cache, time_t cutoff, int force) +{ + microdesc_t **mdp, *victim; + int dropped=0, kept=0; + size_t bytes_dropped = 0; + time_t now = time(NULL); + + (void) force; + /* In 0.2.2, we let this proceed unconditionally: only authorities have + * microdesc caches. */ + + if (cutoff <= 0) + cutoff = now - TOLERATE_MICRODESC_AGE; + + for (mdp = HT_START(microdesc_map, &cache->map); mdp != NULL; ) { + if ((*mdp)->last_listed < cutoff) { + ++dropped; + victim = *mdp; + mdp = HT_NEXT_RMV(microdesc_map, &cache->map, mdp); + bytes_dropped += victim->bodylen; + microdesc_free(victim); + } else { + ++kept; + mdp = HT_NEXT(microdesc_map, &cache->map, mdp); + } + } + + if (dropped) { + log_notice(LD_DIR, "Removed %d/%d microdescriptors as old.", + dropped,dropped+kept); + cache->bytes_dropped += bytes_dropped; + } +} + /** Regenerate the main cache file for cache, clear the journal file, * and update every microdesc_t in the cache with pointers to its new * location. */ @@ -291,6 +338,10 @@ microdesc_cache_rebuild(microdesc_cache_t *cache) int orig_size, new_size; log_info(LD_DIR, "Rebuilding the microdescriptor cache..."); + + /* Remove dead descriptors */ + microdesc_cache_clean(cache, 0/*cutoff*/, 0/*force*/); + orig_size = (int)(cache->cache_content ? cache->cache_content->size : 0); orig_size += (int)cache->journal_len; diff --git a/src/or/microdesc.h b/src/or/microdesc.h index b3e12f8..30cb25d 100644 --- a/src/or/microdesc.h +++ b/src/or/microdesc.h @@ -21,6 +21,7 @@ smartlist_t *microdescs_add_list_to_cache(microdesc_cache_t *cache, smartlist_t *descriptors, saved_location_t where, int no_save); +void microdesc_cache_clean(microdesc_cache_t *cache, time_t cutoff, int force); int microdesc_cache_rebuild(microdesc_cache_t *cache); int microdesc_cache_reload(microdesc_cache_t *cache); void microdesc_cache_clear(microdesc_cache_t *cache);

1 0

[tor/release-0.2.2] Fix a check for when to rebuild the microdesc cache. (Backport from 0.2.3.
by nickm＠torproject.org 04 May '11

04 May '11

commit 970715dd8f52a79bbaddfcfa5af1f11608ddacf9 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue May 3 16:29:39 2011 -0400 Fix a check for when to rebuild the microdesc cache. (Backport from 0.2.3. --- changes/bug2230_part2 | 5 +++++ src/or/microdesc.c | 2 +- 2 files changed, 6 insertions(+), 1 deletions(-) diff --git a/changes/bug2230_part2 b/changes/bug2230_part2 new file mode 100644 index 0000000..2664ecc --- /dev/null +++ b/changes/bug2230_part2 @@ -0,0 +1,5 @@ + o Minor bugfixes + - The microdesc journal is supposed to get rebuilt only if it is + at least _half_ the length of the store, not _twice_ the length + of the store. Bugfix on 0.2.2.6-alpha; fixes part of bug 2230. + diff --git a/src/or/microdesc.c b/src/or/microdesc.c index 3566277..0ceb134 100644 --- a/src/or/microdesc.c +++ b/src/or/microdesc.c @@ -208,7 +208,7 @@ microdescs_add_list_to_cache(microdesc_cache_t *cache, size_t old_content_len = cache->cache_content ? cache->cache_content->size : 0; if (cache->journal_len > 16384 + old_content_len && - cache->journal_len > old_content_len * 2) { + cache->journal_len > old_content_len / 2) { microdesc_cache_rebuild(cache); } }

1 0

[tor/release-0.2.2] Clean up a warning a bit
by nickm＠torproject.org 04 May '11

04 May '11

commit dbd73b9689e8008adc149472ca5c02692a49923d Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Apr 29 11:14:53 2011 -0400 Clean up a warning a bit --- src/or/config.c | 7 +++---- 1 files changed, 3 insertions(+), 4 deletions(-) diff --git a/src/or/config.c b/src/or/config.c index 8671822..666a1bd 100644 --- a/src/or/config.c +++ b/src/or/config.c @@ -3488,10 +3488,9 @@ options_validate(or_options_t *old_options, or_options_t *options, } if (options->CookieAuthFileGroupReadable && !options->CookieAuthFile) { - log_warn(LD_CONFIG, "You set the CookieAuthFileGroupReadable but did " - "not configure a the path for the cookie file via " - "CookieAuthFile. This means your cookie will not be group " - "readable."); + log_warn(LD_CONFIG, "CookieAuthFileGroupReadable is set, but will have " + "no effect: you must specify an explicit CookieAuthFile to " + "have it group-readable."); } if (options->UseEntryGuards && ! options->NumEntryGuards)

1 0

[tor/release-0.2.2] Update hacking file with terse notes on formatting changelog
by nickm＠torproject.org 04 May '11

04 May '11

commit 676190e895a94978fbda82527f7ba2dbf8ccda80 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Apr 28 23:44:48 2011 -0400 Update hacking file with terse notes on formatting changelog --- doc/HACKING | 33 +++++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-) diff --git a/doc/HACKING b/doc/HACKING index b612953..6407ff2 100644 --- a/doc/HACKING +++ b/doc/HACKING @@ -414,10 +414,43 @@ Here are the steps Roger takes when putting out a new Tor release: and as a directory authority. See if it has any obvious bugs, and resolve those. +1.5) As applicable, merge the maint-X branch into the release-X branch. + 2) Gather the changes/* files into a changelog entry, rewriting many of them and reordering to focus on what users and funders would find interesting and understandable. + 2.1) Make sure that everything that wants a bug number has one. + 2.2) Concatenate them. + 2.3) Sort them by section. Within each section, try to make the + first entry or two and the last entry most interesting: they're + the ones that skimmers tend to read. + + 2.4) Clean them up + + Standard idioms: + "Fixes bug 9999; Bugfix on 0.3.3.3-alpha." + + One period after a space. + + Make stuff very terse + + Describe the user-visible problem right away + + Mention relevant config options by name. If they're rare or unusual, + remind people what they're for + + Avoid starting lines with open-paren + + Present and imperative tense: not past. + + 2.5) Merge them in. + + 2.6) Clean everything one last time. + + 2.7) Run it through fmt to make it pretty. + + 3) Compose a short release blurb to highlight the user-facing changes. Insert said release blurb into the ChangeLog stanza. If it's a stable release, add it to the ReleaseNotes file too. If we're adding

1 0