November 2017 - tor-commits - lists.torproject.org

[tor/master] add an explanatory comment about the error codes
by nickm＠torproject.org 21 Nov '17

21 Nov '17

commit ea13a47791c1a6b50fed27322a073c54868ac738 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Nov 21 12:36:32 2017 -0500 add an explanatory comment about the error codes --- src/or/conscache.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/or/conscache.c b/src/or/conscache.c index 4919dba3d..33a549597 100644 --- a/src/or/conscache.c +++ b/src/or/conscache.c @@ -539,6 +539,10 @@ consensus_cache_rescan(consensus_cache_t *cache) map = storage_dir_map_labeled(cache->dir, fname, &labels, &body, &bodylen); if (! map) { + /* The ERANGE error might come from tor_mmap_file() -- it means the file + * was empty. EINVAL might come from ..map_labeled() -- it means the + * file was misformatted. In both cases, we should just delete it. + */ if (errno == ERANGE || errno == EINVAL) { log_warn(LD_FS, "Found %s file %s in consensus cache; removing it.", errno == ERANGE ? "empty" : "misformatted",

1 0

[tor/master] Correctly handle partial success in consensus diff calculation.
by nickm＠torproject.org 21 Nov '17

21 Nov '17

commit 3bb29dd707fbc825501c30ed9a6fe4aecdf4fd22 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Nov 20 11:02:40 2017 -0500 Correctly handle partial success in consensus diff calculation. Previously, if store_multiple() reported a partial success, we would store all the handles it gave us as if they had succeeded. But it's possible for the diff to be only partially successful -- for example, if LZMA failed but the other compressors succeeded. Fixes bug 24086; bugfix on 0.3.1.1-alpha. --- changes/bug24086 | 7 +++++++ src/or/consdiffmgr.c | 9 +++++++-- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/changes/bug24086 b/changes/bug24086 new file mode 100644 index 000000000..2ae0b37e6 --- /dev/null +++ b/changes/bug24086 @@ -0,0 +1,7 @@ + o Minor bugfixes (directory cache): + - When a consensus diff calculation is only partially successful, only + record the successful parts as having succeeded. Partial success + can happen if (for example) one compression method fails but + the others succeed. Previously we misrecorded all the calculations as + having succeeded, which would later cause a nonfatal assertion failure. + Fixes bug 24086; bugfix on 0.3.1.1-alpha. diff --git a/src/or/consdiffmgr.c b/src/or/consdiffmgr.c index 831d5d45c..8c25ff201 100644 --- a/src/or/consdiffmgr.c +++ b/src/or/consdiffmgr.c @@ -1589,8 +1589,13 @@ consensus_diff_worker_replyfn(void *work_) for (u = 0; u < ARRAY_LENGTH(handles); ++u) { compress_method_t method = compress_diffs_with[u]; if (cache) { - cdm_diff_ht_set_status(flav, from_sha3, to_sha3, method, status, - handles[u]); + consensus_cache_entry_handle_t *h = handles[u]; + int this_status = status; + if (h == NULL) { + this_status = CDM_DIFF_ERROR; + } + tor_assert_nonfatal(h != NULL || this_status == CDM_DIFF_ERROR); + cdm_diff_ht_set_status(flav, from_sha3, to_sha3, method, this_status, h); } else { consensus_cache_entry_handle_free(handles[u]); }

1 0

[tor/master] Merge branches 'bug24099_031' and 'bug24086_031' into maint-0.3.2
by nickm＠torproject.org 21 Nov '17

21 Nov '17

commit ba94dc28e8e96a64420d42b97ba38e715a35e74c Merge: da82e0b57 ea13a4779 68c21860e Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Nov 21 12:37:46 2017 -0500 Merge branches 'bug24099_031' and 'bug24086_031' into maint-0.3.2 changes/bug24086 | 7 +++++++ changes/bug24099 | 4 ++++ src/common/storagedir.c | 25 +++++++++++++++++++++---- src/or/conscache.c | 17 ++++++++++++++--- src/or/consdiffmgr.c | 13 +++++++++++-- 5 files changed, 57 insertions(+), 9 deletions(-)

1 0

[tor/master] Add another assertion to check for 24086 root causes
by nickm＠torproject.org 21 Nov '17

21 Nov '17

commit 68c21860e32ca04d77c2bfbf7576b96de5110f59 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Nov 20 11:04:44 2017 -0500 Add another assertion to check for 24086 root causes In cdm_diff_ht_set_status(), we shouldn't have been allowing the status CDM_DIFF_PRESENT to be set if there wasn't actually a handle. --- src/or/consdiffmgr.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/or/consdiffmgr.c b/src/or/consdiffmgr.c index 8c25ff201..c5f55b6f3 100644 --- a/src/or/consdiffmgr.c +++ b/src/or/consdiffmgr.c @@ -283,6 +283,10 @@ cdm_diff_ht_set_status(consensus_flavor_t flav, int status, consensus_cache_entry_handle_t *handle) { + if (handle == NULL) { + tor_assert_nonfatal(status != CDM_DIFF_PRESENT); + } + struct cdm_diff_t search, *ent; memset(&search, 0, sizeof(cdm_diff_t)); search.flavor = flav;

1 0

[tor/master] Merge branch 'maint-0.3.2'
by nickm＠torproject.org 21 Nov '17

21 Nov '17

commit 517032b8ce14ac2f468e0fba66da43212c00a02e Merge: b1ab360ec ba94dc28e Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Nov 21 12:41:39 2017 -0500 Merge branch 'maint-0.3.2' changes/bug24086 | 7 +++++++ changes/bug24099 | 4 ++++ src/common/storagedir.c | 25 +++++++++++++++++++++---- src/or/conscache.c | 17 ++++++++++++++--- src/or/consdiffmgr.c | 13 +++++++++++-- 5 files changed, 57 insertions(+), 9 deletions(-)

1 0

[tor/release-0.3.2] Recover better from empty/invalid storagedir files
by nickm＠torproject.org 21 Nov '17

21 Nov '17

commit c8ee12b2e8108658d647aedb92885311291b6f71 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Nov 20 10:14:41 2017 -0500 Recover better from empty/invalid storagedir files If we can't read a file because of an FS issue, we say "we can't read that" and move on. But if we can't read it because it's empty, because it has no labels, or because its labels are misformatted, we should remove it. Fixes bug 24099; bugfix on 0.3.1.1-alpha. --- changes/bug24099 | 4 ++++ src/or/conscache.c | 13 ++++++++++--- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/changes/bug24099 b/changes/bug24099 new file mode 100644 index 000000000..dca399266 --- /dev/null +++ b/changes/bug24099 @@ -0,0 +1,4 @@ + o Minor bugfixes (directory cache): + - Recover better from empty or corrupt files in the consensus cache + directory. Fixes bug 24099; bugfix on 0.3.1.1-alpha. + diff --git a/src/or/conscache.c b/src/or/conscache.c index 9e13ce8e4..4919dba3d 100644 --- a/src/or/conscache.c +++ b/src/or/conscache.c @@ -539,9 +539,16 @@ consensus_cache_rescan(consensus_cache_t *cache) map = storage_dir_map_labeled(cache->dir, fname, &labels, &body, &bodylen); if (! map) { - /* Can't load this; continue */ - log_warn(LD_FS, "Unable to map file %s from consensus cache: %s", - escaped(fname), strerror(errno)); + if (errno == ERANGE || errno == EINVAL) { + log_warn(LD_FS, "Found %s file %s in consensus cache; removing it.", + errno == ERANGE ? "empty" : "misformatted", + escaped(fname)); + storage_dir_remove_file(cache->dir, fname); + } else { + /* Can't load this; continue */ + log_warn(LD_FS, "Unable to map file %s from consensus cache: %s", + escaped(fname), strerror(errno)); + } continue; } consensus_cache_entry_t *ent =

1 0

[tor/release-0.3.2] add an explanatory comment about the error codes
by nickm＠torproject.org 21 Nov '17

21 Nov '17

commit ea13a47791c1a6b50fed27322a073c54868ac738 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Nov 21 12:36:32 2017 -0500 add an explanatory comment about the error codes --- src/or/conscache.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/or/conscache.c b/src/or/conscache.c index 4919dba3d..33a549597 100644 --- a/src/or/conscache.c +++ b/src/or/conscache.c @@ -539,6 +539,10 @@ consensus_cache_rescan(consensus_cache_t *cache) map = storage_dir_map_labeled(cache->dir, fname, &labels, &body, &bodylen); if (! map) { + /* The ERANGE error might come from tor_mmap_file() -- it means the file + * was empty. EINVAL might come from ..map_labeled() -- it means the + * file was misformatted. In both cases, we should just delete it. + */ if (errno == ERANGE || errno == EINVAL) { log_warn(LD_FS, "Found %s file %s in consensus cache; removing it.", errno == ERANGE ? "empty" : "misformatted",

1 0

[tor/release-0.3.2] Merge branches 'bug24099_031' and 'bug24086_031' into maint-0.3.2
by nickm＠torproject.org 21 Nov '17

21 Nov '17

commit ba94dc28e8e96a64420d42b97ba38e715a35e74c Merge: da82e0b57 ea13a4779 68c21860e Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Nov 21 12:37:46 2017 -0500 Merge branches 'bug24099_031' and 'bug24086_031' into maint-0.3.2 changes/bug24086 | 7 +++++++ changes/bug24099 | 4 ++++ src/common/storagedir.c | 25 +++++++++++++++++++++---- src/or/conscache.c | 17 ++++++++++++++--- src/or/consdiffmgr.c | 13 +++++++++++-- 5 files changed, 57 insertions(+), 9 deletions(-)

1 0

[tor/release-0.3.2] Correctly handle partial success in consensus diff calculation.
by nickm＠torproject.org 21 Nov '17

21 Nov '17

commit 3bb29dd707fbc825501c30ed9a6fe4aecdf4fd22 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Nov 20 11:02:40 2017 -0500 Correctly handle partial success in consensus diff calculation. Previously, if store_multiple() reported a partial success, we would store all the handles it gave us as if they had succeeded. But it's possible for the diff to be only partially successful -- for example, if LZMA failed but the other compressors succeeded. Fixes bug 24086; bugfix on 0.3.1.1-alpha. --- changes/bug24086 | 7 +++++++ src/or/consdiffmgr.c | 9 +++++++-- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/changes/bug24086 b/changes/bug24086 new file mode 100644 index 000000000..2ae0b37e6 --- /dev/null +++ b/changes/bug24086 @@ -0,0 +1,7 @@ + o Minor bugfixes (directory cache): + - When a consensus diff calculation is only partially successful, only + record the successful parts as having succeeded. Partial success + can happen if (for example) one compression method fails but + the others succeed. Previously we misrecorded all the calculations as + having succeeded, which would later cause a nonfatal assertion failure. + Fixes bug 24086; bugfix on 0.3.1.1-alpha. diff --git a/src/or/consdiffmgr.c b/src/or/consdiffmgr.c index 831d5d45c..8c25ff201 100644 --- a/src/or/consdiffmgr.c +++ b/src/or/consdiffmgr.c @@ -1589,8 +1589,13 @@ consensus_diff_worker_replyfn(void *work_) for (u = 0; u < ARRAY_LENGTH(handles); ++u) { compress_method_t method = compress_diffs_with[u]; if (cache) { - cdm_diff_ht_set_status(flav, from_sha3, to_sha3, method, status, - handles[u]); + consensus_cache_entry_handle_t *h = handles[u]; + int this_status = status; + if (h == NULL) { + this_status = CDM_DIFF_ERROR; + } + tor_assert_nonfatal(h != NULL || this_status == CDM_DIFF_ERROR); + cdm_diff_ht_set_status(flav, from_sha3, to_sha3, method, this_status, h); } else { consensus_cache_entry_handle_free(handles[u]); }

1 0

[tor/release-0.3.2] Merge branch 'maint-0.3.2' into release-0.3.2
by nickm＠torproject.org 21 Nov '17

21 Nov '17

commit c6365055a1bf5a20cd17dd2b134ab7bfa2365ae3 Merge: b9be385fd ba94dc28e Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Nov 21 12:41:39 2017 -0500 Merge branch 'maint-0.3.2' into release-0.3.2 changes/bug24086 | 7 +++++++ changes/bug24099 | 4 ++++ src/common/storagedir.c | 25 +++++++++++++++++++++---- src/or/conscache.c | 17 ++++++++++++++--- src/or/consdiffmgr.c | 13 +++++++++++-- 5 files changed, 57 insertions(+), 9 deletions(-)

1 0