January 2015 - tor-commits - lists.torproject.org

[tor/master] Merge branch 'bug14001-clang-warning' into bug13111-empty-key-files-fn-empty
by nickm＠torproject.org 12 Jan '15

12 Jan '15

commit c200ab46b85cece87a4bcdbaacd41bc6539d1671 Merge: fd7e9e9 6a9cae2 Author: teor <teor2345(a)gmail.com> Date: Sat Jan 10 16:34:10 2015 +1100 Merge branch 'bug14001-clang-warning' into bug13111-empty-key-files-fn-empty Conflicts: src/or/router.c Choose newer comment. Merge changes to comment and function invocation. README | 3 + changes/bug13126 | 10 + changes/bug13214 | 7 + changes/… [View More]bug13296 | 5 + changes/bug13315 | 5 + changes/bug13399 | 12 + changes/bug13447 | 5 + changes/bug13644 | 4 + changes/bug13678 | 6 + changes/bug13698 | 6 + changes/bug13701 | 4 + changes/bug13707 | 4 + changes/bug13713 | 3 + changes/bug13840 | 3 + changes/bug13941 | 6 + changes/bug13942 | 5 + changes/bug14001-clang-warning | 6 + changes/bug7484 | 4 + changes/bug7803 | 5 + changes/bug9812 | 6 + changes/doc13381 | 5 + changes/feature13212 | 4 + changes/feature9503 | 4 + changes/geoip-november2014 | 3 + changes/geoip6-november2014 | 3 + changes/global_scheduler | 12 + changes/no_global_ccc | 3 + changes/ticket-11291 | 4 + changes/ticket13172 | 4 + changes/tickets6456 | 6 + doc/tor.1.txt | 64 +- scripts/coccinelle/calloc.cocci | 15 +- scripts/maint/checkSpace.pl | 3 +- src/common/address.c | 8 +- src/common/compat.c | 2 +- src/common/compat.h | 9 + src/common/compat_libevent.c | 6 +- src/common/compat_libevent.h | 2 +- src/common/crypto.c | 2 +- src/common/log.c | 2 +- src/common/torlog.h | 4 +- src/common/util.c | 150 +- src/common/util.h | 7 +- src/config/geoip |20036 +++++++++++++++++++++++++------------ src/config/geoip6 | 5649 ++++++++++- src/ext/tinytest_demo.c | 16 +- src/or/Makefile.nmake | 1 + src/or/buffers.c | 18 +- src/or/buffers.h | 2 +- src/or/channel.c | 477 +- src/or/channel.h | 87 +- src/or/channeltls.c | 144 +- src/or/channeltls.h | 2 - src/or/circuitbuild.c | 10 +- src/or/circuitlist.c | 46 +- src/or/circuitlist.h | 3 +- src/or/circuitmux.c | 56 +- src/or/circuitmux.h | 12 +- src/or/circuitmux_ewma.c | 58 +- src/or/circuitstats.c | 6 +- src/or/circuituse.c | 12 +- src/or/config.c | 348 +- src/or/config.h | 5 +- src/or/connection.c | 2 + src/or/connection_edge.c | 15 +- src/or/connection_or.c | 67 +- src/or/connection_or.h | 8 +- src/or/control.c | 38 +- src/or/control.h | 6 +- src/or/cpuworker.c | 2 +- src/or/directory.c | 22 +- src/or/dirserv.c | 18 +- src/or/dirvote.c | 30 +- src/or/entrynodes.c | 4 +- src/or/entrynodes.h | 2 +- src/or/geoip.c | 4 +- src/or/include.am | 2 + src/or/main.c | 51 +- src/or/main.h | 4 +- src/or/microdesc.c | 36 +- src/or/microdesc.h | 2 +- src/or/networkstatus.c | 2 +- src/or/nodelist.c | 2 +- src/or/or.h | 57 +- src/or/relay.c | 15 +- src/or/relay.h | 3 +- src/or/rendclient.c | 25 +- src/or/rendcommon.c | 19 + src/or/rendcommon.h | 1 + src/or/rendservice.c | 244 +- src/or/router.c | 24 +- src/or/router.h | 2 +- src/or/routerlist.c | 288 +- src/or/routerlist.h | 4 +- src/or/scheduler.c | 709 ++ src/or/scheduler.h | 50 + src/or/transports.c | 12 +- src/or/transports.h | 10 +- src/test/Makefile.nmake | 11 +- src/test/fakechans.h | 25 + src/test/include.am | 6 + src/test/test.c | 171 +- src/test/test.h | 2 +- src/test/test_addr.c | 469 +- src/test/test_buffers.c | 338 +- src/test/test_cell_formats.c | 775 +- src/test/test_cell_queue.c | 58 +- src/test/test_channel.c | 1669 +++ src/test/test_channeltls.c | 332 + src/test/test_checkdir.c | 140 + src/test/test_circuitlist.c | 128 +- src/test/test_circuitmux.c | 15 +- src/test/test_config.c | 315 +- src/test/test_containers.c | 391 +- src/test/test_controller_events.c | 52 +- src/test/test_crypto.c | 541 +- src/test/test_dir.c | 710 +- src/test/test_entrynodes.c | 108 +- src/test/test_extorport.c | 187 +- src/test/test_hs.c | 18 +- src/test/test_introduce.c | 2 +- src/test/test_logging.c | 30 +- src/test/test_microdesc.c | 162 +- src/test/test_nodelist.c | 4 +- src/test/test_oom.c | 80 +- src/test/test_options.c | 6 +- src/test/test_policy.c | 40 +- src/test/test_pt.c | 72 +- src/test/test_relay.c | 134 + src/test/test_relaycell.c | 54 +- src/test/test_replay.c | 52 +- src/test/test_routerkeys.c | 16 +- src/test/test_routerset.c | 276 +- src/test/test_scheduler.c | 763 ++ src/test/test_socks.c | 320 +- src/test/test_status.c | 245 +- src/test/test_util.c | 2253 +++-- 137 files changed, 28930 insertions(+), 11239 deletions(-) diff --cc src/or/router.c index 94ae2e7,56bb909..f0317aa --- a/src/or/router.c +++ b/src/or/router.c @@@ -885,9 -880,7 +889,9 @@@ init_keys(void keydir = get_datadir_fname2("keys", "secret_onion_key.old"); if (!lastonionkey && file_status(keydir) == FN_FILE) { - prkey = init_key_from_file(keydir, 1, LOG_ERR, 0); /* XXXX Why 1? */ + /* Load keys from non-empty files only. + * Missing old keys won't be replaced with freshly generated keys. */ - prkey = init_key_from_file(keydir, 0, LOG_ERR); ++ prkey = init_key_from_file(keydir, 0, LOG_ERR, 0); if (prkey) lastonionkey = prkey; } [View Less]

1 0

[tor/master] Test that tor correctly handles zero-length keys
by nickm＠torproject.org 12 Jan '15

12 Jan '15

commit debd7862bb6d641c13fe4ee019da633c683d740e Author: teor <teor2345(a)gmail.com> Date: Sat Jan 10 17:14:29 2015 +1100 Test that tor correctly handles zero-length keys Check that tor generates new keys, and overwrites the empty key files. Test that tor generates new keys when keys are missing (existing behaviour). Test that tor does not overwrite key files that already contain data (existing behaviour). Tests fixes to bug 13111. --- changes/… [View More]bug13111-generate-keys-on-empty-file | 22 +++-- src/test/include.am | 4 +- src/test/zero_length_keys.sh | 113 ++++++++++++++++++++++++++ 3 files changed, 130 insertions(+), 9 deletions(-) diff --git a/changes/bug13111-generate-keys-on-empty-file b/changes/bug13111-generate-keys-on-empty-file index 345e87a..20c10c7 100644 --- a/changes/bug13111-generate-keys-on-empty-file +++ b/changes/bug13111-generate-keys-on-empty-file @@ -1,17 +1,23 @@ - o Minor bugfixes + o Minor bugfixes (file handling): - Stop failing when key files are zero-length. Instead, generate new keys, and overwrite the empty key files. - Fixes bug 13111. + Fixes bug 13111. Patch by "teor". - Stop generating a fresh .old RSA key file when the .old file is missing. - Fixed as part of bug 13111. - Avoid overwriting .old key files with empty key files. - Fixed as part of bug 13111. - Stop crashing when a NULL filename is passed to file_status(). - Fixed as part of bug 13111. + Fixed as part of bug 13111. Patches by "teor". - o Minor enhancements: + o Minor enhancements (file handling): - Skip loading zero-length extra info store, router store, stats, state, and key files. - Implemented with bug 13111. - Return FN_ERROR when a zero-length filename is passed to file_status(). - Fixed as part of bug 13111. + Fixed as part of bug 13111. Patches by "teor". + + o Minor enhancements (testing): + - Test that tor does not fail when key files are zero-length. + Check that tor generates new keys, and overwrites the empty key files. + - Test that tor generates new keys when keys are missing (existing + behaviour). + - Test that tor does not overwrite key files that already contain data + (existing behaviour). + Tests bug 13111. Patch by "teor". diff --git a/src/test/include.am b/src/test/include.am index d7a6479..4518f8f 100644 --- a/src/test/include.am +++ b/src/test/include.am @@ -120,9 +120,11 @@ if USEPYTHON ./src/test/test-bt-cl assert | $(PYTHON) $(top_srcdir)/src/test/bt_test.py ./src/test/test-bt-cl crash | $(PYTHON) $(top_srcdir)/src/test/bt_test.py endif + ./src/test/zero_length_keys.sh EXTRA_DIST += \ src/test/bt_test.py \ src/test/ntor_ref.py \ src/test/slownacl_curve25519.py \ - src/test/test_cmdline_args.py + src/test/test_cmdline_args.py \ + src/test/zero_length_keys.sh diff --git a/src/test/zero_length_keys.sh b/src/test/zero_length_keys.sh new file mode 100755 index 0000000..e7cb900 --- /dev/null +++ b/src/test/zero_length_keys.sh @@ -0,0 +1,113 @@ +#!/bin/sh +# Check that tor regenerates keys when key files are zero-length +# Test for bug #13111 - Tor fails to start if onion keys are zero length +# +# Usage: +# ./zero_length_keys.sh +# Run all the tests below +# ./zero_length_keys.sh -z +# Check tor will launch and regenerate zero-length keys +# ./zero_length_keys.sh -d +# Check tor regenerates deleted keys (existing behaviour) +# ./zero_length_keys.sh -e +# Check tor does not overwrite existing keys (existing behaviour) +# +# Exit Statuses: +# -2: test failed - tor did not generate the key files on first run +# -1: a command failed - the test could not be completed +# 0: test succeeded - tor regenerated/kept the files +# 1: test failed - tor did not regenerate/keep the files +# + +if [ $# -lt 1 ]; then + echo "Testing that tor correctly handles zero-length keys" + "$0" -z && "$0" -d && "$0" -e + exit $? +fi + +export DATA_DIR=`mktemp -d -t tor_zero_length_keys` +TOR="src/or/tor --hush --DisableNetwork 1 --ShutdownWaitLength 0 --ORPort 12345" + +if [ -s "$DATA_DIR"/keys/secret_id_key -a -s "$DATA_DIR"/keys/secret_onion_key -a -s "$DATA_DIR"/keys/secret_onion_key_ntor ]; then + echo "Failure: Previous tor keys present in tor data directory" + exit -1 +else + echo "Generating initial tor keys" + $TOR --DataDirectory "$DATA_DIR" --PidFile "$DATA_DIR"/pid & + TOR_PID=$! + # generate SIGTERM, hopefully after the keys have been regenerated + sleep 5 + kill $TOR_PID + wait $TOR_PID + + # tor must successfully generate non-zero-length key files + if [ -s "$DATA_DIR"/keys/secret_id_key -a -s "$DATA_DIR"/keys/secret_onion_key -a -s "$DATA_DIR"/keys/secret_onion_key_ntor ]; then + true #echo "tor generated the initial key files" + else + echo "Failure: tor failed to generate the initial key files" + exit -2 + fi +fi + +#ls -lh "$DATA_DIR"/keys/ || exit -1 + +# backup and keep/delete/create zero-length files for the keys + +FILE_DESC="keeps existing" +# make a backup +cp -r "$DATA_DIR"/keys "$DATA_DIR"/keys.old + +# delete keys for -d or -z +if [ "$1" != "-e" ]; then + FILE_DESC="regenerates deleted" + rm "$DATA_DIR"/keys/secret_id_key || exit -1 + rm "$DATA_DIR"/keys/secret_onion_key || exit -1 + rm "$DATA_DIR"/keys/secret_onion_key_ntor || exit -1 +fi + +# create empty files for -z +if [ "$1" == "-z" ]; then + FILE_DESC="regenerates zero-length" + touch "$DATA_DIR"/keys/secret_id_key || exit -1 + touch "$DATA_DIR"/keys/secret_onion_key || exit -1 + touch "$DATA_DIR"/keys/secret_onion_key_ntor || exit -1 +fi + +echo "Running tor again to check if it $FILE_DESC keys" +$TOR --DataDirectory "$DATA_DIR" --PidFile "$DATA_DIR"/pid & +TOR_PID=$! +# generate SIGTERM, hopefully after the keys have been regenerated +sleep 5 +kill $TOR_PID +wait $TOR_PID + +#ls -lh "$DATA_DIR"/keys/ || exit -1 + +# tor must always have non-zero-length key files +if [ -s "$DATA_DIR"/keys/secret_id_key -a -s "$DATA_DIR"/keys/secret_onion_key -a -s "$DATA_DIR"/keys/secret_onion_key_ntor ]; then + # check if the keys are different to the old ones + diff -q -r "$DATA_DIR"/keys "$DATA_DIR"/keys.old > /dev/null + SAME_KEYS=$? + # if we're not testing existing keys, + # the current keys should be different to the old ones + if [ "$1" != "-e" ]; then + if [ $SAME_KEYS -ne 0 ]; then + echo "Success: test that tor $FILE_DESC key files: different keys" + exit 0 + else + echo "Failure: test that tor $FILE_DESC key files: same keys" + exit 1 + fi + else #[ "$1" == "-e" ]; then + if [ $SAME_KEYS -eq 0 ]; then + echo "Success: test that tor $FILE_DESC key files: same keys" + exit 0 + else + echo "Failure: test that tor $FILE_DESC key files: different keys" + exit 1 + fi + fi +else + echo "Failure: test that tor $FILE_DESC key files: no key files" + exit 1 +fi [View Less]

1 0

[tor/master] Stop crashing when a NULL filename is passed to file_status()
by nickm＠torproject.org 12 Jan '15

12 Jan '15

commit ce7fd6e160e2e3acb824b29e29afe15cd5e7bf4f Author: teor <teor2345(a)gmail.com> Date: Sat Nov 8 20:03:21 2014 +1100 Stop crashing when a NULL filename is passed to file_status() Stop crashing when a NULL filename is passed to file_status(), instead, return FN_ERROR. Also return FN_ERROR when a zero-length filename is passed to file_status(). Fixed as part of bug 13111. --- changes/bug13111-generate-keys-on-empty-file | 7 +++++++ src/common/util.c … [View More]

1 0

[tor/master] Stop failing when key files are zero-length
by nickm＠torproject.org 12 Jan '15

12 Jan '15

commit fd7e9e9030cee9d8e863cea3f3f90226ae66fdfe Author: teor <teor2345(a)gmail.com> Date: Sun Oct 19 17:48:07 2014 +1100 Stop failing when key files are zero-length Instead, generate new keys, and overwrite the empty key files. Adds FN_EMPTY to file_status_t and file_status. Fixes bug 13111. Related changes due to review of FN_FILE usage: Stop generating a fresh .old RSA key file when the .old file is missing. Avoid overwriting .old key files with … [View More]empty key files. Skip loading zero-length extra info store, router store, stats, state, and key files. --- changes/bug13111-generate-keys-on-empty-file | 10 +++++++ src/common/compat.c | 1 + src/common/util.c | 30 ++++++++++++++------- src/common/util.h | 2 +- src/or/config.c | 22 +++++++++++----- src/or/router.c | 36 +++++++++++++++++++------- src/or/routerlist.c | 1 + src/or/statefile.c | 3 +++ 8 files changed, 79 insertions(+), 26 deletions(-) diff --git a/changes/bug13111-generate-keys-on-empty-file b/changes/bug13111-generate-keys-on-empty-file index f6a56ef..345e87a 100644 --- a/changes/bug13111-generate-keys-on-empty-file +++ b/changes/bug13111-generate-keys-on-empty-file @@ -1,7 +1,17 @@ o Minor bugfixes + - Stop failing when key files are zero-length. Instead, generate new + keys, and overwrite the empty key files. + Fixes bug 13111. + - Stop generating a fresh .old RSA key file when the .old file is missing. + Fixed as part of bug 13111. + - Avoid overwriting .old key files with empty key files. + Fixed as part of bug 13111. - Stop crashing when a NULL filename is passed to file_status(). Fixed as part of bug 13111. o Minor enhancements: + - Skip loading zero-length extra info store, router store, stats, state, + and key files. + Implemented with bug 13111. - Return FN_ERROR when a zero-length filename is passed to file_status(). Fixed as part of bug 13111. diff --git a/src/common/compat.c b/src/common/compat.c index e4758aa..b28790f 100644 --- a/src/common/compat.c +++ b/src/common/compat.c @@ -823,6 +823,7 @@ replace_file(const char *from, const char *to) case FN_NOENT: break; case FN_FILE: + case FN_EMPTY: if (unlink(to)) return -1; break; case FN_ERROR: diff --git a/src/common/util.c b/src/common/util.c index e850b14..1c35338 100644 --- a/src/common/util.c +++ b/src/common/util.c @@ -1888,10 +1888,15 @@ clean_name_for_stat(char *name) #endif } -/** Return FN_ERROR if filename can't be read, or is NULL or zero-length, - * FN_NOENT if it doesn't - * exist, FN_FILE if it is a regular file, or FN_DIR if it's a - * directory. On FN_ERROR, sets errno. */ +/** Return: + * FN_ERROR if filename can't be read, is NULL, or is zero-length, + * FN_NOENT if it doesn't exist, + * FN_FILE if it is a non-empty regular file, or a FIFO on unix-like systems, + * FN_EMPTY for zero-byte regular files, + * FN_DIR if it's a directory, and + * FN_ERROR for any other file type. + * On FN_ERROR and FN_NOENT, sets errno. (errno is not set when FN_ERROR + * is returned due to an unhandled file type.) */ file_status_t file_status(const char *fname) { @@ -1912,16 +1917,23 @@ file_status(const char *fname) } return FN_ERROR; } - if (st.st_mode & S_IFDIR) + if (st.st_mode & S_IFDIR) { return FN_DIR; - else if (st.st_mode & S_IFREG) - return FN_FILE; + } else if (st.st_mode & S_IFREG) { + if (st.st_size > 0) { + return FN_FILE; + } else if (st.st_size == 0) { + return FN_EMPTY; + } else { + return FN_ERROR; + } #ifndef _WIN32 - else if (st.st_mode & S_IFIFO) + } else if (st.st_mode & S_IFIFO) { return FN_FILE; #endif - else + } else { return FN_ERROR; + } } /** Check whether dirname exists and is private. If yes return 0. If diff --git a/src/common/util.h b/src/common/util.h index d7feb6e..c5471ff 100644 --- a/src/common/util.h +++ b/src/common/util.h @@ -334,7 +334,7 @@ enum stream_status get_string_from_pipe(FILE *stream, char *buf, size_t count); /** Return values from file_status(); see that function's documentation * for details. */ -typedef enum { FN_ERROR, FN_NOENT, FN_FILE, FN_DIR } file_status_t; +typedef enum { FN_ERROR, FN_NOENT, FN_FILE, FN_DIR, FN_EMPTY } file_status_t; file_status_t file_status(const char *filename); /** Possible behaviors for check_private_dir() on encountering a nonexistent diff --git a/src/or/config.c b/src/or/config.c index 3f31e87..3e7a73f 100644 --- a/src/or/config.c +++ b/src/or/config.c @@ -4017,17 +4017,24 @@ find_torrc_filename(config_line_t *cmd_arg, if (*using_default_fname) { /* didn't find one, try CONFDIR */ const char *dflt = get_default_conf_file(defaults_file); - if (dflt && file_status(dflt) == FN_FILE) { + file_status_t st = file_status(dflt); + if (dflt && (st == FN_FILE || st == FN_EMPTY)) { fname = tor_strdup(dflt); } else { #ifndef _WIN32 char *fn = NULL; - if (!defaults_file) + if (!defaults_file) { fn = expand_filename("~/.torrc"); - if (fn && file_status(fn) == FN_FILE) { - fname = fn; + } + if (fn) { + file_status_t hmst = file_status(fn); + if (hmst == FN_FILE || hmst == FN_EMPTY) { + fname = fn; + } else { + tor_free(fn); + fname = tor_strdup(dflt); + } } else { - tor_free(fn); fname = tor_strdup(dflt); } #else @@ -4063,7 +4070,8 @@ load_torrc_from_disk(config_line_t *cmd_arg, int defaults_file) *fname_var = fname; /* Open config file */ - if (file_status(fname) != FN_FILE || + file_status_t st = file_status(fname); + if (!(st == FN_FILE || st == FN_EMPTY) || !(cf = read_file_to_str(fname,0,NULL))) { if (using_default_torrc == 1 || ignore_missing_torrc) { if (!defaults_file) @@ -6413,7 +6421,9 @@ write_configuration_file(const char *fname, const or_options_t *options) tor_assert(fname); switch (file_status(fname)) { + /* create backups of old config files, even if they're empty */ case FN_FILE: + case FN_EMPTY: old_val = read_file_to_str(fname, 0, NULL); if (!old_val || strcmpstart(old_val, GENERATED_FILE_PREFIX)) { rename_old = 1; diff --git a/src/or/router.c b/src/or/router.c index 01838b4..94ae2e7 100644 --- a/src/or/router.c +++ b/src/or/router.c @@ -313,6 +313,7 @@ rotate_onion_key(void) time_t now; fname = get_datadir_fname2("keys", "secret_onion_key"); fname_prev = get_datadir_fname2("keys", "secret_onion_key.old"); + /* There isn't much point replacing an old key with an empty file */ if (file_status(fname) == FN_FILE) { if (replace_file(fname, fname_prev)) goto error; @@ -335,6 +336,7 @@ rotate_onion_key(void) fname_prev = get_datadir_fname2("keys", "secret_onion_key_ntor.old"); if (curve25519_keypair_generate(&new_curve25519_keypair, 1) < 0) goto error; + /* There isn't much point replacing an old key with an empty file */ if (file_status(fname) == FN_FILE) { if (replace_file(fname, fname_prev)) goto error; @@ -389,9 +391,9 @@ log_new_relay_greeting(void) already_logged = 1; } -/** Try to read an RSA key from fname. If fname doesn't exist - * and generate is true, create a new RSA key and save it in - * fname. Return the read/created key, or NULL on error. Log all +/** Try to read an RSA key from fname. If fname doesn't exist, + * or is empty, and generate is true, create a new RSA key and save it + * in fname. Return the read/created key, or NULL on error. Log all * errors at level severity. */ crypto_pk_t * @@ -409,7 +411,11 @@ init_key_from_file(const char *fname, int generate, int severity) case FN_ERROR: tor_log(severity, LD_FS,"Can't read key from \"%s\"", fname); goto error; + /* treat empty key files as if the file doesn't exist, and, + * if generate is set, replace the empty file in + * crypto_pk_write_private_key_to_filename() */ case FN_NOENT: + case FN_EMPTY: if (generate) { if (!have_lockfile()) { if (try_locking(get_options(), 0)<0) { @@ -460,10 +466,10 @@ init_key_from_file(const char *fname, int generate, int severity) } /** Load a curve25519 keypair from the file fname, writing it into - * keys_out. If the file isn't found and generate is true, - * create a new keypair and write it into the file. If there are errors, log - * them at level severity. Generate files using tag in their - * ASCII wrapper. */ + * keys_out. If the file isn't found, or is empty, and generate + * is true, create a new keypair and write it into the file. If there are + * errors, log them at level severity. Generate files using tag + * in their ASCII wrapper. */ static int init_curve25519_keypair_from_file(curve25519_keypair_t *keys_out, const char *fname, @@ -476,7 +482,10 @@ init_curve25519_keypair_from_file(curve25519_keypair_t *keys_out, case FN_ERROR: tor_log(severity, LD_FS,"Can't read key from \"%s\"", fname); goto error; + /* treat empty key files as if the file doesn't exist, and, if generate + * is set, replace the empty file in curve25519_keypair_write_to_file() */ case FN_NOENT: + case FN_EMPTY: if (generate) { if (!have_lockfile()) { if (try_locking(get_options(), 0)<0) { @@ -876,7 +885,9 @@ init_keys(void) keydir = get_datadir_fname2("keys", "secret_onion_key.old"); if (!lastonionkey && file_status(keydir) == FN_FILE) { - prkey = init_key_from_file(keydir, 1, LOG_ERR); /* XXXX Why 1? */ + /* Load keys from non-empty files only. + * Missing old keys won't be replaced with freshly generated keys. */ + prkey = init_key_from_file(keydir, 0, LOG_ERR); if (prkey) lastonionkey = prkey; } @@ -897,6 +908,8 @@ init_keys(void) last_curve25519_onion_key.pubkey.public_key, CURVE25519_PUBKEY_LEN) && file_status(keydir) == FN_FILE) { + /* Load keys from non-empty files only. + * Missing old keys won't be replaced with freshly generated keys. */ init_curve25519_keypair_from_file(&last_curve25519_onion_key, keydir, 0, LOG_ERR, "onion"); } @@ -2562,8 +2575,9 @@ router_has_orport(const routerinfo_t *router, const tor_addr_port_t *orport) * end_line, ensure that its timestamp is not more than 25 hours in * the past or more than 1 hour in the future with respect to now, * and write the file contents starting with that line to *out. - * Return 1 for success, 0 if the file does not exist, or -1 if the file - * does not contain a line matching these criteria or other failure. */ + * Return 1 for success, 0 if the file does not exist or is empty, or -1 + * if the file does not contain a line matching these criteria or other + * failure. */ static int load_stats_file(const char *filename, const char *end_line, time_t now, char **out) @@ -2597,7 +2611,9 @@ load_stats_file(const char *filename, const char *end_line, time_t now, notfound: tor_free(contents); break; + /* treat empty stats files as if the file doesn't exist */ case FN_NOENT: + case FN_EMPTY: r = 0; break; case FN_ERROR: diff --git a/src/or/routerlist.c b/src/or/routerlist.c index 0099486..5c08f59 100644 --- a/src/or/routerlist.c +++ b/src/or/routerlist.c @@ -1204,6 +1204,7 @@ router_reload_router_list_impl(desc_store_t *store) tor_free(fname); fname = get_datadir_fname_suffix(store->fname_base, ".new"); + /* don't load empty files - we wouldn't get any data, even if we tried */ if (file_status(fname) == FN_FILE) contents = read_file_to_str(fname, RFTS_BIN|RFTS_IGNORE_MISSING, &st); if (contents) { diff --git a/src/or/statefile.c b/src/or/statefile.c index 2ce53fd..6640aed 100644 --- a/src/or/statefile.c +++ b/src/or/statefile.c @@ -323,7 +323,10 @@ or_state_load(void) goto done; } break; + /* treat empty state files as if the file doesn't exist, and generate + * a new state file, overwriting the empty file in or_state_save() */ case FN_NOENT: + case FN_EMPTY: break; case FN_ERROR: case FN_DIR: [View Less]

1 0

[tor/master] Fix clang warning, IPv6 address comment, buffer size typo
by nickm＠torproject.org 12 Jan '15

12 Jan '15

commit 6a9cae2e1dafb756b30fda541e8b5d68cfd45f89 Author: teor <teor2345(a)gmail.com> Date: Sat Dec 20 22:20:54 2014 +1100 Fix clang warning, IPv6 address comment, buffer size typo The address of an array in the middle of a structure will always be non-NULL. clang recognises this and complains. Disable the tautologous and redundant check to silence this warning. A comment about an IPv6 address string incorrectly refers to an IPv4 address format. … [View More] A log buffer is sized 10024 rather than 10240. Fixes bug 14001. --- changes/bug14001-clang-warning | 6 ++++++ src/common/address.c | 3 ++- src/common/log.c | 2 +- src/or/connection_edge.c | 11 ++++++++++- 4 files changed, 19 insertions(+), 3 deletions(-) diff --git a/changes/bug14001-clang-warning b/changes/bug14001-clang-warning new file mode 100644 index 0000000..b932af6 --- /dev/null +++ b/changes/bug14001-clang-warning @@ -0,0 +1,6 @@ + o Minor bugfixes: + - The address of an array in the middle of a structure will + always be non-NULL. clang recognises this and complains. + Disable the tautologous and redundant check to silence + this warning. + Fixes bug 14001. diff --git a/src/common/address.c b/src/common/address.c index a3b5df6..0b475fc 100644 --- a/src/common/address.c +++ b/src/common/address.c @@ -1119,7 +1119,8 @@ fmt_addr32(uint32_t addr) int tor_addr_parse(tor_addr_t *addr, const char *src) { - char *tmp = NULL; /* Holds substring if we got a dotted quad. */ + /* Holds substring of IPv6 address after removing square brackets */ + char *tmp = NULL; int result; struct in_addr in_tmp; struct in6_addr in6_tmp; diff --git a/src/common/log.c b/src/common/log.c index ad0da7d..0a21ffb 100644 --- a/src/common/log.c +++ b/src/common/log.c @@ -451,7 +451,7 @@ MOCK_IMPL(STATIC void, logv,(int severity, log_domain_mask_t domain, const char *funcname, const char *suffix, const char *format, va_list ap)) { - char buf[10024]; + char buf[10240]; size_t msg_len = 0; int formatted = 0; logfile_t *lf; diff --git a/src/or/connection_edge.c b/src/or/connection_edge.c index 9ace375..9859cc2 100644 --- a/src/or/connection_edge.c +++ b/src/or/connection_edge.c @@ -744,8 +744,17 @@ connection_ap_fail_onehop(const char *failed_digest, /* we don't know the digest; have to compare addr:port */ tor_addr_t addr; if (!build_state || !build_state->chosen_exit || - !entry_conn->socks_request || !entry_conn->socks_request->address) + !entry_conn->socks_request) { + /* clang thinks that an array midway through a structure + * will never have a NULL address, under either: + * -Wpointer-bool-conversion if using !, or + * -Wtautological-pointer-compare if using == or != + * It's probably right (unless pointers overflow and wrap), + * so we just skip this check + || !entry_conn->socks_request->address + */ continue; + } if (tor_addr_parse(&addr, entry_conn->socks_request->address)<0 || !tor_addr_eq(&build_state->chosen_exit->addr, &addr) || build_state->chosen_exit->port != entry_conn->socks_request->port) [View Less]

1 0

[tor/master] When OOM, free cached hidden service descriptors too.
by nickm＠torproject.org 12 Jan '15

12 Jan '15

commit 3033ba9f5ec8eb4dea3e6778f6d7546cacf7ae57 Author: Nick Mathewson <nickm(a)torproject.org> Date: Sun Jan 4 20:05:29 2015 -0500 When OOM, free cached hidden service descriptors too. --- changes/bug13806 | 8 ++++ src/or/main.c | 2 +- src/or/or.h | 2 + src/or/relay.c | 12 +++++ src/or/rendcommon.c | 121 ++++++++++++++++++++++++++++++++++++++++++--------- src/or/rendcommon.h | 4 +- 6 files changed, 126 insertions(+), 23 deletions(-) … [View More]diff --git a/changes/bug13806 b/changes/bug13806 new file mode 100644 index 0000000..0a6b268 --- /dev/null +++ b/changes/bug13806 @@ -0,0 +1,8 @@ + o Minor features (DOS resistance): + - Count the total number of bytes used storing hidden service descriptors + against the value of MaxMemInQueues. If we're low on memory, and more + than 20% of our memory is used holding hidden service descriptors, free + them until no more than 10% of our memory holds hidden service + descriptors. Free the least recently fetched descriptors first. + Resolves ticket 13806. + diff --git a/src/or/main.c b/src/or/main.c index 58e3ad3..3ee9644 100644 --- a/src/or/main.c +++ b/src/or/main.c @@ -1437,7 +1437,7 @@ run_scheduled_events(time_t now) if (time_to_clean_caches < now) { rep_history_clean(now - options->RephistTrackTime); rend_cache_clean(now); - rend_cache_clean_v2_descs_as_dir(now); + rend_cache_clean_v2_descs_as_dir(now, 0); microdesc_cache_rebuild(NULL, 0); #define CLEAN_CACHES_INTERVAL (30*60) time_to_clean_caches = now + CLEAN_CACHES_INTERVAL; diff --git a/src/or/or.h b/src/or/or.h index ee86697..d978719 100644 --- a/src/or/or.h +++ b/src/or/or.h @@ -4939,6 +4939,8 @@ typedef struct rend_service_descriptor_t { typedef struct rend_cache_entry_t { size_t len; /**< Length of desc */ time_t received; /**< When was the descriptor received? */ + time_t last_served; /**< When did we last write this one to somebody? + * (HSDir only) */ char *desc; /**< Service descriptor */ rend_service_descriptor_t *parsed; /**< Parsed value of 'desc' */ } rend_cache_entry_t; diff --git a/src/or/relay.c b/src/or/relay.c index b95e584..50070b7 100644 --- a/src/or/relay.c +++ b/src/or/relay.c @@ -2441,7 +2441,19 @@ cell_queues_check_size(void) size_t alloc = cell_queues_get_total_allocation(); alloc += buf_get_total_allocation(); alloc += tor_zlib_get_total_allocation(); + const size_t rend_cache_total = rend_cache_get_total_allocation(); + alloc += rend_cache_total; if (alloc >= get_options()->MaxMemInQueues) { + /* If we're spending over 20% of the memory limit on hidden service + * descriptors, free them until we're down to 10%. + */ + if (rend_cache_total > get_options()->MaxMemInQueues / 5) { + const size_t bytes_to_remove = + rend_cache_total - (get_options()->MaxMemInQueues / 10); + rend_cache_clean_v2_descs_as_dir(time(NULL), bytes_to_remove); + alloc -= rend_cache_total; + alloc += rend_cache_get_total_allocation(); + } circuits_handle_oom(alloc); return 1; } diff --git a/src/or/rendcommon.c b/src/or/rendcommon.c index e779ecf..d85e323 100644 --- a/src/or/rendcommon.c +++ b/src/or/rendcommon.c @@ -704,6 +704,9 @@ static strmap_t *rend_cache = NULL; * directories. */ static digestmap_t *rend_cache_v2_dir = NULL; +/** DOCDOC */ +static size_t rend_cache_total_allocation = 0; + /** Initializes the service descriptor cache. */ void @@ -713,12 +716,64 @@ rend_cache_init(void) rend_cache_v2_dir = digestmap_new(); } +/** Return the approximate number of bytes needed to hold e. */ +static size_t +rend_cache_entry_allocation(const rend_cache_entry_t *e) +{ + if (!e) + return 0; + + /* This doesn't count intro_nodes or key size */ + return sizeof(*e) + e->len + sizeof(*e->parsed); +} + +/** DOCDOC */ +size_t +rend_cache_get_total_allocation(void) +{ + return rend_cache_total_allocation; +} + +/** Decrement the total bytes attributed to the rendezvous cache by n. */ +static void +rend_cache_decrement_allocation(size_t n) +{ + static int have_underflowed = 0; + + if (rend_cache_total_allocation >= n) { + rend_cache_total_allocation -= n; + } else { + rend_cache_total_allocation = 0; + if (! have_underflowed) { + have_underflowed = 1; + log_warn(LD_BUG, "Underflow in rend_cache_decrement_allocation"); + } + } +} + +/** Increase the total bytes attributed to the rendezvous cache by n. */ +static void +rend_cache_increment_allocation(size_t n) +{ + static int have_overflowed = 0; + if (rend_cache_total_allocation <= SIZE_MAX - n) { + rend_cache_total_allocation += n; + } else { + rend_cache_total_allocation = SIZE_MAX; + if (! have_overflowed) { + have_overflowed = 1; + log_warn(LD_BUG, "Overflow in rend_cache_increment_allocation"); + } + } +} + /** Helper: free storage held by a single service descriptor cache entry. */ static void rend_cache_entry_free(rend_cache_entry_t *e) { if (!e) return; + rend_cache_decrement_allocation(rend_cache_entry_allocation(e)); rend_service_descriptor_free(e->parsed); tor_free(e->desc); tor_free(e); @@ -740,6 +795,7 @@ rend_cache_free_all(void) digestmap_free(rend_cache_v2_dir, rend_cache_entry_free_); rend_cache = NULL; rend_cache_v2_dir = NULL; + rend_cache_total_allocation = 0; } /** Removes all old entries from the service descriptor cache. @@ -777,31 +833,46 @@ rend_cache_purge(void) } /** Remove all old v2 descriptors and those for which this hidden service - * directory is not responsible for any more. */ + * directory is not responsible for any more. + * + * If at all possible, remove at least force_remove bytes of data. + */ void -rend_cache_clean_v2_descs_as_dir(time_t now) +rend_cache_clean_v2_descs_as_dir(time_t now, size_t force_remove) { digestmap_iter_t *iter; time_t cutoff = now - REND_CACHE_MAX_AGE - REND_CACHE_MAX_SKEW; - for (iter = digestmap_iter_init(rend_cache_v2_dir); - !digestmap_iter_done(iter); ) { - const char *key; - void *val; - rend_cache_entry_t *ent; - digestmap_iter_get(iter, &key, &val); - ent = val; - if (ent->parsed->timestamp < cutoff || - !hid_serv_responsible_for_desc_id(key)) { - char key_base32[REND_DESC_ID_V2_LEN_BASE32 + 1]; - base32_encode(key_base32, sizeof(key_base32), key, DIGEST_LEN); - log_info(LD_REND, "Removing descriptor with ID '%s' from cache", - safe_str_client(key_base32)); - iter = digestmap_iter_next_rmv(rend_cache_v2_dir, iter); - rend_cache_entry_free(ent); - } else { - iter = digestmap_iter_next(rend_cache_v2_dir, iter); + const int LAST_SERVED_CUTOFF_STEP = 1800; + time_t last_served_cutoff = cutoff; + size_t bytes_removed = 0; + do { + for (iter = digestmap_iter_init(rend_cache_v2_dir); + !digestmap_iter_done(iter); ) { + const char *key; + void *val; + rend_cache_entry_t *ent; + digestmap_iter_get(iter, &key, &val); + ent = val; + if (ent->parsed->timestamp < cutoff || + ent->last_served < last_served_cutoff || + !hid_serv_responsible_for_desc_id(key)) { + char key_base32[REND_DESC_ID_V2_LEN_BASE32 + 1]; + base32_encode(key_base32, sizeof(key_base32), key, DIGEST_LEN); + log_info(LD_REND, "Removing descriptor with ID '%s' from cache", + safe_str_client(key_base32)); + bytes_removed += rend_cache_entry_allocation(ent); + iter = digestmap_iter_next_rmv(rend_cache_v2_dir, iter); + rend_cache_entry_free(ent); + } else { + iter = digestmap_iter_next(rend_cache_v2_dir, iter); + } } - } + + /* In case we didn't remove enough bytes, advance the cutoff a little. */ + last_served_cutoff += LAST_SERVED_CUTOFF_STEP; + if (last_served_cutoff > now) + break; + } while (bytes_removed < force_remove); } /** Determines whether a is in the interval of b (excluded) and @@ -903,6 +974,7 @@ rend_cache_lookup_v2_desc_as_dir(const char *desc_id, const char **desc) e = digestmap_get(rend_cache_v2_dir, desc_id_digest); if (e) { *desc = e->desc; + e->last_served = approx_time(); return 1; } return 0; @@ -993,7 +1065,13 @@ rend_cache_store_v2_desc_as_dir(const char *desc) if (!e) { e = tor_malloc_zero(sizeof(rend_cache_entry_t)); digestmap_set(rend_cache_v2_dir, desc_id, e); + /* Treat something just uploaded as having been served a little + * while ago, so that flooding with new descriptors doesn't help + * too much. + */ + e->last_served = approx_time() - 3600; } else { + rend_cache_decrement_allocation(rend_cache_entry_allocation(e)); rend_service_descriptor_free(e->parsed); tor_free(e->desc); } @@ -1001,6 +1079,7 @@ rend_cache_store_v2_desc_as_dir(const char *desc) e->parsed = parsed; e->desc = tor_strndup(current_desc, encoded_size); e->len = encoded_size; + rend_cache_increment_allocation(rend_cache_entry_allocation(e)); log_info(LD_REND, "Successfully stored service descriptor with desc ID " "'%s' and len %d.", safe_str(desc_id_base32), (int)encoded_size); @@ -1189,6 +1268,7 @@ rend_cache_store_v2_desc_as_client(const char *desc, e = tor_malloc_zero(sizeof(rend_cache_entry_t)); strmap_set_lc(rend_cache, key, e); } else { + rend_cache_decrement_allocation(rend_cache_entry_allocation(e)); rend_service_descriptor_free(e->parsed); tor_free(e->desc); } @@ -1197,6 +1277,7 @@ rend_cache_store_v2_desc_as_client(const char *desc, e->desc = tor_malloc_zero(encoded_size + 1); strlcpy(e->desc, desc, encoded_size + 1); e->len = encoded_size; + rend_cache_increment_allocation(rend_cache_entry_allocation(e)); log_debug(LD_REND,"Successfully stored rend desc '%s', len %d.", safe_str_client(service_id), (int)encoded_size); return RCS_OKAY; diff --git a/src/or/rendcommon.h b/src/or/rendcommon.h index 186326a..1d02bb4 100644 --- a/src/or/rendcommon.h +++ b/src/or/rendcommon.h @@ -33,7 +33,7 @@ void rend_intro_point_free(rend_intro_point_t *intro); void rend_cache_init(void); void rend_cache_clean(time_t now); -void rend_cache_clean_v2_descs_as_dir(time_t now); +void rend_cache_clean_v2_descs_as_dir(time_t now, size_t min_to_remove); void rend_cache_purge(void); void rend_cache_free_all(void); int rend_valid_service_id(const char *query); @@ -51,7 +51,6 @@ rend_cache_store_status_t rend_cache_store_v2_desc_as_dir(const char *desc); rend_cache_store_status_t rend_cache_store_v2_desc_as_client(const char *desc, const char *desc_id_base32, const rend_data_t *rend_query); - int rend_encode_v2_descriptors(smartlist_t *descs_out, rend_service_descriptor_t *desc, time_t now, uint8_t period, rend_auth_type_t auth_type, @@ -64,6 +63,7 @@ int rend_id_is_in_interval(const char *a, const char *b, const char *c); void rend_get_descriptor_id_bytes(char *descriptor_id_out, const char *service_id, const char *secret_id_part); +size_t rend_cache_get_total_allocation(void); #endif [View Less]

1 0

[tor/master] Merge branch 'bug13806_squashed'
by nickm＠torproject.org 12 Jan '15

12 Jan '15

commit c2e200cef8b910434d8b5d251597081935392377 Merge: b0c3210 3033ba9 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Jan 12 13:59:26 2015 -0500 Merge branch 'bug13806_squashed' Conflicts: src/or/relay.c changes/bug13806 | 8 ++++ src/or/main.c | 2 +- src/or/or.h | 2 + src/or/relay.c | 18 ++++++-- src/or/rendcommon.c | 121 ++++++++++++++++++++++++++++++++++++++++++--------- src/or/rendcommon.h | 4 +- 6 files … [View More]

1 0

[tor/master] reindent cell_queues_check_size()
by nickm＠torproject.org 12 Jan '15

12 Jan '15

commit cacea9102a2431ce8e7e431629a91a43331ac9f9 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Jan 12 13:59:50 2015 -0500 reindent cell_queues_check_size() --- src/or/relay.c | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/src/or/relay.c b/src/or/relay.c index 28211ff..f878c9d 100644 --- a/src/or/relay.c +++ b/src/or/relay.c @@ -2451,20 +2451,20 @@ cell_queues_check_size(void) alloc += rend_cache_total; if (alloc … [View More]

1 0

[tor/master] If libsystemd-daemon.pc is not found, look for libsystemd
by nickm＠torproject.org 12 Jan '15

12 Jan '15

commit b0c32106b3559b4ee9fabfb1a49e2e328c850305 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Jan 12 09:49:52 2015 -0500 If libsystemd-daemon.pc is not found, look for libsystemd (Thanks to toralf for noting that the build was broken, and to Sebastian for telling me that my fix still wasn't right.) --- configure.ac | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/configure.ac b/configure.ac index b89ae8d..c254725 100644 --- a/configure.ac +++ … [View More]

1 0

[translation/bridgedb] Update translations for bridgedb
by translation＠torproject.org 12 Jan '15

12 Jan '15

commit cb7384903d0f8ba3377b557d377012e609c65e4c Author: Translation commit bot <translation(a)torproject.org> Date: Mon Jan 12 11:15:02 2015 +0000 Update translations for bridgedb --- ko/LC_MESSAGES/bridgedb.po | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/ko/LC_MESSAGES/bridgedb.po b/ko/LC_MESSAGES/bridgedb.po index 5b450ae..65df099 100644 --- a/ko/LC_MESSAGES/bridgedb.po +++ b/ko/LC_MESSAGES/bridgedb.po @@ -17,7 +17,7 @@ msgstr "" "Project-Id-… [View More]

1 0