October 2019 - tor-commits - lists.torproject.org

[tor/master] Edit 01b-collections.md a bit for md and missing content
by nickm＠torproject.org 14 Oct '19

14 Oct '19

commit 358436592befbdeb3249c5301f3e2b802de61aca Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Oct 14 15:05:47 2019 -0400 Edit 01b-collections.md a bit for md and missing content --- doc/HACKING/design/01b-collections.md | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/doc/HACKING/design/01b-collections.md b/doc/HACKING/design/01b-collections.md index def60b0f1..ed6fdc907 100644 --- a/doc/HACKING/design/01b-collections.md +++ b/doc/HACKING/design/01b-collections.md @@ -4,27 +4,27 @@ ### Smartlists: Neither lists, nor especially smart. For historical reasons, we call our dynamic-allocated array type -"smartlist_t". It can grow or shrink as elements are added and removed. +`smartlist_t`. It can grow or shrink as elements are added and removed. -All smartlists hold an array of void \*. Whenever you expose a smartlist +All smartlists hold an array of `void *`. Whenever you expose a smartlist in an API you *must* document which types its pointers actually hold.  -Smartlists are created empty with smartlist_new() and freed with -smartlist_free(). See the containers.h module documentation for more +Smartlists are created empty with `smartlist_new()` and freed with +`smartlist_free()`. See the `containers.h` module documentation for more information; there are many convenience functions for commonly needed operations. + ### Digest maps, string maps, and more. Tor makes frequent use of maps from 160-bit digests, 256-bit digests, -or nul-terminated strings to void \*. These types are digestmap_t, -digest256map_t, and strmap_t respectively. See the containers.h +or nul-terminated strings to `void *`. These types are `digestmap_t`, +`digest256map_t`, and `strmap_t` respectively. See the containers.h module documentation for more information. - ### Intrusive lists and hashtables For performance-sensitive cases, we sometimes want to use "intrusive" @@ -32,12 +32,14 @@ collections: ones where the bookkeeping pointers are stuck inside the structures that belong to the collection. If you've used the BSD-style sys/queue.h macros, you'll be familiar with these. -Unfortunately, the sys/queue.h macros vary significantly between the +Unfortunately, the `sys/queue.h` macros vary significantly between the platforms that have them, so we provide our own variants in -src/ext/tor_queue.h . +`src/ext/tor_queue.h`. -We also provide an intrusive hashtable implementation in src/ext/ht.h -. When you're using it, you'll need to define your own hash +We also provide an intrusive hashtable implementation in `src/ext/ht.h`. +When you're using it, you'll need to define your own hash functions. If attacker-induced collisions are a worry here, use the cryptographic siphash24g function to extract hashes. +

1 0

[tor/master] Rewrite "common" overview into a "lib" overview.
by nickm＠torproject.org 14 Oct '19

14 Oct '19

commit 8ef5d96c2e7c026feff3a4dd20f0096f6d8cf901 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Oct 14 13:49:27 2019 -0400 Rewrite "common" overview into a "lib" overview. --- doc/HACKING/design/01.00-lib-overview.md | 206 +++++++++++++++++++------------ 1 file changed, 128 insertions(+), 78 deletions(-) diff --git a/doc/HACKING/design/01.00-lib-overview.md b/doc/HACKING/design/01.00-lib-overview.md index 79a6a7b7d..08dec51a0 100644 --- a/doc/HACKING/design/01.00-lib-overview.md +++ b/doc/HACKING/design/01.00-lib-overview.md @@ -1,121 +1,171 @@ -## Utility code in Tor +## Library code in Tor. -Most of Tor's utility code is in modules in the src/common subdirectory. +Most of Tor's utility code is in modules in the `src/lib` subdirectory. In +general, this code is not necessarily Tor-specific, but is instead possibly +useful for other applications. -These are divided, broadly, into _compatibility_ functions, _utility_ -functions, _containers_, and _cryptography_. (Someday in the future, it -would be great to split these modules into separate directories. Also, some -functions are probably put in the wrong modules) +This code includes: -### Compatibility code + * Compatibility wrappers, to provide a uniform API across different + platforms. -These functions live in src/common/compat\*.c; some corresponding macros live -in src/common/compat\*.h. They serve as wrappers around platform-specific or -compiler-specific logic functionality. + * Library wrappers, to provide a tor-like API over different libraries + that Tor uses for things like compression and cryptography. -In general, the rest of the Tor code *should not* be calling platform-specific -or otherwise non-portable functions. Instead, they should call wrappers from -compat.c, which implement a common cross-platform API. (If you don't know -whether a function is portable, it's usually good enough to see whether it -exists on OSX, Linux, and Windows.) + * Containers, to implement some general-purpose data container types. -Other compatibility modules include backtrace.c, which generates stack traces -for crash reporting; sandbox.c, which implements the Linux seccomp2 sandbox; -and procmon.c, which handles monitoring a child process. +The modules in `src/lib` are currently well-factored: each one depends +only on lower-level modules. You can see an up-to-date list of the +modules sorted from lowest to highest level by running +`./scripts/maint/practracker/includes.py --toposort`. -Parts of address.c are compatibility code for handling network addressing -issues; other parts are in util.c. +As of this writing, the library modules are (from lowest to highest +level): -Notable compatibility areas are: + * `lib/cc` -- Macros for managing the C compiler and + language. Includes macros for improving compatibility and clarity + across different C compilers. - * mmap support for mapping files into the address space (read-only) + * `lib/version` -- Holds the current version of Tor. - * Code to work around the intricacies + * `lib/testsupport` -- Helpers for making test-only code and test + mocking support. - * Workaround code for Windows's horrible winsock incompatibilities and - Linux's intricate socket extensions. + * `lib/defs` -- Lowest-level constants used in many places across the + code. - * Helpful string functions like memmem, memstr, asprintf, strlcpy, and - strlcat that not all platforms have. + * `lib/subsys` -- Types used for declaring a "subsystem". A subsystem + is a module with support for initialization, shutdown, + configuration, and so on. - * Locale-ignoring variants of the ctypes functions. + * `lib/conf` -- Types and macros used for declaring configuration + options. - * Time-manipulation functions + * `lib/arch` -- Compatibility functions and macros for handling + differences in CPU architecture. - * File locking function + * `lib/err` -- Lowest-level error handling code: responsible for + generating stack traces, handling raw assertion failures, and + otherwise reporting problems that might not be safe to report + via the regular logging module. - * IPv6 functions for platforms that don't have enough IPv6 support + * `lib/malloc` -- Wrappers and utilities for memory management. - * Endianness functions + * `lib/intmath` -- Utilities for integer mathematics. - * OS functions + * `lib/fdio` -- Utilities and compatibility code for reading and + writing data on file descriptors (and on sockets, for platforms + where a socket is not a kind of fd). - * Threading and locking functions. + * `lib/lock` -- Compatibility code for declaring and using locks. + Lower-level than the rest of the threading code. -=== Utility functions + * `lib/ctime` -- Constant-time implementations for data comparison + and table lookup, used to avoid timing side-channels from standard + implementations of memcmp() and so on. -General-purpose utilities are in util.c; they include higher-level wrappers -around many of the compatibility functions to provide things like -file-at-once access, memory management functions, math, string manipulation, -time manipulation, filesystem manipulation, etc. + * `lib/string` -- Low-level compatibility wrappers and utility + functions for string manipulation. -(Some functionality, like daemon-launching, would be better off in a -compatibility module.) + * `lib/wallclock` -- Compatibility and utility functions for + inspecting and manipulating the current (UTC) time. -In util_format.c, we have code to implement stuff like base-32 and base-64 -encoding. + * `lib/osinfo` -- Functions for inspecting the version and + capabilities of the operating system. -The address.c module interfaces with the system resolver and implements -address parsing and formatting functions. It converts sockaddrs to and from -a more compact tor_addr_t type. + * `lib/smartlist_core` -- The bare-bones pieces of our dynamic array + ("smartlist") implementation. There are higher-level pieces, but + these ones are used by (and therefore cannot use) the logging code. -The di_ops.c module provides constant-time comparison and associative-array -operations, for side-channel avoidance. + * `lib/log` -- Implements the logging system used by all higher-level + Tor code. You can think of this as the logical "midpoint" of the + library code: much of the higher-level code is higher-level + _because_ it uses the logging module, and much of the lower-level + code is specifically written to avoid having to log, because the + logging module depends on it. -The logging subsystem in log.c supports logging to files, to controllers, to -stdout/stderr, or to the system log. + * `lib/container` -- General purpose containers, including dynamic arrays, + hashtables, bit arrays, weak-reference-like "handles", bloom + filters, and a bit more. -The abstraction in memarea.c is used in cases when a large amount of -temporary objects need to be allocated, and they can all be freed at the same -time. + * `lib/trace` -- A general-purpose API for introducing + function-tracing functionality into Tor. Currently not much used. -The torgzip.c module wraps the zlib library to implement compression. + * `lib/thread` -- Threading compatibility and utility functionality, + other than low-level locks (which are in `lib/lock`) and + workqueue/threadpool code (which belongs in `lib/evloop`). -Workqueue.c provides a simple multithreaded work-queue implementation. + * `lib/term` -- Code for terminal manipulation functions (like + reading a password from the user). -### Containers + * `lib/memarea` -- A data structure for a fast "arena" style allocator, + where the data is freed all at once. Used for parsing. -The container.c module defines these container types, used throughout the Tor -codebase. + * `lib/encoding` -- Implementations for encoding data in various + formats, datatypes, and transformations. -There is a dynamic array called **smartlist**, used as our general resizeable -array type. It supports sorting, searching, common set operations, and so -on. It has specialized functions for smartlists of strings, and for -heap-based priority queues. + * `lib/dispatch` -- A general-purpose in-process message delivery + system. Used by `lib/pubsub` to implement our inter-module + publish/subscribe system. -There's a bit-array type. + * `lib/sandbox` -- Our Linux seccomp2 sandbox implementation. -A set of mapping types to map strings, 160-bit digests, and 256-bit digests -to void \*. These are what we generally use when we want O(1) lookup. + * `lib/pubsub` -- Code and macros to implement our publish/subscribe + message passing system. -Additionally, for containers, we use the ht.h and tor_queue.h headers, in -src/ext. These provide intrusive hashtable and linked-list macros. + * `lib/fs` -- Utility and compatibility code for manipulating files, + filenames, directories, and so on. -### Cryptography + * `lib/confmgt` -- Code to parse, encode, and manipulate our + configuration files, state files, and so forth. -Once, we tried to keep our cryptography code in a single "crypto.c" file, -with an "aes.c" module containing an AES implementation for use with older -OpenSSLs. + * `lib/crypt_ops` -- Cryptographic operations. This module contains + wrappers around the cryptographic libraries that we support, + and implementations for some higher-level cryptographic + constructions that we use. -Now, our practice has become to introduce crypto_\*.c modules when adding new -cryptography backend code. We have modules for Ed25519, Curve25519, -secret-to-key algorithms, and password-based boxed encryption. + * `lib/meminfo` -- Functions for inspecting our memory usage, if the + malloc implementation exposes that to us. -Our various TLS compatibility code, wrappers, and hacks are kept in -tortls.c, which is probably too full of Tor-specific kludges. I'm -hoping we can eliminate most of those kludges when we finally remove -support for older versions of our TLS handshake. + * `lib/time` -- Higher level time functions, including fine-gained and + monotonic timers. + * `lib/math` -- Floating-point mathematical utilities, including + compatibility code, and probability distributions. + * `lib/buf` -- A general purpose queued buffer implementation, + similar to the BSD kernel's "mbuf" structure. + * `lib/net` -- Networking code, including address manipulation, + compatibility wrappers, + + * `lib/compress` -- A compatibility wrapper around several + compression libraries, currently including zlib, zstd, and lzma. + + * `lib/geoip` -- Utilities to manage geoip (IP to country) lookups + and formats. + + * `lib/tls` -- Compatibility wrappers around the library (NSS or + OpenSSL, depending on configuration) that Tor uses to implement the + TLS link security protocol. + + * `lib/evloop` -- Tools to manage the event loop and related + functionality, in order to implement asynchronous networking, + timers, periodic events, and other scheduling tasks. + + * `lib/process` -- Utilities and compatibility code to launch and + manage subprocesses. + +### What belongs in lib? + +In general, if you can imagine some program wanting the functionality +you're writing, even if that program had nothing to do with Tor, your +functionality belongs in lib. + +If it falls into one of the existing "lib" categories, your +functionality belongs in lib. + +If you are using platform-specific `#ifdef`s to manage compatibility +issues among platforms, you should probably consider whether you can +put your code into lib.

1 0

[tor/master] Edit 01a-memory.md
by nickm＠torproject.org 14 Oct '19

14 Oct '19

commit 77accf937e2272b6e46a1486301d87b4df8f9bab Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Oct 14 15:02:26 2019 -0400 Edit 01a-memory.md --- doc/HACKING/design/01a-memory.md | 62 +++++++++++++++++++++++----------------- 1 file changed, 36 insertions(+), 26 deletions(-) diff --git a/doc/HACKING/design/01a-memory.md b/doc/HACKING/design/01a-memory.md index 9a2078296..4c6bb0901 100644 --- a/doc/HACKING/design/01a-memory.md +++ b/doc/HACKING/design/01a-memory.md @@ -1,7 +1,7 @@ ## Memory management -### Heap-allocation functions +### Heap-allocation functions: lib/malloc/malloc.h Tor imposes a few light wrappers over C's native malloc and free functions, to improve convenience, and to allow wholescale replacement @@ -12,63 +12,71 @@ own; always use the variants prefixed with 'tor_'. They are the same as the standard C functions, with the following exceptions: - * tor_free(NULL) is a no-op. - * tor_free() is a macro that takes an lvalue as an argument and sets it to - NULL after freeing it. To avoid this behavior, you can use tor_free_() + * `tor_free(NULL)` is a no-op. + * `tor_free()` is a macro that takes an lvalue as an argument and sets it to + NULL after freeing it. To avoid this behavior, you can use `tor_free_()` instead. * tor_malloc() and friends fail with an assertion if they are asked to allocate a value so large that it is probably an underflow. - * It is always safe to tor_malloc(0), regardless of whether your libc + * It is always safe to `tor_malloc(0)`, regardless of whether your libc allows it. - * tor_malloc(), tor_realloc(), and friends are never allowed to fail. + * `tor_malloc()`, `tor_realloc()`, and friends are never allowed to fail. Instead, Tor will die with an assertion. This means that you never need to check their return values. See the next subsection for information on why we think this is a good idea. We define additional general-purpose memory allocation functions as well: - * tor_malloc_zero(x) behaves as calloc(1, x), except the it makes clear + * `tor_malloc_zero(x)` behaves as `calloc(1, x)`, except the it makes clear the intent to allocate a single zeroed-out value. - * tor_reallocarray(x,y) behaves as the OpenBSD reallocarray function. + * `tor_reallocarray(x,y)` behaves as the OpenBSD reallocarray function. Use it for cases when you need to realloc() in a multiplication-safe way. And specific-purpose functions as well: - * tor_strdup() and tor_strndup() behaves as the underlying libc functions, - but use tor_malloc() instead of the underlying function. - * tor_memdup() copies a chunk of memory of a given size. - * tor_memdup_nulterm() copies a chunk of memory of a given size, then + * `tor_strdup()` and `tor_strndup()` behaves as the underlying libc + functions, but use `tor_malloc()` instead of the underlying function. + * `tor_memdup()` copies a chunk of memory of a given size. + * `tor_memdup_nulterm()` copies a chunk of memory of a given size, then NUL-terminates it just to be safe. -#### Why assert on failure? +#### Why assert on allocation failure? -Why don't we allow tor_malloc() and its allies to return NULL? +Why don't we allow `tor_malloc()` and its allies to return NULL? First, it's error-prone. Many programmers forget to check for NULL return -values, and testing for malloc() failures is a major pain. +values, and testing for `malloc()` failures is a major pain. Second, it's not necessarily a great way to handle OOM conditions. It's probably better (we think) to have a memory target where we dynamically free things ahead of time in order to stay under the target. Trying to respond to -an OOM at the point of tor_malloc() failure, on the other hand, would involve +an OOM at the point of `tor_malloc()` failure, on the other hand, would involve a rare operation invoked from deep in the call stack. (Again, that's error-prone and hard to debug.) Third, thanks to the rise of Linux and other operating systems that allow memory to be overcommitted, you can't actually ever rely on getting a NULL -from malloc() when you're out of memory; instead you have to use an approach +from `malloc()` when you're out of memory; instead you have to use an approach closer to tracking the total memory usage. #### Conventions for your own allocation functions. Whenever you create a new type, the convention is to give it a pair of -x_new() and x_free() functions, named after the type. +`x_new()` and `x_free_()` functions, named after the type. -Calling x_free(NULL) should always be a no-op. +Calling `x_free(NULL)` should always be a no-op. +There should additionally be an `x_free()` macro, defined in terms of +`x_free_()`. This macro should set its lvalue to NULL. You can define it +using the FREE_AND_NULL macro, as follows: -### Grow-only memory allocation: memarea.c +``` +#define x_free(ptr) FREE_AND_NULL(x_t, x_free_, (ptr)) +``` + + +### Grow-only memory allocation: lib/memarea It's often handy to allocate a large number of tiny objects, all of which need to disappear at the same time. You can do this in tor using the @@ -82,12 +90,14 @@ objects of similar size. But if you use tor_malloc() for the long-lived ones and a memarea for the temporary object, the malloc implementation is likelier to do better. -To create a new memarea, use memarea_new(). To drop all the storage from a -memarea, and invalidate its pointers, use memarea_drop_all(). +To create a new memarea, use `memarea_new()`. To drop all the storage from a +memarea, and invalidate its pointers, use `memarea_drop_all()`. -The allocation functions memarea_alloc(), memarea_alloc_zero(), -memarea_memdup(), memarea_strdup(), and memarea_strndup() are analogous to -the similarly-named malloc() functions. There is intentionally no -memarea_free() or memarea_realloc(). +The allocation functions `memarea_alloc()`, `memarea_alloc_zero()`, +`memarea_memdup()`, `memarea_strdup()`, and `memarea_strndup()` are analogous +to the similarly-named malloc() functions. There is intentionally no +`memarea_free()` or `memarea_realloc()`. +### Special allocation: lib/malloc/map_anon.h +TODO: WRITEME.

1 0

[tor/master] Merge branch 'design_revision'
by nickm＠torproject.org 14 Oct '19

14 Oct '19

commit dfe7f004df46edaab0b23e260218d3c6d422d5fe Merge: 6bbeb08f9 1332d3b6f Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Oct 14 15:56:49 2019 -0400 Merge branch 'design_revision' doc/HACKING/design/00-overview.md | 87 ++++++++++------ doc/HACKING/design/01-common-utils.md | 121 ---------------------- doc/HACKING/design/01.00-lib-overview.md | 171 +++++++++++++++++++++++++++++++ doc/HACKING/design/01a-memory.md | 62 ++++++----- doc/HACKING/design/01b-collections.md | 24 +++-- 5 files changed, 273 insertions(+), 192 deletions(-)

1 0

[tor/master] doc/HACKING/design: address comments from ahf
by nickm＠torproject.org 14 Oct '19

14 Oct '19

commit 1332d3b6fa4faa2ef961d1057044fb4cd2608257 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Oct 14 15:53:04 2019 -0400 doc/HACKING/design: address comments from ahf --- doc/HACKING/design/00-overview.md | 34 ++++++++++++++++---------------- doc/HACKING/design/01.00-lib-overview.md | 6 +++--- 2 files changed, 20 insertions(+), 20 deletions(-) diff --git a/doc/HACKING/design/00-overview.md b/doc/HACKING/design/00-overview.md index fde476dfa..ff40a566b 100644 --- a/doc/HACKING/design/00-overview.md +++ b/doc/HACKING/design/00-overview.md @@ -56,38 +56,38 @@ will be scheduled. The codebase is divided into a few top-level subdirectories, each of which contains several sub-modules. - src/ext -- Code maintained elsewhere that we include in the Tor - source distribution. + * `src/ext` -- Code maintained elsewhere that we include in the Tor + source distribution. - src/lib -- Lower-level utility code, not necessarily tor-specific. + * src/lib` -- Lower-level utility code, not necessarily tor-specific. - src/trunnel -- Automatically generated code (from the Trunnel) - tool: used to parse and encode binary formats. + * `src/trunnel` -- Automatically generated code (from the Trunnel + tool): used to parse and encode binary formats. - src/core -- Networking code that is implements the central parts of + * `src/core` -- Networking code that is implements the central parts of the Tor protocol and main loop. - src/feature -- Aspects of Tor (like directory management, running a + * `src/feature` -- Aspects of Tor (like directory management, running a relay, running a directory authorities, managing a list of nodes, running and using onion services) that are built on top of the mainloop code. - src/app -- Highest-level functionality; responsible for setting up - and configuring the Tor project, making sure all the lower-level + * `src/app` -- Highest-level functionality; responsible for setting up + and configuring the Tor daemon, making sure all the lower-level modules start up when required, and so on. - src/tools -- Binaries other than Tor that we produce. Currently this + * `src/tools` -- Binaries other than Tor that we produce. Currently this is tor-resolve, tor-gencert, and the tor_runner.o helper module. - src/test -- unit tests, regression tests, and a few integration + * `src/test` -- unit tests, regression tests, and a few integration tests. -In theory, the above parts of the codebase are sorted from highest-level -to lowest-level, where high-level code is only allowed to invoke -lower-level code, and lower-level code never includes or depends on code -of a higher level. In practice, this refactoring is incomplete: The -modules in src/lib are well-factored, but there are many "upward -dependencies" in src/core and src/feature. We aim to eliminate those +In theory, the above parts of the codebase are sorted from highest-level to +lowest-level, where high-level code is only allowed to invoke lower-level +code, and lower-level code never includes or depends on code of a higher +level. In practice, this refactoring is incomplete: The modules in `src/lib` +are well-factored, but there are many layer violations ("upward +dependencies") in `src/core` and `src/feature`. We aim to eliminate those over time. ### Some key high-level abstractions ### diff --git a/doc/HACKING/design/01.00-lib-overview.md b/doc/HACKING/design/01.00-lib-overview.md index 08dec51a0..58a92f406 100644 --- a/doc/HACKING/design/01.00-lib-overview.md +++ b/doc/HACKING/design/01.00-lib-overview.md @@ -85,9 +85,9 @@ level): code is specifically written to avoid having to log, because the logging module depends on it. - * `lib/container` -- General purpose containers, including dynamic arrays, - hashtables, bit arrays, weak-reference-like "handles", bloom - filters, and a bit more. + * `lib/container` -- General purpose containers, including dynamic arrays + ("smartlists"), hashtables, bit arrays, weak-reference-like "handles", + bloom filters, and a bit more. * `lib/trace` -- A general-purpose API for introducing function-tracing functionality into Tor. Currently not much used.

1 0

[tor/master] Unify backend implementations for blocking hostname lookup
by nickm＠torproject.org 14 Oct '19

14 Oct '19

commit ac8f6d51f4200b27adc0da9fc1eeb44b89c8ebab Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Oct 2 21:22:53 2019 -0400 Unify backend implementations for blocking hostname lookup We have a getaddrinfo() implementation that we prefer, and a gethostbyname*() implementation that we fall back on. Give them both the same interface, and let them be called by the same name. This is a preparatory step for making them both mockable. --- src/lib/net/resolve.c | 23 ++++++++++------------- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/src/lib/net/resolve.c b/src/lib/net/resolve.c index 78e72fba4..c8fe338e5 100644 --- a/src/lib/net/resolve.c +++ b/src/lib/net/resolve.c @@ -71,9 +71,9 @@ tor_lookup_hostname,(const char *name, uint32_t *addr)) * See tor_addr_lookup() for details. */ static int -tor_addr_lookup_host_getaddrinfo(const char *name, - uint16_t family, - tor_addr_t *addr) +tor_addr_lookup_host_impl(const char *name, + uint16_t family, + tor_addr_t *addr) { int err; struct addrinfo *res=NULL, *res_p; @@ -120,15 +120,17 @@ tor_addr_lookup_host_getaddrinfo(const char *name, #else /* !defined(HAVE_GETADDRINFO) */ -/* Host lookup helper for tor_addr_lookup(), which calls getaddrinfo(). - * Used when gethostbyname() is not available on this system. +/* Host lookup helper for tor_addr_lookup(), which calls gethostbyname(). + * Used when getaddrinfo() is not available on this system. * * See tor_addr_lookup() for details. */ static int -tor_addr_lookup_host_gethostbyname(const char *name, - tor_addr_t *addr) +tor_addr_lookup_host_impl(const char *name, + uint16_t family, + tor_addr_t *addr) { + (void) family; struct hostent *ent; int err; #ifdef HAVE_GETHOSTBYNAME_R_6_ARG @@ -215,13 +217,8 @@ tor_addr_lookup,(const char *name, uint16_t family, tor_addr_t *addr)) } else { /* Clear the address after a failed tor_addr_parse(). */ memset(addr, 0, sizeof(tor_addr_t)); -#ifdef HAVE_GETADDRINFO - result = tor_addr_lookup_host_getaddrinfo(name, family, addr); - goto done; -#else /* !(defined(HAVE_GETADDRINFO)) */ - result = tor_addr_lookup_host_gethostbyname(name, addr); + result = tor_addr_lookup_host_impl(name, family, addr); goto done; -#endif /* defined(HAVE_GETADDRINFO) */ } /* If we weren't successful, and haven't already set the result,

1 0

[tor/master] Use {mock, unmock}_hostname_resolver() in relevant tests
by nickm＠torproject.org 14 Oct '19

14 Oct '19

commit fdfb4b196b7c38f45f2d37f73fcc96d746816cc3 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Oct 3 08:57:14 2019 -0400 Use {mock,unmock}_hostname_resolver() in relevant tests These tests all invoke the hostname resolver in one way or another, and therefore potentially block if our DNS server is missing, absent, or extremely slow. Closes ticket 31841. --- changes/ticket31841 | 5 +++++ src/test/test_addr.c | 6 +++++- src/test/test_config.c | 4 ++++ src/test/test_hs_config.c | 8 +++++--- src/test/test_options.c | 3 +++ 5 files changed, 22 insertions(+), 4 deletions(-) diff --git a/changes/ticket31841 b/changes/ticket31841 new file mode 100644 index 000000000..6e7fbc1da --- /dev/null +++ b/changes/ticket31841 @@ -0,0 +1,5 @@ + o Minor features (testing): + - When running tests that attempt to look up hostname, replace the libc + name lookup functions with ones that do not actually touch the network. + This way, the tests complete more quickly in the presence of a slow or + missing DNS resolver. Closes ticket 31841. diff --git a/src/test/test_addr.c b/src/test/test_addr.c index f99e3be8f..c89c6e78d 100644 --- a/src/test/test_addr.c +++ b/src/test/test_addr.c @@ -12,6 +12,7 @@ #include "test/log_test_helpers.h" #include "lib/net/resolve.h" #include "test/rng_test_helpers.h" +#include "test/resolve_test_helpers.h" #ifdef HAVE_SYS_UN_H #include <sys/un.h> @@ -1160,6 +1161,7 @@ test_addr_parse_canonical(void *arg) static void test_addr_parse(void *arg) { + int r; tor_addr_t addr; uint16_t port; @@ -1169,6 +1171,8 @@ test_addr_parse(void *arg) (void)arg; + mock_hostname_resolver(); + /* IPv6-mapped IPv4 addresses. Tor doesn't really use these. */ TEST_ADDR_V6_PARSE("11:22:33:44:55:66:1.2.3.4", 0, "11:22:33:44:55:66:102:304"); @@ -1273,7 +1277,7 @@ test_addr_parse(void *arg) "11:22::88",99); done: - ; + unmock_hostname_resolver(); } static void diff --git a/src/test/test_config.c b/src/test/test_config.c index 1c6c91307..cbb84e4dc 100644 --- a/src/test/test_config.c +++ b/src/test/test_config.c @@ -45,6 +45,7 @@ #include "app/config/statefile.h" #include "test/test_helpers.h" +#include "test/resolve_test_helpers.h" #include "feature/dirclient/dir_server_st.h" #include "core/or/port_cfg_st.h" @@ -4068,6 +4069,8 @@ test_config_parse_port_config__ports__ports_given(void *data) slout = smartlist_new(); + mock_hostname_resolver(); + // Test error when encounters an invalid Port specification config_port_invalid = mock_config_line("DNSPort", ""); ret = parse_port_config(NULL, config_port_invalid, "DNS", 0, NULL, @@ -4764,6 +4767,7 @@ test_config_parse_port_config__ports__ports_given(void *data) #endif /* defined(_WIN32) */ done: + unmock_hostname_resolver(); if (slout) SMARTLIST_FOREACH(slout,port_cfg_t *,pf,port_cfg_free(pf)); smartlist_free(slout); diff --git a/src/test/test_hs_config.c b/src/test/test_hs_config.c index 2b3afbb6e..71e152921 100644 --- a/src/test/test_hs_config.c +++ b/src/test/test_hs_config.c @@ -12,6 +12,7 @@ #include "test/test.h" #include "test/test_helpers.h" #include "test/log_test_helpers.h" +#include "test/resolve_test_helpers.h" #include "app/config/config.h" #include "feature/hs/hs_common.h" @@ -272,6 +273,7 @@ test_valid_service_v2(void *arg) int ret; (void) arg; + mock_hostname_resolver(); /* Valid complex configuration. Basic client authorization. */ { @@ -314,7 +316,7 @@ test_valid_service_v2(void *arg) } done: - ; + unmock_hostname_resolver(); } static void @@ -392,6 +394,7 @@ test_valid_service_v3(void *arg) int ret; (void) arg; + mock_hostname_resolver(); /* Valid complex configuration. */ { @@ -448,7 +451,7 @@ test_valid_service_v3(void *arg) } done: - ; + unmock_hostname_resolver(); } static void @@ -623,4 +626,3 @@ struct testcase_t hs_config_tests[] = { END_OF_TESTCASES }; - diff --git a/src/test/test_options.c b/src/test/test_options.c index 2d45ecd18..d8757491f 100644 --- a/src/test/test_options.c +++ b/src/test/test_options.c @@ -14,6 +14,7 @@ #include "feature/nodelist/routerset.h" #include "core/mainloop/mainloop.h" #include "test/log_test_helpers.h" +#include "test/resolve_test_helpers.h" #include "lib/sandbox/sandbox.h" #include "lib/memarea/memarea.h" @@ -241,6 +242,7 @@ test_options_validate(void *arg) (void)arg; setup_log_callback(); sandbox_disable_getaddrinfo_cache(); + mock_hostname_resolver(); WANT_ERR("ExtORPort 500000", "Invalid ExtORPort", PH_VALIDATE); @@ -282,6 +284,7 @@ test_options_validate(void *arg) close_temp_logs(); clear_log_messages(); + unmock_hostname_resolver(); return; }

1 0

[tor/master] Make tor_addr_lookup_host_impl mockable.
by nickm＠torproject.org 14 Oct '19

14 Oct '19

commit 9c24ceeb3f2082193bf76a77ad6d6d011561e239 Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Oct 3 07:50:18 2019 -0400 Make tor_addr_lookup_host_impl mockable. --- src/lib/net/resolve.c | 14 +++++++------- src/lib/net/resolve.h | 15 +++++++++++++++ 2 files changed, 22 insertions(+), 7 deletions(-) diff --git a/src/lib/net/resolve.c b/src/lib/net/resolve.c index c8fe338e5..442bc4a6b 100644 --- a/src/lib/net/resolve.c +++ b/src/lib/net/resolve.c @@ -8,6 +8,7 @@ * \brief Use the libc DNS resolver to convert hostnames into addresses. **/ +#define RESOLVE_PRIVATE #include "lib/net/resolve.h" #include "lib/net/address.h" @@ -70,10 +71,10 @@ tor_lookup_hostname,(const char *name, uint32_t *addr)) * * See tor_addr_lookup() for details. */ -static int -tor_addr_lookup_host_impl(const char *name, +MOCK_IMPL(STATIC int, +tor_addr_lookup_host_impl,(const char *name, uint16_t family, - tor_addr_t *addr) + tor_addr_t *addr)) { int err; struct addrinfo *res=NULL, *res_p; @@ -125,10 +126,10 @@ tor_addr_lookup_host_impl(const char *name, * * See tor_addr_lookup() for details. */ -static int -tor_addr_lookup_host_impl(const char *name, +MOCK_IMPL(STATIC int, +tor_addr_lookup_host_impl,(const char *name, uint16_t family, - tor_addr_t *addr) + tor_addr_t *addr)) { (void) family; struct hostent *ent; @@ -172,7 +173,6 @@ tor_addr_lookup_host_impl(const char *name, return (err == TRY_AGAIN) ? 1 : -1; #endif } - #endif /* defined(HAVE_GETADDRINFO) */ /** Similar behavior to Unix gethostbyname: resolve <b>name</b>, and set diff --git a/src/lib/net/resolve.h b/src/lib/net/resolve.h index d7b60be91..b979b2fb4 100644 --- a/src/lib/net/resolve.h +++ b/src/lib/net/resolve.h @@ -24,12 +24,18 @@ struct tor_addr_t; +/* + * Primary lookup functions. + */ MOCK_DECL(int, tor_lookup_hostname,(const char *name, uint32_t *addr)); MOCK_DECL(int, tor_addr_lookup,(const char *name, uint16_t family, struct tor_addr_t *addr_out)); int tor_addr_port_lookup(const char *s, struct tor_addr_t *addr_out, uint16_t *port_out); +/* + * Sandbox helpers + */ struct addrinfo; #ifdef USE_SANDBOX_GETADDRINFO /** Pre-calls getaddrinfo in order to pre-record result. */ @@ -55,4 +61,13 @@ void tor_free_getaddrinfo_cache(void); void sandbox_disable_getaddrinfo_cache(void); void tor_make_getaddrinfo_cache_active(void); +/* + * Internal resolver wrapper; exposed for mocking. + */ +#ifdef RESOLVE_PRIVATE +MOCK_DECL(STATIC int, tor_addr_lookup_host_impl, (const char *name, + uint16_t family, + struct tor_addr_t *addr)); +#endif + #endif /* !defined(TOR_RESOLVE_H) */

1 0

[tor/master] Add a mock replacement for blocking hostname resolution
by nickm＠torproject.org 14 Oct '19

14 Oct '19

commit 4a0749596cc21198dd853bb0c631e9658db7fe2d Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Oct 3 08:54:17 2019 -0400 Add a mock replacement for blocking hostname resolution Based on examination of our tests, this mock function accepts "localhost" and "torproject.org", and rejects everything else. --- src/test/include.am | 2 + src/test/resolve_test_helpers.c | 85 +++++++++++++++++++++++++++++++++++++++++ src/test/resolve_test_helpers.h | 18 +++++++++ 3 files changed, 105 insertions(+) diff --git a/src/test/include.am b/src/test/include.am index 2dd4d8c58..acd0824d0 100644 --- a/src/test/include.am +++ b/src/test/include.am @@ -102,6 +102,7 @@ src_test_test_SOURCES += \ src/test/log_test_helpers.c \ src/test/hs_test_helpers.c \ src/test/rend_test_helpers.c \ + src/test/resolve_test_helpers.c \ src/test/rng_test_helpers.c \ src/test/test.c \ src/test/test_accounting.c \ @@ -340,6 +341,7 @@ noinst_HEADERS+= \ src/test/hs_test_helpers.h \ src/test/log_test_helpers.h \ src/test/rend_test_helpers.h \ + src/test/resolve_test_helpers.h \ src/test/rng_test_helpers.h \ src/test/test.h \ src/test/ptr_helpers.h \ diff --git a/src/test/resolve_test_helpers.c b/src/test/resolve_test_helpers.c new file mode 100644 index 000000000..73ea73014 --- /dev/null +++ b/src/test/resolve_test_helpers.c @@ -0,0 +1,85 @@ +/* Copyright (c) 2001 Matej Pfajfar. + * Copyright (c) 2001-2004, Roger Dingledine. + * Copyright (c) 2004-2006, Roger Dingledine, Nick Mathewson. + * Copyright (c) 2007-2019, The Tor Project, Inc. */ +/* See LICENSE for licensing information */ + +/** + * @file resolve_test_helpers.c + * @brief Helper functions for mocking libc's blocking hostname lookup + * facilities. + **/ + +#define RESOLVE_PRIVATE +#include "orconfig.h" +#include "test/resolve_test_helpers.h" +#include "lib/net/address.h" +#include "lib/net/resolve.h" +#include "test/test.h" + +#include <stdio.h> +#include <string.h> + +/** + * Mock replacement for our getaddrinfo/gethostbyname wrapper. + **/ +static int +replacement_host_lookup(const char *name, uint16_t family, tor_addr_t *addr) +{ + static const struct lookup_table_ent { + const char *name; + const char *ipv4; + const char *ipv6; + } entries[] = { + { "localhost", "127.0.0.1", "::1" }, + { "torproject.org", "198.51.100.6", "2001:DB8::700" }, + { NULL, NULL, NULL }, + }; + + int r = -1; + + for (unsigned i = 0; entries[i].name != NULL; ++i) { + if (!strcasecmp(name, entries[i].name)) { + if (family == AF_INET6) { + int s = tor_addr_parse(addr, entries[i].ipv6); + tt_int_op(s, OP_EQ, AF_INET6); + } else { + int s = tor_addr_parse(addr, entries[i].ipv4); + tt_int_op(s, OP_EQ, AF_INET); + } + r = 0; + break; + } + } + + log_debug(LD_GENERAL, "resolve(%s,%d) => %s", + name, family, r == 0 ? fmt_addr(addr) : "-1"); + + return r; + done: + return -1; +} + +/** + * Set up a mock replacement for our wrapper on libc's resolver code. + * + * According to our replacement, only "localhost" and "torproject.org" + * are real addresses; everything else doesn't exist. + * + * Use this function to avoid using the DNS resolver during unit tests; + * call unmock_hostname_resolver() when you're done. + **/ +void +mock_hostname_resolver(void) +{ + MOCK(tor_addr_lookup_host_impl, replacement_host_lookup); +} + +/** + * Unmock our wrappers for libc's blocking hostname resolver code. + **/ +void +unmock_hostname_resolver(void) +{ + UNMOCK(tor_addr_lookup_host_impl); +} diff --git a/src/test/resolve_test_helpers.h b/src/test/resolve_test_helpers.h new file mode 100644 index 000000000..e7d2e2937 --- /dev/null +++ b/src/test/resolve_test_helpers.h @@ -0,0 +1,18 @@ +/* Copyright (c) 2001 Matej Pfajfar. + * Copyright (c) 2001-2004, Roger Dingledine. + * Copyright (c) 2004-2006, Roger Dingledine, Nick Mathewson. + * Copyright (c) 2007-2019, The Tor Project, Inc. */ +/* See LICENSE for licensing information */ + +/** + * @file resolve_test_helpers.h + * @brief Header for test/resolve_test_helpers.c + **/ + +#ifndef TOR_TEST_RESOLVE_TEST_HELPERS_H +#define TOR_TEST_RESOLVE_TEST_HELPERS_H + +void mock_hostname_resolver(void); +void unmock_hostname_resolver(void); + +#endif /* !defined(TOR_TEST_RESOLVE_TEST_HELPERS_H) */

1 0

[tor/master] Merge remote-tracking branch 'tor-github/pr/1393' into maint-0.4.2
by nickm＠torproject.org 14 Oct '19

14 Oct '19

commit eebd2d44a1f45366f372a0ece2a851912607e0a2 Merge: a4790e7d4 fdfb4b196 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Oct 14 15:55:13 2019 -0400 Merge remote-tracking branch 'tor-github/pr/1393' into maint-0.4.2 changes/ticket31841 | 5 +++ src/lib/net/resolve.c | 29 +++++++------- src/lib/net/resolve.h | 15 ++++++++ src/test/include.am | 2 + src/test/resolve_test_helpers.c | 85 +++++++++++++++++++++++++++++++++++++++++ src/test/resolve_test_helpers.h | 18 +++++++++ src/test/test_addr.c | 6 ++- src/test/test_config.c | 4 ++ src/test/test_hs_config.c | 8 ++-- src/test/test_options.c | 3 ++ 10 files changed, 155 insertions(+), 20 deletions(-)

1 0