commit 8ef5d96c2e7c026feff3a4dd20f0096f6d8cf901 Author: Nick Mathewson nickm@torproject.org Date: Mon Oct 14 13:49:27 2019 -0400
Rewrite "common" overview into a "lib" overview. --- doc/HACKING/design/01.00-lib-overview.md | 206 +++++++++++++++++++------------ 1 file changed, 128 insertions(+), 78 deletions(-)
diff --git a/doc/HACKING/design/01.00-lib-overview.md b/doc/HACKING/design/01.00-lib-overview.md index 79a6a7b7d..08dec51a0 100644 --- a/doc/HACKING/design/01.00-lib-overview.md +++ b/doc/HACKING/design/01.00-lib-overview.md @@ -1,121 +1,171 @@
-## Utility code in Tor +## Library code in Tor.
-Most of Tor's utility code is in modules in the src/common subdirectory. +Most of Tor's utility code is in modules in the `src/lib` subdirectory. In +general, this code is not necessarily Tor-specific, but is instead possibly +useful for other applications.
-These are divided, broadly, into _compatibility_ functions, _utility_ -functions, _containers_, and _cryptography_. (Someday in the future, it -would be great to split these modules into separate directories. Also, some -functions are probably put in the wrong modules) +This code includes:
-### Compatibility code + * Compatibility wrappers, to provide a uniform API across different + platforms.
-These functions live in src/common/compat*.c; some corresponding macros live -in src/common/compat*.h. They serve as wrappers around platform-specific or -compiler-specific logic functionality. + * Library wrappers, to provide a tor-like API over different libraries + that Tor uses for things like compression and cryptography.
-In general, the rest of the Tor code *should not* be calling platform-specific -or otherwise non-portable functions. Instead, they should call wrappers from -compat.c, which implement a common cross-platform API. (If you don't know -whether a function is portable, it's usually good enough to see whether it -exists on OSX, Linux, and Windows.) + * Containers, to implement some general-purpose data container types.
-Other compatibility modules include backtrace.c, which generates stack traces -for crash reporting; sandbox.c, which implements the Linux seccomp2 sandbox; -and procmon.c, which handles monitoring a child process. +The modules in `src/lib` are currently well-factored: each one depends +only on lower-level modules. You can see an up-to-date list of the +modules sorted from lowest to highest level by running +`./scripts/maint/practracker/includes.py --toposort`.
-Parts of address.c are compatibility code for handling network addressing -issues; other parts are in util.c. +As of this writing, the library modules are (from lowest to highest +level):
-Notable compatibility areas are: + * `lib/cc` -- Macros for managing the C compiler and + language. Includes macros for improving compatibility and clarity + across different C compilers.
- * mmap support for mapping files into the address space (read-only) + * `lib/version` -- Holds the current version of Tor.
- * Code to work around the intricacies + * `lib/testsupport` -- Helpers for making test-only code and test + mocking support.
- * Workaround code for Windows's horrible winsock incompatibilities and - Linux's intricate socket extensions. + * `lib/defs` -- Lowest-level constants used in many places across the + code.
- * Helpful string functions like memmem, memstr, asprintf, strlcpy, and - strlcat that not all platforms have. + * `lib/subsys` -- Types used for declaring a "subsystem". A subsystem + is a module with support for initialization, shutdown, + configuration, and so on.
- * Locale-ignoring variants of the ctypes functions. + * `lib/conf` -- Types and macros used for declaring configuration + options.
- * Time-manipulation functions + * `lib/arch` -- Compatibility functions and macros for handling + differences in CPU architecture.
- * File locking function + * `lib/err` -- Lowest-level error handling code: responsible for + generating stack traces, handling raw assertion failures, and + otherwise reporting problems that might not be safe to report + via the regular logging module.
- * IPv6 functions for platforms that don't have enough IPv6 support + * `lib/malloc` -- Wrappers and utilities for memory management.
- * Endianness functions + * `lib/intmath` -- Utilities for integer mathematics.
- * OS functions + * `lib/fdio` -- Utilities and compatibility code for reading and + writing data on file descriptors (and on sockets, for platforms + where a socket is not a kind of fd).
- * Threading and locking functions. + * `lib/lock` -- Compatibility code for declaring and using locks. + Lower-level than the rest of the threading code.
-=== Utility functions + * `lib/ctime` -- Constant-time implementations for data comparison + and table lookup, used to avoid timing side-channels from standard + implementations of memcmp() and so on.
-General-purpose utilities are in util.c; they include higher-level wrappers -around many of the compatibility functions to provide things like -file-at-once access, memory management functions, math, string manipulation, -time manipulation, filesystem manipulation, etc. + * `lib/string` -- Low-level compatibility wrappers and utility + functions for string manipulation.
-(Some functionality, like daemon-launching, would be better off in a -compatibility module.) + * `lib/wallclock` -- Compatibility and utility functions for + inspecting and manipulating the current (UTC) time.
-In util_format.c, we have code to implement stuff like base-32 and base-64 -encoding. + * `lib/osinfo` -- Functions for inspecting the version and + capabilities of the operating system.
-The address.c module interfaces with the system resolver and implements -address parsing and formatting functions. It converts sockaddrs to and from -a more compact tor_addr_t type. + * `lib/smartlist_core` -- The bare-bones pieces of our dynamic array + ("smartlist") implementation. There are higher-level pieces, but + these ones are used by (and therefore cannot use) the logging code.
-The di_ops.c module provides constant-time comparison and associative-array -operations, for side-channel avoidance. + * `lib/log` -- Implements the logging system used by all higher-level + Tor code. You can think of this as the logical "midpoint" of the + library code: much of the higher-level code is higher-level + _because_ it uses the logging module, and much of the lower-level + code is specifically written to avoid having to log, because the + logging module depends on it.
-The logging subsystem in log.c supports logging to files, to controllers, to -stdout/stderr, or to the system log. + * `lib/container` -- General purpose containers, including dynamic arrays, + hashtables, bit arrays, weak-reference-like "handles", bloom + filters, and a bit more.
-The abstraction in memarea.c is used in cases when a large amount of -temporary objects need to be allocated, and they can all be freed at the same -time. + * `lib/trace` -- A general-purpose API for introducing + function-tracing functionality into Tor. Currently not much used.
-The torgzip.c module wraps the zlib library to implement compression. + * `lib/thread` -- Threading compatibility and utility functionality, + other than low-level locks (which are in `lib/lock`) and + workqueue/threadpool code (which belongs in `lib/evloop`).
-Workqueue.c provides a simple multithreaded work-queue implementation. + * `lib/term` -- Code for terminal manipulation functions (like + reading a password from the user).
-### Containers + * `lib/memarea` -- A data structure for a fast "arena" style allocator, + where the data is freed all at once. Used for parsing.
-The container.c module defines these container types, used throughout the Tor -codebase. + * `lib/encoding` -- Implementations for encoding data in various + formats, datatypes, and transformations.
-There is a dynamic array called **smartlist**, used as our general resizeable -array type. It supports sorting, searching, common set operations, and so -on. It has specialized functions for smartlists of strings, and for -heap-based priority queues. + * `lib/dispatch` -- A general-purpose in-process message delivery + system. Used by `lib/pubsub` to implement our inter-module + publish/subscribe system.
-There's a bit-array type. + * `lib/sandbox` -- Our Linux seccomp2 sandbox implementation.
-A set of mapping types to map strings, 160-bit digests, and 256-bit digests -to void *. These are what we generally use when we want O(1) lookup. + * `lib/pubsub` -- Code and macros to implement our publish/subscribe + message passing system.
-Additionally, for containers, we use the ht.h and tor_queue.h headers, in -src/ext. These provide intrusive hashtable and linked-list macros. + * `lib/fs` -- Utility and compatibility code for manipulating files, + filenames, directories, and so on.
-### Cryptography + * `lib/confmgt` -- Code to parse, encode, and manipulate our + configuration files, state files, and so forth.
-Once, we tried to keep our cryptography code in a single "crypto.c" file, -with an "aes.c" module containing an AES implementation for use with older -OpenSSLs. + * `lib/crypt_ops` -- Cryptographic operations. This module contains + wrappers around the cryptographic libraries that we support, + and implementations for some higher-level cryptographic + constructions that we use.
-Now, our practice has become to introduce crypto_*.c modules when adding new -cryptography backend code. We have modules for Ed25519, Curve25519, -secret-to-key algorithms, and password-based boxed encryption. + * `lib/meminfo` -- Functions for inspecting our memory usage, if the + malloc implementation exposes that to us.
-Our various TLS compatibility code, wrappers, and hacks are kept in -tortls.c, which is probably too full of Tor-specific kludges. I'm -hoping we can eliminate most of those kludges when we finally remove -support for older versions of our TLS handshake. + * `lib/time` -- Higher level time functions, including fine-gained and + monotonic timers.
+ * `lib/math` -- Floating-point mathematical utilities, including + compatibility code, and probability distributions.
+ * `lib/buf` -- A general purpose queued buffer implementation, + similar to the BSD kernel's "mbuf" structure.
+ * `lib/net` -- Networking code, including address manipulation, + compatibility wrappers, + + * `lib/compress` -- A compatibility wrapper around several + compression libraries, currently including zlib, zstd, and lzma. + + * `lib/geoip` -- Utilities to manage geoip (IP to country) lookups + and formats. + + * `lib/tls` -- Compatibility wrappers around the library (NSS or + OpenSSL, depending on configuration) that Tor uses to implement the + TLS link security protocol. + + * `lib/evloop` -- Tools to manage the event loop and related + functionality, in order to implement asynchronous networking, + timers, periodic events, and other scheduling tasks. + + * `lib/process` -- Utilities and compatibility code to launch and + manage subprocesses. + +### What belongs in lib? + +In general, if you can imagine some program wanting the functionality +you're writing, even if that program had nothing to do with Tor, your +functionality belongs in lib. + +If it falls into one of the existing "lib" categories, your +functionality belongs in lib. + +If you are using platform-specific `#ifdef`s to manage compatibility +issues among platforms, you should probably consider whether you can +put your code into lib.