[tor-bugs] #25386 [Core Tor/Tor]: Link Rust Tests to C Dependencies in Tor (allow integration testing from Rust to C)

[Tor Bug Tracker & Wiki]

#25386: Link Rust Tests to C Dependencies in Tor (allow integration testing from
Rust to C)
-------------------------------------------------+-------------------------
 Reporter:  Hello71                              |          Owner:  Hello71
     Type:  defect                               |         Status:
                                                 |  needs_revision
 Priority:  High                                 |      Milestone:  Tor:
                                                 |  0.3.4.x-final
Component:  Core Tor/Tor                         |        Version:  Tor:
                                                 |  0.3.3.1-alpha
 Severity:  Normal                               |     Resolution:
 Keywords:  rust, tor-test, 033-backport,        |  Actual Points:
  review-group-34, 034-triage-20180328,          |
  034-included-20180401                          |
Parent ID:                                       |         Points:  3
 Reviewer:  isis                                 |        Sponsor:
                                                 |  SponsorQ
-------------------------------------------------+-------------------------

Comment (by Hello71):

 My plan was originally to work on this instead of writing down the
 problems, but apparently I really don't want to work on it, so I'll write
 them down as I see them. This obsoletes most if not all of what I said
 above.

 If we want to use `cargo test` to test things, then the following must be
 resolved:

 1. Cargo needs to learn to link C Tor libraries in when running Rust
 tests. It's fine if the solution to this problem actually links C Tor
 libraries always instead of only during tests, because static libraries do
 not carry dependency information on any modern platform. (I believe not
 even Windows, but correct me if I'm wrong.) This can be accomplished a few
 ways:

   a. set RUSTFLAGS and RUSTDOCFLAGS to "-L... -L... -L... -l... -l..."
 etc, either always or only when testing. the downside of this option is
 that it also links C Tor libraries when compiling Rust Tor dependencies,
 which sounds bad (but I suspect is actually fine if everything is linked
 together in "big bag of parts" manner at the end). always is better
 because cargo will save the value of these variables and recompile *all
 Rust code* (due to the previous problem). this isn't a huge issue now, but
 if we add a bunch of dependencies (like winapi, ugh) that could be an
 issue later.

   b. do the same thing but specify rustflags in cargo configuration

   c. use one or more build scripts that output all C Tor libraries as
 `cargo:rustc-link-lib` lines (make sure you add `cargo:rustc-link-search`
 too obviously). it is possible that using one in `external` only will
 work. I suspect we should rename and split `external` into `tor_c` and
 `tor_c-sys`, where the latter uses bindgen or manual curation for pure
 bindings only, and the former does high-level wrappers, but that can
 probably be done separately. I believe this is supposed to automatically
 work for rustdoc too.

   d. use "links" key and provide the libraries in cargo configuration.
 this is possibly less preferable because it forecloses other uses of build
 scripts, like bindgen.

   I am in favor of method c. It is possible that *none* of these work,
 however. This might be the case if Rust calls into C which calls back into
 the crate we are currently testing. If so, then a complicated build script
 might be necessary that excludes the current crate from the RUSTFLAGS.
 Additionally, note that "make a shared library" does *not* work, because
 then there will be multiple copies of global data structures.

 2. sanitizers: if rustc is used to link tests (mandatory if using cargo
 test), then one of the following must be done:

   a. interpose the linker to add `-fsanitize` options. I suspect `-C link-
 arg` won't work because of the weird order dependency of linker arguments.
 I also suspect that this will not work properly, especially if we want
 Rust tests to be run with sanitizers enabled.

   b. pass -Z sanitizer options to Rust, either via exporting RUSTFLAGS or
 setting build.rustflags in cargo config. this requires that the Rust
 sanitizers are of a compatible version with the C sanitizers, otherwise
 there will be very strange errors, like multiple definition errors.
 usually, using clang to build C code is both necessary and sufficient.
 presently, gcc 8 should work with rustc 1.26, but probably not with newer
 Rust versions (gcc tends to trail behind on sanitizer versions).

 3. a method should be devised if possible to test only one crate at a
 time. komlo asked for this somewhere above. probably a wrapper script is
 the best solution, but I'm not sure about the details. for example, should
 running it from the build directory be necessary? does it need to be
 autoconf generated, or can it be static?

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25386#comment:47>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online