[tor-commits] [tor/master] Update documentation and testing integration for fuzzing

[nickm at torproject.org]

commit d71fc474385281453eaa93522479d32af85c94ef
Author: Nick Mathewson <nickm at torproject.org>
Date:   Fri Jan 27 11:16:23 2017 -0500

    Update documentation and testing integration for fuzzing
---
 doc/HACKING/Fuzzing.md            | 63 ++++++++++++++++++++++++++++-----------
 src/test/fuzz/include.am          |  2 ++
 src/test/fuzz_static_testcases.sh | 20 ++++++++++---
 3 files changed, 63 insertions(+), 22 deletions(-)

diff --git a/doc/HACKING/Fuzzing.md b/doc/HACKING/Fuzzing.md
index 36f0fc4..f5502b3 100644
--- a/doc/HACKING/Fuzzing.md
+++ b/doc/HACKING/Fuzzing.md
@@ -1,12 +1,53 @@
 = Fuzzing Tor
 
+== The simple version (no fuzzing, only tests)
+
+Check out fuzzing-corpora, and set TOR_FUZZ_CORPORA to point to the place
+where you checked it out.
+
 To run the fuzzing test cases in a deterministic fashion, use:
   make fuzz
 
-  [I've turned this off for now. - NM]
 
-To build the fuzzing harness binaries, use:
-  make fuzzers
+== Different kinds of fuzzing
+
+Right now we support three different kinds of fuzzer.
+
+First, there's American Fuzzy Lop (AFL), a fuzzer that works by forking
+a target binary and passing it lots of different inputs on stdin.  It's the
+trickiest one to set up, so I'll be describing it more below.
+
+Second, there's libFuzzer, a llvm-based fuzzer that you link in as a library,
+and it runs a target function over and over.  To use this one, you'll need to
+have a reasonably recent clang and libfuzzer installed.  At that point, you
+just build with --enable-expensive-hardening and --enable-libfuzzer.  That
+will produce a set of binaries in src/test/fuzz/lf-fuzz-* .  These programs
+take as input a series of directories full of fuzzing examples.  For more
+information on libfuzzer, see http://llvm.org/docs/LibFuzzer.html
+
+Third, there's Google's OSS-Fuzz infrastructure, which expects to get all of
+its.  For more on this, see https://github.com/google/oss-fuzz and the
+projects/tor subdirectory.  You'll need to mess around with Docker a bit to
+test this one out; it's meant to run on Google's infrastructure.
+
+In all cases, you'll need some starting examples to give the fuzzer when it
+starts out.  There's a set in the "fuzzing-corpora" git repository.  Try
+setting TOR_FUZZ_CORPORA to point to a checkout of that repository
+
+== Writing Tor fuzzers
+
+A tor fuzzing harness should have:
+* a fuzz_init() function to set up any necessary global state.
+* a fuzz_main() function to receive input and pass it to a parser.
+* a fuzz_cleanup() function to clear global state.
+
+Most fuzzing frameworks will produce many invalid inputs - a tor fuzzing
+harness should rejecting invalid inputs without crashing or behaving badly.
+
+But the fuzzing harness should crash if tor fails an assertion, triggers a
+bug, or accesses memory it shouldn't. This helps fuzzing frameworks detect
+"interesting" cases.
+
 
 == Guided Fuzzing with AFL
 
@@ -47,7 +88,7 @@ don't care about memory limits.
 
 To Run:
   mkdir -p src/test/fuzz/fuzz_http_findings
-  ../afl/afl-fuzz -i src/test/fuzz/data/http -x src/test/fuzz/dict/http -o src/test/fuzz/fuzz_http_findings -m <asan-memory-limit> -- src/test/fuzz_dir
+  ../afl/afl-fuzz -i ${TOR_FUZZ_CORPORA}/http -o src/test/fuzz/fuzz_http_findings -m <asan-memory-limit> -- src/test/fuzz_dir
 
 
 AFL has a multi-core mode, check the documentation for details.
@@ -57,20 +98,6 @@ macOS (OS X) requires slightly more preparation, including:
 * using afl-clang (or afl-clang-fast from the llvm directory)
 * disabling external crash reporting (AFL will guide you through this step)
 
-== Writing Tor fuzzers
-
-A tor fuzzing harness should have:
-* a fuzz_init() function to set up any necessary global state.
-* a fuzz_main() function to receive input and pass it to a parser.
-* a fuzz_cleanup() function to clear global state.
-
-Most fuzzing frameworks will produce many invalid inputs - a tor fuzzing
-harness should rejecting invalid inputs without crashing or behaving badly.
-
-But the fuzzing harness should crash if tor fails an assertion, triggers a
-bug, or accesses memory it shouldn't. This helps fuzzing frameworks detect
-"interesting" cases.
-
 == Triaging Issues
 
 Crashes are usually interesting, particularly if using AFL_HARDEN=1 and --enable-expensive-hardening. Sometimes crashes are due to bugs in the harness code.
diff --git a/src/test/fuzz/include.am b/src/test/fuzz/include.am
index bca0a86..c9c1747 100644
--- a/src/test/fuzz/include.am
+++ b/src/test/fuzz/include.am
@@ -246,3 +246,5 @@ noinst_LIBRARIES += $(OSS_FUZZ_FUZZERS)
 oss-fuzz-fuzzers:  oss-fuzz-prereqs $(OSS_FUZZ_FUZZERS)
 fuzzers: $(FUZZERS) $(LIBFUZZER_FUZZERS)
 
+fuzz: $(FUZZERS)
+	$(top_srcdir)/src/test/fuzz_static_testcases.sh
diff --git a/src/test/fuzz_static_testcases.sh b/src/test/fuzz_static_testcases.sh
index 276bc6e..bfe1677 100755
--- a/src/test/fuzz_static_testcases.sh
+++ b/src/test/fuzz_static_testcases.sh
@@ -5,11 +5,23 @@
 
 set -e
 
+if [ -z "${TOR_FUZZ_CORPORA}" ] || [ ! -d "${TOR_FUZZ_CORPORA}" ] ; then
+    echo "You need to set TOR_FUZZ_CORPORA to point to a checkout of "
+    echo "the 'fuzzing-corpora' repository."
+    exit 77
+fi
+
+
+
 for fuzzer in "${builddir:-.}"/src/test/fuzz/fuzz-* ; do
     f=`basename $fuzzer`
     case="${f#fuzz-}"
-    echo "Running tests for ${case}"
-    for entry in ${abs_top_srcdir:-.}/src/test/fuzz/data/${case}/*; do
-	"${fuzzer}" "--err" < "$entry"
-    done
+    if [ -d "${TOR_FUZZ_CORPORA}/${case}" ]; then
+        echo "Running tests for ${case}"
+        for entry in "${TOR_FUZZ_CORPORA}/${case}/"*; do
+	    "${fuzzer}" "--err" < "$entry"
+        done
+    else
+	echo "No tests found for ${case}"
+    fi
 done