commit 329bd0506c68b514ff3b8d6b61aab223aafca16a Author: Karsten Loesing karsten.loesing@gmx.net Date: Wed Oct 30 11:19:42 2013 +0100
Add libutp report. --- 2013/libutp/.gitignore | 3 + 2013/libutp/chutney.pdf | Bin 0 -> 4843 bytes 2013/libutp/client.pdf | Bin 0 -> 4835 bytes 2013/libutp/libutp.bib | 20 ++ 2013/libutp/libutp.tex | 646 ++++++++++++++++++++++++++++++++++++++++++++ 2013/libutp/shadow.pdf | Bin 0 -> 4841 bytes 2013/libutp/tortechrep.cls | 1 + 7 files changed, 670 insertions(+)
diff --git a/2013/libutp/.gitignore b/2013/libutp/.gitignore new file mode 100644 index 0000000..65ebb17 --- /dev/null +++ b/2013/libutp/.gitignore @@ -0,0 +1,3 @@ +libutp.pdf +libutp-2013-10-30.pdf + diff --git a/2013/libutp/chutney.pdf b/2013/libutp/chutney.pdf new file mode 100644 index 0000000..c7fd5b8 Binary files /dev/null and b/2013/libutp/chutney.pdf differ diff --git a/2013/libutp/client.pdf b/2013/libutp/client.pdf new file mode 100644 index 0000000..d76e842 Binary files /dev/null and b/2013/libutp/client.pdf differ diff --git a/2013/libutp/libutp.bib b/2013/libutp/libutp.bib new file mode 100644 index 0000000..260a3ef --- /dev/null +++ b/2013/libutp/libutp.bib @@ -0,0 +1,20 @@ +@techreport{tor-2012-03-002, + author = {Steven J. Murdoch}, + title = {Datagram Testing Plan}, + institution = {The Tor Project}, + number = {2012-03-002}, + year = {2012}, + month = {March}, + url = {https://research.torproject.org/techreports/datagram-testing-plan-2012-03-16... +} + +@techreport{tor-2011-11-001, + author = {Steven J. Murdoch}, + title = {Comparison of {Tor} Datagram Designs}, + institution = {The Tor Project}, + number = {2011-11-001}, + year = {2011}, + month = {November}, + url = {https://research.torproject.org/techreports/datagram-comparison-2011-11-07.p... +} + diff --git a/2013/libutp/libutp.tex b/2013/libutp/libutp.tex new file mode 100644 index 0000000..5a2c8ff --- /dev/null +++ b/2013/libutp/libutp.tex @@ -0,0 +1,646 @@ +\documentclass{tortechrep} +\usepackage{url} +\usepackage{graphicx} +\usepackage{enumerate} +\usepackage{hyperref} +\usepackage{upgreek} + +\begin{document} + +\title{Evaluation of a libutp-based\Tor Datagram Implementation} + +\author{Karsten Loesing, Steven J. Murdoch, and Rob Jansen} + +\contact{\href{mailto:karsten@torproject.org}{karsten@torproject.org},% +\href{mailto:steven.murdoch@cl.cam.ac.uk}{steven.murdoch@cl.cam.ac.uk},% +\href{mailto:rob.g.jansen@nrl.navy.mil}{rob.g.jansen@nrl.navy.mil}} + +\reportid{2013-10-001} +\date{October 30, 2013} + +\maketitle + +\section{Introduction} + +Datagram designs are a promising approach to overcome Tor's +performance-related problems. +Advantages of datagram designs over stream designs are that they allow +better end-to-end congestion management, reduce queue lengths on nodes, +and prevent cell loss on one circuit delaying cells on other circuits. +In earlier reports we compared possible Tor datagram +designs~\cite{tor-2011-11-001} and outlined a testing +plan~\cite{tor-2012-03-002}. + +In this report we evaluate our implementation of a libutp-based Tor +datagram design. +However, this evaluation does not focus on performance improvements over +the current stream-based Tor implementation, because we found that our +datagram-based Tor implementation is not mature enough for such a +comparison. +The focus is rather on our approach to integrate a datagram design into +the Tor source code and on setting up test environments to evaluate the +new design. +We hope that our findings will be useful when further tweaking our +libutp-based design or implementing other datagram designs. + +In the next section we describe our libutp-based Tor implementation in +sufficient detail for others to understand our source code and to act as +blueprint for implementing other datagram designs. +Sections~\ref{sec:client-private-bridge} to \ref{sec:shadow} outline the +experimental setup that we used to evaluate our implementation and point +out roadblocks that kept us busy longer than necessary. +Section~\ref{sec:conclusion} concludes the report by sketching out +possible next steps. + +\section{Overview of our libutp-based implementation} + +We implemented a Tor datagram design using a slightly modified variant of +Bittorrent's libutp library. +libutp provides $\upmu$TP connections which are similar to TCP connections +but which are transported via UDP using the $\upmu$TP protocol. +In this implementation we don't fully replace TLS connections with +$\upmu$TP connections. +Instead, we open a new $\upmu$TP connection accompanying each TLS +connection. +The TLS connection is used for the handshake between client and relay or +between two relays, namely for \texttt{VERSIONS}, \texttt{NETINFO}, +\texttt{CERTS}, \texttt{AUTH_CHALLENGE}, and \texttt{AUTHENTICATE} cells. +All subsequent cells are then sent over the $\upmu$TP connection. + +libutp itself does not talk to the network, but defines interfaces +for sending and receiving UDP packets. +We implemented all network communication using libevent in four +functions:% +\footnote{Are we missing a function that unbinds our UDP socket when Tor +terminates? +Should we do this in \texttt{channel_tls_free_all}?} + +\begin{itemize} +\item \texttt{retry_utp_listener}, called by +\texttt{retry_all_listeners}, binds a UDP socket to the same port that +we're using as OR listener port for incoming TCP connections. +\item \texttt{utp_read_callback} is called by libevent to inform us of +UDP packets received on the UDP socket which we immediately hand over to +libutp. +\item \texttt{tor_UTPSendToProc} is registered with libutp to tell us +whenever it wants to send UDP packets to the network; +this function does not send UDP packets directly, but buffers them and +asks libevent to inform us when the UDP socket becomes writable. +\item \texttt{utp_write_callback} is called by libevent to send out +previously buffered UDP packets until the buffer is empty or the UDP +socket is not writable anymore. +\end{itemize} + +libutp provides a connection abstraction that is similar to TCP +connections. +The main concept is the $\upmu$TP connection that we can open, close, read +bytes from, and write bytes to. +Opening and closing $\upmu$TP connections happens in the following +functions: + +\begin{itemize} +\item \texttt{channel_tls_connect} opens a new $\upmu$TP connection to a +remote relay, possibly after deciding whether the remote relay is expected +to speak $\upmu$TP with us or not. +\item \texttt{tor_UTPGotIncomingConnection} is called whenever a new +$\upmu$TP connection is opened by a remote client or relay. +\item \texttt{channel_tls_close_method} closes a $\upmu$TP connection +when its corresponding TCP connection is closed.\footnote{Does this mean +incoming $\upmu$TP connections that are never assigned to a TLS connection +are never closed?} +\item \texttt{channel_tls_free_method} unsets callback functions on a +$\upmu$TP connection when the TLS connection is freed.\footnote{Should +free the $\upmu$TP connection, too, unless libutp does that for us; and is +this really necessary, or should libutp not call any callbacks anymore +after we ask it to close a connection?} +\end{itemize} + +Once a $\upmu$TP connection is established, we can read or write bytes +from/to it. +These bytes do not include $\upmu$TP headers which libutp adds and removes +transparently for us. +We wrote six callback functions that are called by libutp to read from and +write to $\upmu$TP connections and to inform us about about any state +change of established $\upmu$TP connections: + +\begin{itemize} +\item \texttt{tor_UTPGetRBSize} is called when new bytes have been +received on an existing $\upmu$TP connection and libutp wants to find out +if there's still space in our read buffer. +\item \texttt{tor_UTPOnReadProc} is called to hand over newly received +bytes. +\item \texttt{tor_UTPOnWriteProc} is called when libutp wants to write +bytes to an existing $\upmu$TP connection; note the additional level of +indirection here: we're going to handle the part where we tell libutp that +we want to write bytes to a $\upmu$TP connection further down below. +\item \texttt{tor_UTPOnStateChangeProc} is called when a $\upmu$TP +connection becomes writable% +\footnote{Maybe we do the +wrong thing here by calling \texttt{UTP_Write} if the connection becomes +writable; maybe libutp can already handle that just fine?} +or when it becomes non-writable; +if a $\upmu$TP connection is destroyed, we also close the TLS connection +associated with it. +\item \texttt{tor_UTPOnErrorProc} informs us of errors with a $\upmu$TP +connection; we currently don't do anything with this information. +\item \texttt{tor_UTPOnOverheadProc} informs us of the overhead produced +by UDP headers, retransmissions, etc.; we currently don't do anything with +this information. +\end{itemize} + +As a result of always using a TLS and a $\upmu$TP connection in tandem, we +had to define a simple protocol to match the two connection types: +the first 48~bytes on a $\upmu$TP connection sent from initiator to +responder are the connection identifier which is simply the SSL session +key of the corresponding TLS connection. +This protocol is implemented in three functions: + +\begin{itemize} +\item \texttt{tor_tls_copy_master_key} is used by both initiator and +responder of a $\upmu$TP connection to obtain the SSL session key of a TLS +connection. +\item \texttt{channel_tls_send_utp_id} is used by the initiator +of a $\upmu$TP connection to send the 48~bytes long connection identifier +prior to any other data. +\item \texttt{tor_UTPOnReadProc}, which was already mentioned above, is +used by the responder of a $\upmu$TP connection; it reads the first +48~bytes on a $\upmu$TP connection, goes through the list of TLS +connections, and finds the +one that has the matching SSL session key. +\end{itemize} + +The most important piece that is still missing is where Tor writes cells +to $\upmu$TP connections and where incoming cells received over $\upmu$TP +are handed over to Tor: + +\begin{itemize} +\item \texttt{channel_tls_write.*cell_method} (three quite similar +functions) write Tor cells to the outgoing buffer of a $\upmu$TP +connection and then call \texttt{UTP_Write} to tell libutp that there are +bytes to be written; this is the second part of the additional level of +indirection mentioned above: this call does not make libutp send out bytes +immediately, but libutp will ask for the buffered bytes whenever it's +ready to write them by calling \texttt{tor_UTPOnWriteProc}. +\item \texttt{run_connection_housekeeping} sends \texttt{PADDING} cells +over a $\upmu$TP connection if one exists for a TLS connection. +\item \texttt{utp_process_cells_from_inbuf} handles incoming cells that +\texttt{tor_UTPOnReadProc} put into the incoming buffer of a $\upmu$TP +connection, but only when the TLS connection is done handshaking. +\item \texttt{channel_tls_handle_state_change_on_orconn} informs us +that a TLS connection is done handshaking and calls +\texttt{utp_process_cells_from_inbuf} to handle incoming cells that +have arrived during the handshake process. +\end{itemize} + +Last but not least we extended a maintenance function for $\upmu$TP +connections: + +\begin{itemize} +\item \texttt{run_scheduled_events} calls libutp's +\texttt{UTP_CheckTimeouts} once per second% +\footnote{Maybe this needs to happen much more often?} +which is vital for retransmitting UDP packets and generally keeping +$\upmu$TP connections alive. +\end{itemize} + +\section{Setup: Client and private bridge in public Tor network} +\label{sec:client-private-bridge} + +The easiest way to test a new Tor datagram implementation is to use it +only for the communication between two Tor nodes controlled by the tester. +We came up with a setup consisting of a client and a private bridge. +The client is configured to connect only via our private bridge. +A private bridge is a bridge that doesn't publish its server descriptor to +the bridge authority. +The reason for using a private bridge in this setup is to avoid that other +clients connect via it and mess with our experiment. +This experimental setup is useful to test whether a new Tor datagram +implementation works at all, and to do some basic debugging and tweaking. + +Building our modified libutp and Tor sources is not as trivial as one +would expect. +That is why we decided to list detailed instructions here. +The following instructions assume a freshly installed Ubuntu 13.10 as +operating system. +Instructions apply to both client and bridge: + +{\small +\begin{verbatim} +sudo apt-get install build-essential git automake libssl-dev libevent-dev +mkdir src +cd src/ +git clone https://github.com/kloesing/libutp +cd libutp/ +git checkout -b utp origin/utp +make +cd ../ +git clone https://git.torproject.org/tor.git +cd tor/ +git remote add karsten https://git.torproject.org/karsten/tor.git +git fetch karsten +git checkout -b utp karsten/utp +\end{verbatim} +} + +The client's source can be compiled unchanged. +But the \emph{bridge's} source code must be changed to avoid connecting to +any other relay via $\upmu$TP. +In order to do so, \texttt{channel_tls_connect} must be edited. +Otherwise, the bridge will not be able to connect to the Tor network. +The following code change is necessary: + +{\small +\begin{verbatim} +diff --git a/src/or/channeltls.c b/src/or/channeltls.c +index 96055a3..ce68a9e 100644 +--- a/src/or/channeltls.c ++++ b/src/or/channeltls.c +@@ -553,7 +553,7 @@ channel_tls_connect(const tor_addr_t *addr, uint16_t port, + /* Create a uTP connection */ + tor_addr_to_sockaddr(addr, port, (struct sockaddr*)&sin, sizeof(sin)); + tor_addr_to_str(addr_str, addr, sizeof(addr_str), 0); +- if (1) { ++ if (0) { + log_info(LD_CHANNEL, + "Trying uTP connection to %s", addr_str); + tlschan->utp = UTP_Create(tor_UTPSendToProc, tlschan, +\end{verbatim} +} + +Now, both client and bridge source code can be compiled: + +{\small +\begin{verbatim} +./autogen.sh +LDFLAGS="-L/home/ubuntu/src/libutp" CFLAGS="-I/home/ubuntu/src/libutp" \ + LIBS="-lutp -lrt" ./configure --disable-asciidoc --enable-gcc-warnings +make +cd ../ +\end{verbatim} +} + +The relevant parts of the client's torrc file are: + +{\small +\begin{verbatim} +SocksPort 9000 +UseBridges 1 +Bridge <ip-address-of-our-bridge>:9001 +\end{verbatim} +} + +The relevant parts of the private bridge's torrc file are: + +{\small +\begin{verbatim} +SocksPort 9000 +ORPort 9001 +BridgeRelay 1 +PublishServerDescriptor 0 +\end{verbatim} +} + +\begin{figure} +\centering +\includegraphics[width=0.9\textwidth]{client.pdf} +\caption{Bootstrap process of our client connecting via our private bridge +over a $\upmu$TP connection to the public Tor network.} +\label{fig:client} +\end{figure} + +Figure~\ref{fig:client} shows five runs of our client connecting via our +private bridge over a $\upmu$TP connection to the public Tor network. +One of the bootstrap processes was quite fast, finishing in 30 seconds, +but the others took over 100 seconds to complete. +This plot is only supposed to show that the code is working and does not +serve as performance comparison to current Tor. + +While experimenting with this setup, we found and fixed a few issues in +our implementation. +We want to explain these issues briefly here to give some examples of the +type of bugs that can be found in this setup:% +\footnote{Two more issues that may be worth mentioning: +2e0ac35: If a connection becomes writable, try to write (yet unclear if +this actually fixed a bug); +43a013b: Process incoming cells on new connections immediately.} + +\begin{itemize} +\item Calling \texttt{UTP_CheckTimeouts} is more important for libutp +than one would expect. +We discovered a pattern where the sender sends just a single packet +containing 1382 bytes to the receiver, the receiver processes the full Tor +cells from it, and then they both sit there waiting until the next call to +\texttt{UTP_CheckTimeouts}. +Only then the receiver sends a 20 byte $\upmu$TP control message to the +sender which immediately sends another 1382 bytes before going silent +again. +\item When libutp told us in \texttt{tor_UTPOnStateChangeProc} that a +connection becomes writable, we didn't do anything with that information. +Now we attempt to write in that case.% +\footnote{Actually, it's unclear if we actually \emph{need} to do anything +here, or if libutp can handle this just fine.} +\item We previously missed a case when we received bytes on a new +$\upmu$TP connection that contained both a connection identifier and one +or more full Tor cells. +We should process the bytes exceeding the connection identifier +immediately, because otherwise nothing might trigger processing them +later. +\item Unrelated to the datagram transport, we discovered that we must +delete all \texttt{cached-*} files in the client's data directory +\emph{and} its \texttt{state} file before starting a new bootstrap +process. +Whenever we left the \texttt{state} file in place, the bootstrap process +stalled for one minute which was unrelated to the datagram transport code. +\end{itemize} + +\section{Setup: Small private Tor network created by Chutney} +\label{sec:chutney} + +A more advanced way to test a Tor datagram implementation is to set up a +private Tor network and use the datagram transport for communication +between most or even all nodes. +Chutney makes it easy to generate torrc files for a small private Tor +network that is supposed to run locally. +This setup makes it somewhat harder to track down bugs, because all nodes +are running the possibly broken Tor code, rather than just the initiator +and responder of a single connection. +The advantage of this setup is that it allows to test the new code on all +different roles, including the role of a directory authority. + +Running our implementation in Chutney is, fortunately, quite simple. +We start by building libutp and Tor as explained in +Section~\ref{sec:client-private-bridge}, but without applying the change +for the private bridge. +Then we clone Chutney and start a local Chutney network as follows: + +{\small +\begin{verbatim} +cd ~/src/ +git clone https://git.torproject.org/chutney.git +cd chutney/ +export PATH=~/src/tor/src/or:~/src/tor/src/tools:$PATH +\end{verbatim} +} + +Before running Chutney, we need to remove all torrc options which did not +exist in Tor at the time of branching our code. +The following patch should apply cleanly to commit 562b8f1: + +{\small +\begin{verbatim} +diff --git a/torrc_templates/authority.tmpl b/torrc_templates/authority.tmpl +index 7bf99af..2fcb974 100644 +--- a/torrc_templates/authority.tmpl ++++ b/torrc_templates/authority.tmpl +@@ -4,6 +4,5 @@ V3AuthoritativeDirectory 1 + ContactInfo auth${nodenum}@test.test + ExitPolicy reject *:* + TestingV3AuthInitialVotingInterval 300 +-TestingV3AuthInitialVoteDelay 2 +-TestingV3AuthInitialDistDelay 2 +-TestingV3AuthVotingStartOffset 0 ++TestingV3AuthInitialVoteDelay 20 ++TestingV3AuthInitialDistDelay 20 +diff --git a/torrc_templates/client.tmpl b/torrc_templates/client.tmpl +index 1eb1d99..be2c9dc 100644 +--- a/torrc_templates/client.tmpl ++++ b/torrc_templates/client.tmpl +@@ -3,4 +3,3 @@ SocksPort $socksport + #NOTE: Setting TestingClientConsensusDownloadSchedule doesn't + # help -- dl_stats.schedule is not DL_SCHED_CONSENSUS + # at boostrap time. +-TestingClientDownloadSchedule 10, 2, 2, 4, 4, 8, 13, 18, 25, 40, 60 +diff --git a/torrc_templates/relay.tmpl b/torrc_templates/relay.tmpl +index 2f4b7f1..c2f8eb5 100644 +--- a/torrc_templates/relay.tmpl ++++ b/torrc_templates/relay.tmpl +@@ -6,4 +6,3 @@ DirPort $dirport + #NOTE: Setting TestingServerConsensusDownloadSchedule doesn't + # help -- dl_stats.schedule is not DL_SCHED_CONSENSUS + # at boostrap time. +-TestingServerDownloadSchedule 10, 2, 2, 4, 4, 8, 13, 18, 25, 40, 60 +\end{verbatim} +} + +Once that is done, we can create and start a private Tor network: + +{\small +\begin{verbatim} +./chutney configure networks/basic +./chutney start networks/basic +./chutney hup networks/basic +./chutney stop networks/basic +\end{verbatim} +} + +\begin{figure} +\centering +\includegraphics[width=0.9\textwidth]{chutney.pdf} +\caption{Bootstrap process of clients connecting over $\upmu$TP +connections to a private Tor network created by Chutney.} +\label{fig:chutney} +\end{figure} + +Figure~\ref{fig:chutney} shows two clients connecting over $\upmu$TP +connections to a private Tor network created by Chutney. +These bootstrap times are longer than in our previous setup, because +clients start at the same time as the directory authorities and relays, +so there is no consensus yet and no way for clients to bootstrap +immediately. +But again, this plot is only supposed to show that the code is running, +which is the case here. + +It's worth mentioning that we found and fixed another bug using Chutney: + +\begin{itemize} +\item Our earlier approach to create new connections to a remote relay was +to first create the TLS connection and then the $\upmu$TP connection. +However, we only realized whether creating the TLS connection failed after +creating the $\upmu$TP connection. +In that case, we'd free the TLS connection but not tell the $\upmu$TP +connection. +This led to segmentation faults on directory authorities only which we +would not have found in the earlier setup. +\end{itemize} + +\section{Setup: Large private Tor network using Shadow} +\label{sec:shadow} + +The most sophisticated way to test new Tor datagram designs is to run them +in large private Tor networks using the discrete-event simulator Shadow. +These experiments not only reveal bugs, but can also provide performance +results which can be compared to other datagram designs or to the original +stream design. +However, as indicated in the introduction, we do not focus on performance +improvements here, because we found that our datagram implementation is +not mature enough for such a comparison. +A downside of using Shadow is that it adds even more complexity to the +experiment and may introduce problems that are not present outside of the +Shadow environment. + +Building libutp, Tor, and Shadow is somewhat more complicated than setting +up the earlier test environments. +We again start by building libutp and Tor as explained in +Section~\ref{sec:client-private-bridge}, but without applying the change +for the private bridge. +Next steps for building Shadow, including installing Clang/LLVM from +source, are as follows: + +{\small +\begin{verbatim} +sudo apt-get install cmake tidy libtidy-dev libglib2.0 libglib2.0-dev +wget http://llvm.org/releases/3.2/llvm-3.2.src.tar.gz +wget http://llvm.org/releases/3.2/clang-3.2.src.tar.gz +tar xaf llvm-3.2.src.tar.gz +tar xaf clang-3.2.src.tar.gz +cp -R clang-3.2.src llvm-3.2.src/tools/clang +cd llvm-3.2.src/ +mkdir build +cd build/ +cmake -DCMAKE_INSTALL_PREFIX=/home/ubuntu/.local ../. +make +make install +export PATH=~/.local/bin/:$PATH +cd ~/src/ +git clone https://github.com/shadow/shadow.git +cd shadow +\end{verbatim} +} + +We need to make one change to Shadow to make it simulate our branch +correctly: it needs to build the libutp source files itself, so that each +virtual Tor instance has its own global libutp variables. +The following patch should apply cleanly to commit 3b1a85e: + +{\small +\begin{verbatim} +diff --git a/src/plugins/scallion/CMakeLists.txt b/src/plugins/scallion/CMakeLists.txt +index 8a6974f..2de3046 100644 +--- a/src/plugins/scallion/CMakeLists.txt ++++ b/src/plugins/scallion/CMakeLists.txt +@@ -17,6 +17,8 @@ include_directories(${CURRENT_TOR_DIR}/src) + include_directories(${CURRENT_TOR_DIR}/src/common) + include_directories(${CURRENT_TOR_DIR}/src/or) + include_directories(${CURRENT_TOR_DIR}/src/ext) ++include_directories(/home/ubuntu/src/libutp /home/ubuntu/src/libutp/utp_config_lib) + + include_directories(AFTER ${RT_INCLUDES} ${DL_INCLUDES} ${M_INCLUDES} ${GLIB_INCLUDES} ${EVENT2_INCLUDES} ${OPENSSL_INCLUDES} ${TOR_INCLUDES} ${SHA + +@@ -83,6 +85,7 @@ list(REMOVE_ITEM toror_sources + #endforeach(headerpath) + + ## tor needs these defined ++add_definitions(-DPOSIX="1") + add_definitions(-DLOCALSTATEDIR="/usr/local/var" -DSHARE_DATADIR="/usr/local/var" -DBINDIR="/usr/local/bin") + remove_definitions(-DNDEBUG) + ## disable warnings from tor code +@@ -90,6 +93,7 @@ add_cflags("-w") + + ## create and install a shared library that can plug into shadow + add_bitcode(shadow-plugin-scallion-bitcode ++ /home/ubuntu/src/libutp/utp.cpp /home/ubuntu/src/libutp/utp_utils.cpp + shd-scallion-plugin.c shd-scallion.c ${toror_sources} ${torcommon_sources} ${torextcurve_sources}) + add_plugin(shadow-plugin-scallion shadow-plugin-scallion-bitcode) +\end{verbatim} +} + +Then we run the remaining steps to build, install, and run Shadow: + +{\small +\begin{verbatim} +./setup dependencies +./setup build -g -i /home/ubuntu/src/libutp \ + -i /usr/include/x86_64-linux-gnu/c++/4.8 -l /home/ubuntu/src/libutp \ + --tor-prefix /home/ubuntu/src/tor --tor-lib utp +./setup install +export PATH=~/.shadow/bin/:$PATH +cd resource/examples/scallion/ +tar xf minimal.tar.xz +cd minimal/ +scallion --log-level=INFO +\end{verbatim} +} + +\begin{figure} +\centering +\includegraphics[width=0.9\textwidth]{shadow.pdf} +\caption{Bootstrap process of clients connecting over $\upmu$TP +connections to a private Tor network created by Shadow.} +\label{fig:shadow} +\end{figure} + +Figure~\ref{fig:shadow} shows two clients connecting over $\upmu$TP +connections to a private Tor network created by Shadow. +Once again, this plot is only supposed to show that the code is running. + +Here are a few pitfalls from testing our datagram implementation using +Shadow, again meant as examples for future tests: + +\begin{itemize} +\item We should register an \texttt{EV_WRITE} event with libevent +whenever we want to write to a UDP socket, rather than writing to the +socket directly. +While this doesn't matter as much when running our implementation outside +of Shadow, Shadow cares about this detail. +It's also the better design.% +\footnote{However, it's still unclear whether this actually solved a bug.} +\item Apparently, the \texttt{do_main_loop} function is never called in +Shadow, so our callback that we added to check $\upmu$TP timeouts more +often than once per second was never registered with libevent. +In general, it's a good idea to add log statements to all changed code, +and make sure it gets executed as expected. +\item We found that Shadow had a bug% +\footnote{\url{https://github.com/shadow/shadow/issues/180%7D%7D +where UDP sockets that bind to +\texttt{INADDR_ANY:port} later got their port changed. +They still received packets from the initial bound port, but packets they +sent appeared to come from the changed port. +\item libutp uses a few global variables, including a list of open +$\upmu$TP connections. +In the case of a single Tor process using libutp as a library, this works +just fine. +But in case of Shadow, the global variables are shared among all Tor +instances, which leads to chaos very quickly. +The fix is to build libutp as part of the scallion plug-in to ensure that +Shadow will hoist the libutp variables and make a copy of them for every +virtual Tor node. +\end{itemize} + +\section{Conclusion} +\label{sec:conclusion} + +One conclusion from experimenting with our libutp-based datagram transport +implementation is that it's far from trivial to replace the transport in +Tor. +And we only looked at the allegedly simple case of using the datagram +transport in a separate (part of a) Tor network. + +The next step will be to debug our implementation enough to compare its +performance to an unchanged Tor version. +It may be that there are still bugs in our code, or it may be that we need +to tweak libutp's parameters to make it more suitable for Tor. +We may also want to rethink our design decision to always open both a TCP +connection and a $\upmu$TP connection in parallel, though this should not +affect performance results much. +We might also want to switch from a single $\upmu$TP connection between +client and relay or between two relays to one $\upmu$TP connection per +circuit. +Performance evaluations should also include setups with additional TCP +load in the network to see how much $\upmu$TP yields to TCP and how that +degrades Tor performance. + +More steps further down the road are to try out other datagram transports +than libutp and to think about deploying a new transport in the existing +Tor network. +We have still got a long way to go. + +\section*{Acknowledgments} + +Nick Mathewson, Sebastian Hahn, Matthew Finkel, and Yawning Angel provided +valuable feedback and helped out with code fixes while writing this +technical report. + +\bibliography{libutp} + +\end{document} + diff --git a/2013/libutp/shadow.pdf b/2013/libutp/shadow.pdf new file mode 100644 index 0000000..f532117 Binary files /dev/null and b/2013/libutp/shadow.pdf differ diff --git a/2013/libutp/tortechrep.cls b/2013/libutp/tortechrep.cls new file mode 120000 index 0000000..4c24db2 --- /dev/null +++ b/2013/libutp/tortechrep.cls @@ -0,0 +1 @@ +../../tortechrep.cls \ No newline at end of file