tor-commits March 2016

tor-commits@lists.torproject.org

16 participants
1259 discussions

[snowflake/master] Use 192.81.135.242:9902 as the default relay.
by dcf＠torproject.org 27 Mar '16

27 Mar '16

commit 57c0724491ecab7abb2d57067fe762de48bc0d6a Author: David Fifield <david(a)bamsoftware.com> Date: Sun Mar 27 12:49:20 2016 -0700 Use 192.81.135.242:9902 as the default relay. This is the relay that reports using the transport name "snowflake". https://bugs.torproject.org/18627 --- proxy/README.md | 2 +- proxy/snowflake.coffee | 2 +- server/server.go | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/proxy/README.md b/proxy/README.md index 0398d3d..4686783 100644 --- a/proxy/README.md +++ b/proxy/README.md @@ -23,7 +23,7 @@ To run locally, either: ### Parameters With no parameters, -snowflake uses the default relay `192.81.135.242:9901` and +snowflake uses the default relay `192.81.135.242:9902` and uses automatic signaling with the default broker at `https://snowflake-reg.appspot.com/`. diff --git a/proxy/snowflake.coffee b/proxy/snowflake.coffee index 7e26481..a96dbdf 100644 --- a/proxy/snowflake.coffee +++ b/proxy/snowflake.coffee @@ -10,7 +10,7 @@ this must always act as the answerer. DEFAULT_BROKER = 'snowflake-reg.appspot.com' DEFAULT_RELAY = host: '192.81.135.242' - port: 9901 + port: 9902 COPY_PASTE_ENABLED = false DEBUG = false diff --git a/server/server.go b/server/server.go index c424c29..fc41364 100644 --- a/server/server.go +++ b/server/server.go @@ -4,7 +4,7 @@ // // Usage in torrc: // ExtORPort auto -// ServerTransportPlugin snowflake exec server --port 9901 +// ServerTransportPlugin snowflake exec server --port 9902 package main import (

1 0

[stem/master] Don't provide man's '--encoding' argument on OSX
by atagar＠torproject.org 27 Mar '16

27 Mar '16

commit 878f90cc3aba39fe86f762ef15bac9f848938f4b Author: Damian Johnson <atagar(a)torproject.org> Date: Sun Mar 27 12:33:02 2016 -0700 Don't provide man's '--encoding' argument on OSX Turns out the argument isn't available on OSX. Caught by Sebastian... https://trac.torproject.org/projects/tor/ticket/18660 --- stem/cached_tor_manual.cfg | 6 +++--- stem/manual.py | 10 ++++++++-- test/integ/manual.py | 3 ++- test/unit/manual.py | 1 + 4 files changed, 14 insertions(+), 6 deletions(-) diff --git a/stem/cached_tor_manual.cfg b/stem/cached_tor_manual.cfg index 835024c..fade04b 100644 --- a/stem/cached_tor_manual.cfg +++ b/stem/cached_tor_manual.cfg @@ -6,8 +6,8 @@ description |Basically, Tor provides a distributed network of servers or relays ("onion routers"). Users bounce their TCP streams -- web traffic, ftp, ssh, etc. -- around the network, and recipients, observers, and even the relays themselves have difficulty tracking the source of the stream. | |By default, tor will only act as a client only. To help the network by providing bandwidth as a relay, change the ORPort configuration option -- see below. Please also consult the documentation on the Tor Project's website. -man_commit 424af93ded7e1d9d98733ed17b2b6fee143262b9 -stem_commit 7f4fcf8f6da3941e84376ee67747135e7f34462b +man_commit 94cb8792e8c28e75bc71434fc557ddefa5c03083 +stem_commit 27a654e2e0495c7ac701d792e7db7ed47d4e9753 commandline_options -f FILE => Specify a new configuration file to contain further Tor configuration options OR pass - to make Tor read its configuration from standard input. (Default: @CONFDIR@/torrc, or $HOME/.torrc if that file is not found) commandline_options --ignore-missing-torrc => Specifies that Tor should treat a missing torrc file as though it were empty. Ordinarily, Tor does this for missing default torrc files, but not for those specified on the command line. commandline_options --list-fingerprint => Generate your keys and output your nickname and fingerprint. @@ -110,7 +110,7 @@ config_options.ExcludeNodes.name ExcludeNodes config_options.ExcludeNodes.usage node,node,... config_options.ExcludeNodes.summary Relays or locales never to be used in circuits config_options.ExcludeNodes.description -|A list of identity fingerprints, country codes, and address patterns of nodes to avoid when building a circuit. Country codes are 2-letter ISA3166 codes, and must be wrapped in braces; fingerprints may be preceded by a dollar sign. (Example: ExcludeNodes ABCD1234CDEF5678ABCD1234CDEF5678ABCD1234, {cc}, 255.254.0.0/8) +|A list of identity fingerprints, country codes, and address patterns of nodes to avoid when building a circuit. Country codes are 2-letter ISO3166 codes, and must be wrapped in braces; fingerprints may be preceded by a dollar sign. (Example: ExcludeNodes ABCD1234CDEF5678ABCD1234CDEF5678ABCD1234, {cc}, 255.254.0.0/8) | |By default, this option is treated as a preference that Tor is allowed to override in order to keep working. For example, if you try to connect to a hidden service, but you have excluded all of the hidden service's introduction points, Tor will connect to one of them anyway. If you do not want this behavior, set the StrictNodes option (documented below). | diff --git a/stem/manual.py b/stem/manual.py index 31f9ed1..be7b224 100644 --- a/stem/manual.py +++ b/stem/manual.py @@ -360,6 +360,10 @@ class Manual(object): """ Reads and parses a given man page. + On OSX the man command doesn't have an '--encoding' argument so its results + may not quite match other platforms. For instance, it normalizes long + dashes into '--'. + :param str man_path: path argument for 'man', for example you might want '/path/to/tor/doc/tor.1' to read from tor's git repository @@ -368,10 +372,12 @@ class Manual(object): :raises: **IOError** if unable to retrieve the manual """ + man_cmd = 'man %s -P cat %s' % ('' if stem.util.system.is_mac() else '--encoding=ascii', man_path) + try: - man_output = stem.util.system.call('man --encoding=ascii -P cat %s' % man_path, env = {'MANWIDTH': '10000000'}) + man_output = stem.util.system.call(man_cmd, env = {'MANWIDTH': '10000000'}) except OSError as exc: - raise IOError("Unable to run 'man --encoding=ascii -P cat %s': %s" % (man_path, exc)) + raise IOError("Unable to run '%s': %s" % (man_cmd, exc)) categories, config_options = _get_categories(man_output), OrderedDict() diff --git a/test/integ/manual.py b/test/integ/manual.py index d20c407..dbef86a 100644 --- a/test/integ/manual.py +++ b/test/integ/manual.py @@ -107,7 +107,8 @@ class TestManual(unittest.TestCase): stem.manual.download_man_page(file_handle = tmp) self.man_path = tmp.name - self.man_content = stem.util.system.call('man --encoding=ascii -P cat %s' % self.man_path, env = {'MANWIDTH': '10000000'}) + man_cmd = 'man %s -P cat %s' % ('' if stem.util.system.is_mac() else '--encoding=ascii', self.man_path) + self.man_content = stem.util.system.call(man_cmd, env = {'MANWIDTH': '10000000'}) except Exception as exc: self.download_error = 'Unable to download the man page: %s' % exc diff --git a/test/unit/manual.py b/test/unit/manual.py index 99628b0..be0ee9c 100644 --- a/test/unit/manual.py +++ b/test/unit/manual.py @@ -268,6 +268,7 @@ class TestManual(unittest.TestCase): self.assertEqual(b'a2x output', output.getvalue()) call_mock.assert_called_once_with('a2x -f manpage /no/such/path/tor.1.txt') + @patch('stem.util.system.is_mac', Mock(return_value = False)) @patch('stem.util.system.call', Mock(side_effect = OSError('man --encoding=ascii -P cat tor returned exit status 16'))) def test_from_man_when_manual_is_unavailable(self): try:

1 0

[snowflake/master] begin multiplex snowflake connect loop for #31
by arlo＠torproject.org 27 Mar '16

27 Mar '16

commit 451edb6decf7790db2ec0efceb94dc994952b0e8 Author: Serene Han <keroserene+git(a)gmail.com> Date: Sun Mar 20 11:11:21 2016 -0700 begin multiplex snowflake connect loop for #31 --- client/snowflake.go | 21 ++++++++++++++++++--- client/webrtc.go | 3 --- 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/client/snowflake.go b/client/snowflake.go index c78eed0..ac593a4 100644 --- a/client/snowflake.go +++ b/client/snowflake.go @@ -12,6 +12,7 @@ import ( "os/signal" "sync" "syscall" + "time" "git.torproject.org/pluggable-transports/goptlib.git" "github.com/keroserene/go-webrtc" @@ -19,10 +20,10 @@ import ( var ptInfo pt.ClientInfo -// var logFile *os.File var brokerURL string var frontDomain string var iceServers IceServerList +var snowflakes []*webRTCConn // When a connection handler starts, +1 is written to this channel; when it // ends, -1 is written. @@ -30,6 +31,7 @@ var handlerChan = make(chan int) const ( ReconnectTimeout = 5 + SnowflakeCapacity = 1 ) func copyLoop(a, b net.Conn) { @@ -53,6 +55,20 @@ type SnowflakeChannel interface { Close() error } +// Maintain |WebRTCSlots| number of open connections to +// transfer to SOCKS when needed. TODO: complete +func SnowflakeConnectLoop() { + for len(snowflakes) < SnowflakeCapacity { + s, err := dialWebRTC() + if err != nil { + snowflakes = append(snowflakes, s) + continue + } + log.Println("WebRTC Error: ", err) + <-time.After(time.Second * ReconnectTimeout) + } +} + // Initialize a WebRTC Connection. func dialWebRTC() (*webRTCConn, error) { // TODO: [#3] Fetch ICE server information from Broker. @@ -82,8 +98,6 @@ func handler(conn *pt.SocksConn) error { defer func() { handlerChan <- -1 }() - defer conn.Close() - log.Println("handler fired:", conn) remote, err := dialWebRTC() if err != nil || remote == nil { @@ -91,6 +105,7 @@ func handler(conn *pt.SocksConn) error { return err } defer remote.Close() + defer conn.Close() webrtcRemote = remote err = conn.Grant(&net.TCPAddr{IP: net.IPv4zero, Port: 0}) diff --git a/client/webrtc.go b/client/webrtc.go index 1300212..55f39e0 100644 --- a/client/webrtc.go +++ b/client/webrtc.go @@ -207,9 +207,6 @@ func (c *webRTCConn) establishDataChannel() error { // Disable the DataChannel as a write destination. log.Println("WebRTC: DataChannel.OnClose [remotely]") c.snowflake = nil - // TODO(issue #12): Need a way to update the circuit so that when a new WebRTC - // data channel is available, the relay actually recognizes the new - // snowflake. c.Reset() } dc.OnMessage = func(msg []byte) {

1 0

[snowflake/master] begin client sub-readme
by arlo＠torproject.org 27 Mar '16

27 Mar '16

commit 2515ddb1fa4fa744cc86d9a72468ba53a1dfc7dd Author: Serene Han <keroserene+git(a)gmail.com> Date: Mon Mar 21 23:18:17 2016 -0700 begin client sub-readme --- client/README.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/client/README.md b/client/README.md new file mode 100644 index 0000000..294d9ee --- /dev/null +++ b/client/README.md @@ -0,0 +1,23 @@ +This is the Tor client component of Snowflake. + +It is based on goptlib. + +### Flags + +The client uses these following `torrc` options by default: +``` +ClientTransportPlugin snowflake exec ./client \ +-url https://snowflake-reg.appspot.com/ \ +-front www.google.com \ +-ice stun:stun.l.google.com:19302 +``` + +`-url` should be the URL of a Broker instance. This is required to have +automated signalling (which is desired in most use cases). +When omitted, the client uses copy-paste signalling instead. + +`-front` is an optional front domain for the Broker request. + +`-ice` is a comma-separated list of ICE servers. These can be STUN or TURN +servers. +

1 0

[snowflake/master] client multiplexing using a webRTCConn channel (#31)
by arlo＠torproject.org 27 Mar '16

27 Mar '16

commit a8ea5e586ebf535994d029d1f78c6dd5cfa2b1e4 Author: Serene Han <keroserene+git(a)gmail.com> Date: Tue Mar 22 18:03:25 2016 -0700 client multiplexing using a webRTCConn channel (#31) --- client/snowflake.go | 42 +++++++++++++++++++++++++++--------------- client/webrtc.go | 2 ++ 2 files changed, 29 insertions(+), 15 deletions(-) diff --git a/client/snowflake.go b/client/snowflake.go index ac593a4..101d26f 100644 --- a/client/snowflake.go +++ b/client/snowflake.go @@ -20,20 +20,20 @@ import ( var ptInfo pt.ClientInfo +const ( + ReconnectTimeout = 5 + SnowflakeCapacity = 1 +) + var brokerURL string var frontDomain string var iceServers IceServerList -var snowflakes []*webRTCConn +var snowflakeChan = make(chan *webRTCConn, 1) // When a connection handler starts, +1 is written to this channel; when it // ends, -1 is written. var handlerChan = make(chan int) -const ( - ReconnectTimeout = 5 - SnowflakeCapacity = 1 -) - func copyLoop(a, b net.Conn) { var wg sync.WaitGroup wg.Add(2) @@ -58,14 +58,22 @@ type SnowflakeChannel interface { // Maintain |WebRTCSlots| number of open connections to // transfer to SOCKS when needed. TODO: complete func SnowflakeConnectLoop() { - for len(snowflakes) < SnowflakeCapacity { + for { + if len(snowflakeChan) >= SnowflakeCapacity { + log.Println("At Capacity: ", len(snowflakeChan), "snowflake. Re-checking in 10s") + <-time.After(time.Second * 10) + continue + } s, err := dialWebRTC() - if err != nil { - snowflakes = append(snowflakes, s) + if nil == s || nil != err { + log.Println("WebRTC Error: ", err, " retrying...") + <-time.After(time.Second * ReconnectTimeout) continue } - log.Println("WebRTC Error: ", err) - <-time.After(time.Second * ReconnectTimeout) + + log.Println("Created a snowflake.") + // TODO: Better handling of multiplex snowflakes. + snowflakeChan <- s } } @@ -99,16 +107,18 @@ func handler(conn *pt.SocksConn) error { handlerChan <- -1 }() - remote, err := dialWebRTC() - if err != nil || remote == nil { + remote, ok := <-snowflakeChan + if remote == nil || !ok { conn.Reject() - return err + return errors.New("handler: Received invalid Snowflake") } defer remote.Close() defer conn.Close() + // TODO: Fix this global webrtcRemote = remote + log.Println("handler: Snowflake assigned.") - err = conn.Grant(&net.TCPAddr{IP: net.IPv4zero, Port: 0}) + err := conn.Grant(&net.TCPAddr{IP: net.IPv4zero, Port: 0}) if err != nil { return err } @@ -201,6 +211,8 @@ func main() { go readSignalingMessages(signalFile) } + go SnowflakeConnectLoop() + ptInfo, err = pt.ClientSetup(nil) if err != nil { log.Fatal(err) diff --git a/client/webrtc.go b/client/webrtc.go index 55f39e0..022a855 100644 --- a/client/webrtc.go +++ b/client/webrtc.go @@ -24,6 +24,7 @@ type webRTCConn struct { writePipe *io.PipeWriter buffer bytes.Buffer reset chan struct{} + active bool *BytesInfo } @@ -86,6 +87,7 @@ func NewWebRTCConnection(config *webrtc.Configuration, // creation & local description setting, which happens asynchronously. connection.errorChannel = make(chan error, 1) connection.reset = make(chan struct{}, 1) + connection.active = false // Log every few seconds. connection.BytesInfo = &BytesInfo{

1 0

[snowflake/master] replace webrtcRemote with webrtcRemotes map & indexing, client multiplexing remotes confirmed working (#31)
by arlo＠torproject.org 27 Mar '16

27 Mar '16

commit 22ace32a713558b6c22fc15eed08d781bf503f60 Author: Serene Han <keroserene+git(a)gmail.com> Date: Wed Mar 23 19:40:26 2016 -0700 replace webrtcRemote with webrtcRemotes map & indexing, client multiplexing remotes confirmed working (#31) --- client/snowflake.go | 31 ++++++++++++------------------- client/webrtc.go | 14 ++++++++------ 2 files changed, 20 insertions(+), 25 deletions(-) diff --git a/client/snowflake.go b/client/snowflake.go index 101d26f..a9f17cc 100644 --- a/client/snowflake.go +++ b/client/snowflake.go @@ -22,7 +22,7 @@ var ptInfo pt.ClientInfo const ( ReconnectTimeout = 5 - SnowflakeCapacity = 1 + SnowflakeCapacity = 3 ) var brokerURL string @@ -55,12 +55,13 @@ type SnowflakeChannel interface { Close() error } -// Maintain |WebRTCSlots| number of open connections to -// transfer to SOCKS when needed. TODO: complete +// Maintain |SnowflakeCapacity| number of available WebRTC connections, to +// transfer to the Tor SOCKS handler when needed. func SnowflakeConnectLoop() { for { - if len(snowflakeChan) >= SnowflakeCapacity { - log.Println("At Capacity: ", len(snowflakeChan), "snowflake. Re-checking in 10s") + numRemotes := len(snowflakeChan) + if numRemotes >= SnowflakeCapacity { + log.Println("At Capacity: ", numRemotes, "snowflake. Re-checking in 10s") <-time.After(time.Second * 10) continue } @@ -70,9 +71,6 @@ func SnowflakeConnectLoop() { <-time.After(time.Second * ReconnectTimeout) continue } - - log.Println("Created a snowflake.") - // TODO: Better handling of multiplex snowflakes. snowflakeChan <- s } } @@ -93,11 +91,9 @@ func dialWebRTC() (*webRTCConn, error) { func endWebRTC() { log.Printf("WebRTC: interruped") - if nil == webrtcRemote { - return + for _, r := range webrtcRemotes { + r.Close() } - webrtcRemote.Close() - webrtcRemote = nil } // Establish a WebRTC channel for SOCKS connections. @@ -106,7 +102,7 @@ func handler(conn *pt.SocksConn) error { defer func() { handlerChan <- -1 }() - + // Wait for an available WebRTC remote... remote, ok := <-snowflakeChan if remote == nil || !ok { conn.Reject() @@ -114,8 +110,6 @@ func handler(conn *pt.SocksConn) error { } defer remote.Close() defer conn.Close() - // TODO: Fix this global - webrtcRemote = remote log.Println("handler: Snowflake assigned.") err := conn.Grant(&net.TCPAddr{IP: net.IPv4zero, Port: 0}) @@ -123,8 +117,6 @@ func handler(conn *pt.SocksConn) error { return err } - // TODO: Make SOCKS acceptance more independent from WebRTC so they can - // be more easily interchanged. go copyLoop(conn, remote) // When WebRTC resets, close the SOCKS connection, which induces new handler. <-remote.reset @@ -164,10 +156,10 @@ func readSignalingMessages(f *os.File) { log.Printf("ignoring invalid signal message %+q", msg) continue } - webrtcRemote.answerChannel <- sdp + webrtcRemotes[0].answerChannel <- sdp } log.Printf("close answerChannel") - close(webrtcRemote.answerChannel) + close(webrtcRemotes[0].answerChannel) if err := s.Err(); err != nil { log.Printf("signal FIFO: %s", err) } @@ -211,6 +203,7 @@ func main() { go readSignalingMessages(signalFile) } + webrtcRemotes = make(map[int]*webRTCConn) go SnowflakeConnectLoop() ptInfo, err = pt.ClientSetup(nil) diff --git a/client/webrtc.go b/client/webrtc.go index 022a855..2b907ad 100644 --- a/client/webrtc.go +++ b/client/webrtc.go @@ -24,11 +24,12 @@ type webRTCConn struct { writePipe *io.PipeWriter buffer bytes.Buffer reset chan struct{} - active bool + index int *BytesInfo } -var webrtcRemote *webRTCConn +var webrtcRemotes map[int]*webRTCConn +var remoteIndex int = 0 func (c *webRTCConn) Read(b []byte) (int, error) { return c.recvPipe.Read(b) @@ -53,6 +54,7 @@ func (c *webRTCConn) Close() error { close(c.offerChannel) close(c.answerChannel) close(c.errorChannel) + delete(webrtcRemotes, c.index) return err } @@ -87,7 +89,6 @@ func NewWebRTCConnection(config *webrtc.Configuration, // creation & local description setting, which happens asynchronously. connection.errorChannel = make(chan error, 1) connection.reset = make(chan struct{}, 1) - connection.active = false // Log every few seconds. connection.BytesInfo = &BytesInfo{ @@ -98,13 +99,14 @@ func NewWebRTCConnection(config *webrtc.Configuration, // Pipes remain the same even when DataChannel gets switched. connection.recvPipe, connection.writePipe = io.Pipe() - + connection.index = remoteIndex + webrtcRemotes[connection.index] = connection + remoteIndex++ return connection } -// TODO: Multiplex. func (c *webRTCConn) Connect() error { - log.Println("Establishing WebRTC connection...") + log.Printf("Establishing WebRTC connection #%d...", c.index) // TODO: When go-webrtc is more stable, it's possible that a new // PeerConnection won't need to be re-prepared each time. err := c.preparePeerConnection()

1 0

[snowflake/master] use webrtcRemotes len for capacity instead of the chan
by arlo＠torproject.org 27 Mar '16

27 Mar '16

commit b2d7586cb47fdd69999cb8df02e303b83b41d51c Author: Serene Han <keroserene+git(a)gmail.com> Date: Fri Mar 25 17:03:38 2016 -0700 use webrtcRemotes len for capacity instead of the chan --- client/snowflake.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/client/snowflake.go b/client/snowflake.go index a9f17cc..fa02178 100644 --- a/client/snowflake.go +++ b/client/snowflake.go @@ -59,7 +59,7 @@ type SnowflakeChannel interface { // transfer to the Tor SOCKS handler when needed. func SnowflakeConnectLoop() { for { - numRemotes := len(snowflakeChan) + numRemotes := len(webrtcRemotes) if numRemotes >= SnowflakeCapacity { log.Println("At Capacity: ", numRemotes, "snowflake. Re-checking in 10s") <-time.After(time.Second * 10)

1 0

[snowflake/master] create and removing from webrtcRemotes test
by arlo＠torproject.org 27 Mar '16

27 Mar '16

commit f024f6b832dc587b60809b2cebf29068a13a3759 Author: Serene Han <keroserene+git(a)gmail.com> Date: Sat Mar 26 18:34:04 2016 -0700 create and removing from webrtcRemotes test --- client/client_test.go | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/client/client_test.go b/client/client_test.go index 5e683ba..1f6f0a6 100644 --- a/client/client_test.go +++ b/client/client_test.go @@ -49,6 +49,7 @@ func (m *MockTransport) RoundTrip(req *http.Request) (*http.Response, error) { func TestConnect(t *testing.T) { Convey("Snowflake", t, func() { + webrtcRemotes = make(map[int]*webRTCConn) Convey("WebRTC Connection", func() { c := new(webRTCConn) @@ -58,6 +59,19 @@ func TestConnect(t *testing.T) { } So(c.buffer.Bytes(), ShouldEqual, nil) + Convey("Create and remove from WebRTCConn set", func() { + So(len(webrtcRemotes), ShouldEqual, 0) + So(remoteIndex, ShouldEqual, 0) + s := NewWebRTCConnection(nil, nil) + So(s, ShouldNotBeNil) + So(s.index, ShouldEqual, 0) + So(len(webrtcRemotes), ShouldEqual, 1) + So(remoteIndex, ShouldEqual, 1) + s.Close() + So(len(webrtcRemotes), ShouldEqual, 0) + So(remoteIndex, ShouldEqual, 1) + }) + Convey("Write buffers when datachannel is nil", func() { c.Write([]byte("test")) c.snowflake = nil

1 0

[translation/torbutton-torbuttonproperties] Update translations for torbutton-torbuttonproperties
by translation＠torproject.org 27 Mar '16

27 Mar '16

commit dbd9f562fc8d01ba7be446983f64ea9adc884c1a Author: Translation commit bot <translation(a)torproject.org> Date: Sun Mar 27 15:15:52 2016 +0000 Update translations for torbutton-torbuttonproperties --- tr/torbutton.properties | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tr/torbutton.properties b/tr/torbutton.properties index e705c1d..6722796 100644 --- a/tr/torbutton.properties +++ b/tr/torbutton.properties @@ -76,4 +76,4 @@ profileProblemTitle=%S Profili Sorunu profileReadOnly=%S salt okunur bir dosya sisteminden çalıştırılamaz. Lütfen kullanmadan önce %S yazılımını farklı bir konuma kopyalayın. profileReadOnlyMac=%S salt okunur bir dosya sisteminden çalıştırılamaz. Lütfen kullanmadan önce %S yazılımını Masaüstü veya Uygulamalar klasörüne kopyalayın. profileAccessDenied=%S, profilinize erişemiyor. Lütfen dosya sistemi izinlerini ayarlayarak yeniden deneyin. -profileMigrationFailed=Migration of your existing %S profile failed.\nNew settings will be used. +profileMigrationFailed=Varolan %S profilinizin taşınma işlemi başarısız oldu.\nYeni ayarlar kullanılacak.

1 0

[stem/master] Explicit non-ascii content validation
by atagar＠torproject.org 26 Mar '16

26 Mar '16

commit 27a654e2e0495c7ac701d792e7db7ed47d4e9753 Author: Damian Johnson <atagar(a)torproject.org> Date: Sat Mar 26 16:14:01 2016 -0700 Explicit non-ascii content validation Few hours ago a relay started publishing a malformed extrainfo descriptor... https://trac.torproject.org/projects/tor/ticket/18656 This is fine, bugs happen. This is why we check for malformed content. But this one cost me a few hours since the non-ascii content then caused DocTor to choke, providing me a useless stacktrace... Traceback (most recent call last): File "./descriptor_checker.py", line 99, in <module> main() File "./descriptor_checker.py", line 57, in main log.warn("Unable to retrieve the %s: %s" % (descriptor_type, query.error)) UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 76: ordinal not in range(128) Instead now it provides... source: http://86.59.21.38:80/tor/extra/all.z time: 03/26/2016 16:30 error: 'dirreq-v3-reqs' line had non-ascii content: S?=4026597208,S?=4026597208,S?=4026597208,S?=4026597208,S?=4026597208,S?=4026597208,??=4026591624,6?=4026537520,6?=4026537520,6?=4026537520,us=8 Non-ascii strings are toxic for systems they're in. They break printing, logging, exception handling, and anything else that touches them unless they're handled specially. Changing Stem to explicitly validate that content is ascii, and provide the user with escaped strings in those exceptions. --- docs/change_log.rst | 1 + stem/descriptor/__init__.py | 11 +++++++- stem/descriptor/hidden_service_descriptor.py | 2 +- stem/descriptor/server_descriptor.py | 2 +- test/integ/process.py | 2 +- .../descriptor/data/extrainfo_nonascii_v3_reqs | 30 ++++++++++++++++++++++ test/unit/descriptor/extrainfo_descriptor.py | 13 ++++++++++ test/unit/descriptor/reader.py | 2 +- 8 files changed, 58 insertions(+), 5 deletions(-) diff --git a/docs/change_log.rst b/docs/change_log.rst index 76143bb..4317307 100644 --- a/docs/change_log.rst +++ b/docs/change_log.rst @@ -73,6 +73,7 @@ The following are only available within Stem's `git repository * Deprecated :func:`~stem.descriptor.remote.DescriptorDownloader.get_microdescriptors` as it was never implemented in tor (:trac:`9271`) * Fixed parsing of server descriptor's *allow-single-hop-exits* and *caches-extra-info* lines * Bracketed IPv6 addresses were mistreated as being invalid content + * Better validation for non-ascii descriptor content * Updated dannenberg's v3ident (:trac:`17906`) * **Utilities** diff --git a/stem/descriptor/__init__.py b/stem/descriptor/__init__.py index dc7c1df..65e6ed4 100644 --- a/stem/descriptor/__init__.py +++ b/stem/descriptor/__init__.py @@ -42,6 +42,7 @@ import copy import hashlib import os import re +import string import tarfile import stem.prereq @@ -777,7 +778,7 @@ def _get_pseudo_pgp_block(remaining_contents): return None -def _get_descriptor_components(raw_contents, validate, extra_keywords = ()): +def _get_descriptor_components(raw_contents, validate, extra_keywords = (), non_ascii_fields = ()): """ Initial breakup of the server descriptor contents to make parsing easier. @@ -796,6 +797,7 @@ def _get_descriptor_components(raw_contents, validate, extra_keywords = ()): True, skips these checks otherwise :param list extra_keywords: entity keywords to put into a separate listing with ordering intact + :param list non_ascii_fields: fields containing non-ascii content :returns: **collections.OrderedDict** with the 'keyword => (value, pgp key) entries' @@ -857,6 +859,13 @@ def _get_descriptor_components(raw_contents, validate, extra_keywords = ()): raise + if validate and keyword not in non_ascii_fields: + try: + value.decode('ascii') + except UnicodeError: + replaced = ''.join([(char if char in string.printable else '?') for char in value]) + raise ValueError("'%s' line had non-ascii content: %s" % (keyword, replaced)) + if keyword in extra_keywords: extra_entries.append('%s %s' % (keyword, value)) else: diff --git a/stem/descriptor/hidden_service_descriptor.py b/stem/descriptor/hidden_service_descriptor.py index de7f0e0..43a3e4f 100644 --- a/stem/descriptor/hidden_service_descriptor.py +++ b/stem/descriptor/hidden_service_descriptor.py @@ -227,7 +227,7 @@ class HiddenServiceDescriptor(Descriptor): def __init__(self, raw_contents, validate = False): super(HiddenServiceDescriptor, self).__init__(raw_contents, lazy_load = not validate) - entries = _get_descriptor_components(raw_contents, validate) + entries = _get_descriptor_components(raw_contents, validate, non_ascii_fields = ('introduction-points')) if validate: for keyword in REQUIRED_FIELDS: diff --git a/stem/descriptor/server_descriptor.py b/stem/descriptor/server_descriptor.py index f096499..5749832 100644 --- a/stem/descriptor/server_descriptor.py +++ b/stem/descriptor/server_descriptor.py @@ -553,7 +553,7 @@ class ServerDescriptor(Descriptor): # influences the resulting exit policy, but for everything else the order # does not matter so breaking it into key / value pairs. - entries, self._unparsed_exit_policy = _get_descriptor_components(stem.util.str_tools._to_unicode(raw_contents), validate, ('accept', 'reject')) + entries, self._unparsed_exit_policy = _get_descriptor_components(stem.util.str_tools._to_unicode(raw_contents), validate, extra_keywords = ('accept', 'reject'), non_ascii_fields = ('contact', 'platform')) if validate: self._parse(entries, validate) diff --git a/test/integ/process.py b/test/integ/process.py index 1cce8c7..d8e6b82 100644 --- a/test/integ/process.py +++ b/test/integ/process.py @@ -382,7 +382,7 @@ class TestProcess(unittest.TestCase): runtime = time.time() - start_time if not (runtime > 2 and runtime < 3): - self.fail('Test should have taken 2-3 seconds, took %i instead' % runtime) + self.fail('Test should have taken 2-3 seconds, took %0.1f instead' % runtime) @require_version(stem.version.Requirement.TAKEOWNERSHIP) @only_run_once diff --git a/test/unit/descriptor/data/extrainfo_nonascii_v3_reqs b/test/unit/descriptor/data/extrainfo_nonascii_v3_reqs new file mode 100644 index 0000000..c69d59e --- /dev/null +++ b/test/unit/descriptor/data/extrainfo_nonascii_v3_reqs @@ -0,0 +1,30 @@ +extra-info newton C871C91489886D5E2E94C13EA1A5FDC4B6DC5204 +identity-ed25519 +-----BEGIN ED25519 CERT----- +AQQABjBoAS6cmIIt3t0HjlInkuk+asDf7yAuTXhg20Gdy4uesKH3AQAgBAC6lLwz +7+yzoMFGDKic6l7kOjyGlV3H01GWtErUWy0PIquP5kMVHnrvdhsBwX0zwSA19Bo+ +tdKPIKrMV+89QBGX2KSa7RRXqIdnGBnGdodglrvMFPGgO/LYHOb9YogFJgA= +-----END ED25519 CERT----- +published 2016-03-26 07:47:53 +write-history 2016-03-26 05:42:08 (14400 s) 682447872,1737454592,1207077888,1278699520,1157439488,635725824 +read-history 2016-03-26 05:42:08 (14400 s) 672482304,1680056320,1198174208,1233503232,1105783808,620540928 +dirreq-write-history 2016-03-26 05:42:08 (14400 s) 1960960,46080,2144256,218112,737280,3273728 +dirreq-read-history 2016-03-26 05:42:08 (14400 s) 322560,13312,323584,8192,46080,440320 +geoip-db-digest 6346E26E2BC96F8511588CE2695E9B0339A75D32 +geoip6-db-digest 43CCB43DBC653D8CC16396A882C5F116A6004F0C +dirreq-stats-end 2016-03-25 19:53:29 (86400 s) +dirreq-v3-ips us=8 +dirreq-v3-reqs Sÿ=4026597208,Sÿ=4026597208,Sÿ=4026597208,Sÿ=4026597208,Sÿ=4026597208,Sÿ=4026597208,¥þ=4026591624,6=4026537520,6=4026537520,6=4026537520,us=8 +dirreq-v3-resp ok=8,not-enough-sigs=0,unavailable=0,not-found=0,not-modified=0,busy=0 +dirreq-v3-direct-dl complete=0,timeout=0,running=0 +dirreq-v3-tunneled-dl complete=8,timeout=4,running=0 +hidserv-stats-end 2016-03-25 19:53:29 (86400 s) +hidserv-rend-relayed-cells 5238 delta_f=2048 epsilon=0.30 bin_size=1024 +hidserv-dir-onions-seen -21 delta_f=8 epsilon=0.30 bin_size=8 +router-sig-ed25519 TeDQDiRfI/k31H+fxciE8q02/ddm/qIi4H7wpQ9A8wRwWBoFUBdwrjZu+pACiglomOuc/I2uSt0++/pmZulZDg +router-signature +-----BEGIN SIGNATURE----- +OVkz0CKUwWSXjhUMEsYATzcfT+D24QR8w6IjUtb6Mcz5P5CMggqmSFhL6SzUFXWS +JwipCWcd2/1NsJ7/JLyXKcG0Ak6FPNP3arweah8EwWkVo7rvscagI+u+Uq2FqtO/ +OTNjnriDWTZdSA9FOcQueBZi7Qf7xwA0LEjtadtCjUI= +-----END SIGNATURE----- diff --git a/test/unit/descriptor/extrainfo_descriptor.py b/test/unit/descriptor/extrainfo_descriptor.py index fa68e62..ce898d3 100644 --- a/test/unit/descriptor/extrainfo_descriptor.py +++ b/test/unit/descriptor/extrainfo_descriptor.py @@ -166,6 +166,19 @@ k0d2aofcVbHr4fPQOSST0LXDrhFl5Fqo5um296zpJGvRUeO6S44U/EfJAGShtqWw self.assertEqual('7DSOQz9eGgjDX6GT7qcrVViK8yqJD4aoEnuhdAgYtgA', desc.router_digest_sha256) self.assertEqual([], desc.get_unrecognized_lines()) + def test_nonascii_v3_reqs(self): + """ + Malformed descriptor with non-ascii content for the 'dirreq-v3-reqs' line. + """ + + with open(get_resource('extrainfo_nonascii_v3_reqs'), 'rb') as descriptor_file: + try: + next(stem.descriptor.parse_file(descriptor_file, 'extra-info 1.0', validate = True)) + self.fail("validation should've raised an exception") + except ValueError as exc: + expected = "'dirreq-v3-reqs' line had non-ascii content: S?=4026597208,S?=4026597208,S?=4026597208,S?=4026597208,S?=4026597208,S?=4026597208,??=4026591624,6?=4026537520,6?=4026537520,6?=4026537520,us=8" + self.assertEqual(expected, str(exc)) + def test_minimal_extrainfo_descriptor(self): """ Basic sanity check that we can parse an extrainfo descriptor with minimal diff --git a/test/unit/descriptor/reader.py b/test/unit/descriptor/reader.py index 8f617f9..f21ea75 100644 --- a/test/unit/descriptor/reader.py +++ b/test/unit/descriptor/reader.py @@ -466,7 +466,7 @@ class TestDescriptorReader(unittest.TestCase): reader = stem.descriptor.reader.DescriptorReader(DESCRIPTOR_TEST_DATA) reader.register_skip_listener(skip_listener.listener) - expected_skip_files = ('riddle', 'tiny.png', 'vote', 'new_metrics_type', 'cached-microdesc-consensus_with_carriage_returns') + expected_skip_files = ('riddle', 'tiny.png', 'vote', 'new_metrics_type', 'cached-microdesc-consensus_with_carriage_returns', 'extrainfo_nonascii_v3_reqs') with reader: list(reader) # iterates over all of the descriptors

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-commits March 2016