August 2021 - tor-commits - lists.torproject.org

[snowflake/main] Document /amp/client in broker-spec.txt.
by dcf＠torproject.org 05 Aug '21

05 Aug '21

commit f2dc41d77891816b3f6aec78cf9491fad6999388 Author: David Fifield <david(a)bamsoftware.com> Date: Mon Jul 26 10:23:12 2021 -0600 Document /amp/client in broker-spec.txt. --- doc/broker-spec.txt | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/doc/broker-spec.txt b/doc/broker-spec.txt index f2cd231..f25be79 100644 --- a/doc/broker-spec.txt +++ b/doc/broker-spec.txt @@ -107,6 +107,11 @@ through the exchange of WebRTC SDP information with its endpoints. 2.1. Client interactions with the broker +The broker offers multiple ways for clients to exchange registration +messages. + +2.1.1. HTTPS POST + Clients interact with the broker by making a POST request to `/client` with the offer SDP in the request body: ``` @@ -130,6 +135,38 @@ If no proxies were available, they receive a 503 status code: HTTP 503 Service Unavailable ``` +2.1.2. AMP + +The broker's /amp/client endpoint receives client poll messages encoded +into the URL path, and sends client poll responses encoded as HTML that +conforms to the requirements of AMP (Accelerated Mobile Pages). This +endpoint is intended to be accessed through an AMP cache, using the +-ampcache option of snowflake-client. + +The client encodes its poll message into a GET request as follows: +``` +GET /amp/client/0[0 or more bytes]/[base64 of client poll message] +``` +The components of the path are as follows: +* "/amp/client/", the root of the endpoint. +* "0", a format version number, which controls the interpretation of the + rest of the path. Only the first byte matters as a version indicator + (not the whole first path component). +* Any number of slash or non-slash bytes. These may be used as padding + or to prevent cache collisions in the AMP cache. +* A final slash. +* base64 encoding of the client poll message, using the URL-safe + alphabet (which does not include slash). + +The broker returns a client poll response message in the HTTP response. +The message is encoded using AMP armor, an AMP-compatible HTML encoding. +The data stream is notionally a "0" byte (a format version indicator) +followed by the base64 encoding of the message (using the standard +alphabet, with "=" padding). This stream is broken into +whitespace-separated chunks, which are then bundled into HTML <pre> +elements. The <pre> elements are then surrounded by AMP boilerplate. To +decode, search the HTML for <pre> elements, concatenate their contents +and join on whitespace, discard the "0" prefix, and base64 decode. 2.2 Proxy interactions with the broker

1 0

[snowflake/main] Add info about rendezvous methods to client README.
by dcf＠torproject.org 05 Aug '21

05 Aug '21

commit 521eb4d4d6d76a1d57d3c8fc5c3a8261c171ea4e Author: David Fifield <david(a)bamsoftware.com> Date: Mon Jul 19 09:01:17 2021 -0600 Add info about rendezvous methods to client README. --- client/README.md | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/client/README.md b/client/README.md index aed11c3..0680408 100644 --- a/client/README.md +++ b/client/README.md @@ -52,3 +52,59 @@ To bootstrap Tor, run: tor -f torrc ``` This should start the client plugin, bootstrapping to 100% using WebRTC. + +### Registration methods + +The Snowflake client supports a few different ways of communicating with the broker. +This initial step is sometimes called rendezvous. + +#### Domain fronting HTTPS + +For domain fronting rendezvous, use the `-url` and `-front` command-line options together. +[Domain fronting](https://www.bamsoftware.com/papers/fronting/) +hides the externally visible domain name from an external observer, +making it appear that the Snowflake client is communicating with some server +other than the Snowflake broker. + +* `-url` is the HTTPS URL of a forwarder to the broker, on some service that supports domain fronting, such as a CDN. +* `-front` is the domain name to show externally. It must be another domain on the same service. + +Example: +``` +-url https://snowflake-broker.torproject.net.global.prod.fastly.net/ \ +-front cdn.sstatic.net \ +``` + +#### AMP cache + +For AMP cache rendezvous, use the `-url`, `-ampcache`, and `-front` command-line options together. +[AMP](https://amp.dev/documentation/) is a standard for web pages for mobile computers. +An [AMP cache](https://amp.dev/documentation/guides-and-tutorials/learn/amp-caches-and-cors/how_amp_pages_are_cached/) +is a cache and proxy specialized for AMP pages. +The Snowflake broker has the ability to make its client registration responses look like AMP pages, +so it can be accessed through an AMP cache. +When you use AMP cache rendezvous, it appears to an observer that the Snowflake client +is accessing an AMP cache, or some other domain operated by the same organization. +You still need to use the `-front` command-line option, because the +[format of AMP cache URLs](https://amp.dev/documentation/guides-and-tutorials/learn/amp-caches-and-cors/amp-cache-urls/) +would otherwise reveal the domain name of the broker. + +There is only one AMP cache that works with this option, +the Google AMP cache at https://cdn.ampproject.org/. + +* `-url` is the HTTPS URL of the broker. +* `-ampcache` is `https://cdn.ampproject.org/`. +* `-front` is any Google domain, such as `www.google.com`. + +Example: +``` +-url https://snowflake-broker.torproject.net/ \ +-ampcache https://cdn.ampproject.org/ \ +-front www.google.com \ +``` + +#### Direct access + +It is also possible to access the broker directly using HTTPS, without domain fronting, +for testing purposes. This mode is not suitable for circumvention, because the +broker is easily blocked by its address.

1 0

[snowflake/main] Broker /amp/client route (AMP cache client registration).
by dcf＠torproject.org 05 Aug '21

05 Aug '21

commit e833119befa052e4837fe147f8bc2766a4ca7c54 Author: David Fifield <david(a)bamsoftware.com> Date: Sun Jul 18 23:37:41 2021 -0600 Broker /amp/client route (AMP cache client registration). --- broker/amp.go | 76 +++++++++++++++++++++++++++++++++++++++ broker/broker.go | 2 ++ broker/snowflake-broker_test.go | 78 +++++++++++++++++++++++++++++++++++++++-- 3 files changed, 154 insertions(+), 2 deletions(-) diff --git a/broker/amp.go b/broker/amp.go new file mode 100644 index 0000000..8641e51 --- /dev/null +++ b/broker/amp.go @@ -0,0 +1,76 @@ +package main + +import ( + "log" + "net/http" + "strings" + + "git.torproject.org/pluggable-transports/snowflake.git/common/amp" + "git.torproject.org/pluggable-transports/snowflake.git/common/messages" +) + +// ampClientOffers is the AMP-speaking endpoint for client poll messages, +// intended for access via an AMP cache. In contrast to the other clientOffers, +// the client's encoded poll message is stored in the URL path rather than the +// HTTP request body (because an AMP cache does not support POST), and the +// encoded client poll response is sent back as AMP-armored HTML. +func ampClientOffers(i *IPC, w http.ResponseWriter, r *http.Request) { + // The encoded client poll message immediately follows the /amp/client/ + // path prefix, so this function unfortunately needs to be aware of and + // remote its own routing prefix. + path := strings.TrimPrefix(r.URL.Path, "/amp/client/") + if path == r.URL.Path { + // The path didn't start with the expected prefix. This probably + // indicates an internal bug. + log.Println("ampClientOffers: unexpected prefix in path") + w.WriteHeader(http.StatusInternalServerError) + return + } + + var encPollReq []byte + var response []byte + var err error + + encPollReq, err = amp.DecodePath(path) + if err == nil { + arg := messages.Arg{ + Body: encPollReq, + RemoteAddr: "", + } + err = i.ClientOffers(arg, &response) + } else { + response, err = (&messages.ClientPollResponse{ + Error: "cannot decode URL path", + }).EncodePollResponse() + } + + if err != nil { + // We couldn't even construct a JSON object containing an error + // message :( Nothing to do but signal an error at the HTTP + // layer. The AMP cache will translate this 500 status into a + // 404 status. + // https://amp.dev/documentation/guides-and-tutorials/learn/amp-caches-and-cor… + log.Printf("ampClientOffers: %v", err) + w.WriteHeader(http.StatusInternalServerError) + return + } + + w.Header().Set("Content-Type", "text/html") + // Attempt to hint to an AMP cache not to waste resources caching this + // document. "The Google AMP Cache considers any document fresh for at + // least 15 seconds." + // https://developers.google.com/amp/cache/overview#google-amp-cache-updates + w.Header().Set("Cache-Control", "max-age=15") + w.WriteHeader(http.StatusOK) + + enc, err := amp.NewArmorEncoder(w) + if err != nil { + log.Printf("amp.NewArmorEncoder: %v", err) + return + } + defer enc.Close() + + if _, err := enc.Write(response); err != nil { + log.Printf("ampClientOffers: unable to write answer: %v", err) + } +} diff --git a/broker/broker.go b/broker/broker.go index 437a4d1..6c855f3 100644 --- a/broker/broker.go +++ b/broker/broker.go @@ -218,6 +218,8 @@ func main() { http.Handle("/metrics", MetricsHandler{metricsFilename, metricsHandler}) http.Handle("/prometheus", promhttp.HandlerFor(ctx.metrics.promMetrics.registry, promhttp.HandlerOpts{})) + http.Handle("/amp/client/", SnowflakeHandler{i, ampClientOffers}) + server := http.Server{ Addr: addr, } diff --git a/broker/snowflake-broker_test.go b/broker/snowflake-broker_test.go index 9e1c9f1..233cfea 100644 --- a/broker/snowflake-broker_test.go +++ b/broker/snowflake-broker_test.go @@ -3,6 +3,7 @@ package main import ( "bytes" "container/heap" + "io" "io/ioutil" "log" "net" @@ -13,6 +14,7 @@ import ( "testing" "time" + "git.torproject.org/pluggable-transports/snowflake.git/common/amp" . "github.com/smartystreets/goconvey/convey" ) @@ -24,6 +26,15 @@ func NullLogger() *log.Logger { var promOnce sync.Once +func decodeAMPArmorToString(r io.Reader) (string, error) { + dec, err := amp.NewArmorDecoder(r) + if err != nil { + return "", err + } + p, err := ioutil.ReadAll(dec) + return string(p), err +} + func TestBroker(t *testing.T) { Convey("Context", t, func() { @@ -69,7 +80,7 @@ func TestBroker(t *testing.T) { So(offer.sdp, ShouldResemble, []byte("test offer")) }) - Convey("Responds to client offers...", func() { + Convey("Responds to HTTP client offers...", func() { w := httptest.NewRecorder() data := bytes.NewReader( []byte("1.0\n{\"offer\": \"fake\", \"nat\": \"unknown\"}")) @@ -117,7 +128,7 @@ func TestBroker(t *testing.T) { }) }) - Convey("Responds to legacy client offers...", func() { + Convey("Responds to HTTP legacy client offers...", func() { w := httptest.NewRecorder() data := bytes.NewReader([]byte("{test}")) r, err := http.NewRequest("POST", "snowflake.broker/client", data) @@ -165,6 +176,69 @@ func TestBroker(t *testing.T) { }) + Convey("Responds to AMP client offers...", func() { + w := httptest.NewRecorder() + encPollReq := []byte("1.0\n{\"offer\": \"fake\", \"nat\": \"unknown\"}") + r, err := http.NewRequest("GET", "/amp/client/"+amp.EncodePath(encPollReq), nil) + So(err, ShouldBeNil) + + Convey("with status 200 when request is badly formatted.", func() { + r, err := http.NewRequest("GET", "/amp/client/bad", nil) + So(err, ShouldBeNil) + ampClientOffers(i, w, r) + body, err := decodeAMPArmorToString(w.Body) + So(err, ShouldBeNil) + So(body, ShouldEqual, `{"error":"cannot decode URL path"}`) + }) + + Convey("with error when no snowflakes are available.", func() { + ampClientOffers(i, w, r) + So(w.Code, ShouldEqual, http.StatusOK) + body, err := decodeAMPArmorToString(w.Body) + So(err, ShouldBeNil) + So(body, ShouldEqual, `{"error":"no snowflake proxies currently available"}`) + }) + + Convey("with a proxy answer if available.", func() { + done := make(chan bool) + // Prepare a fake proxy to respond with. + snowflake := ctx.AddSnowflake("fake", "", NATUnrestricted, 0) + go func() { + ampClientOffers(i, w, r) + done <- true + }() + offer := <-snowflake.offerChannel + So(offer.sdp, ShouldResemble, []byte("fake")) + snowflake.answerChannel <- "fake answer" + <-done + body, err := decodeAMPArmorToString(w.Body) + So(err, ShouldBeNil) + So(body, ShouldEqual, `{"answer":"fake answer"}`) + So(w.Code, ShouldEqual, http.StatusOK) + }) + + Convey("Times out when no proxy responds.", func() { + if testing.Short() { + return + } + done := make(chan bool) + snowflake := ctx.AddSnowflake("fake", "", NATUnrestricted, 0) + go func() { + ampClientOffers(i, w, r) + // Takes a few seconds here... + done <- true + }() + offer := <-snowflake.offerChannel + So(offer.sdp, ShouldResemble, []byte("fake")) + <-done + So(w.Code, ShouldEqual, http.StatusOK) + body, err := decodeAMPArmorToString(w.Body) + So(err, ShouldBeNil) + So(body, ShouldEqual, `{"error":"timed out waiting for answer!"}`) + }) + + }) + Convey("Responds to proxy polls...", func() { done := make(chan bool) w := httptest.NewRecorder()

1 0

[snowflake/main] Factor out httpRendezvous separate from BrokerChannel.
by dcf＠torproject.org 05 Aug '21

05 Aug '21

commit 0f34a7778fa1f4c28c7cc161991080d146689591 Author: David Fifield <david(a)bamsoftware.com> Date: Sun Jul 18 12:36:16 2021 -0600 Factor out httpRendezvous separate from BrokerChannel. Makes BrokerChannel abstract over a rendezvousMethod. BrokerChannel itself is responsible for keepLocalAddresses and the NAT type state, as well as encoding and decoding client poll messages. rendezvousMethod is only responsible for delivery of encoded messages. --- client/lib/lib_test.go | 93 +-------------------------- client/lib/rendezvous.go | 99 ++++++++++------------------ client/lib/rendezvous_http.go | 77 ++++++++++++++++++++++ client/lib/rendezvous_test.go | 145 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 258 insertions(+), 156 deletions(-) diff --git a/client/lib/lib_test.go b/client/lib/lib_test.go index 9087eed..86601b1 100644 --- a/client/lib/lib_test.go +++ b/client/lib/lib_test.go @@ -1,33 +1,14 @@ package lib import ( - "bytes" "fmt" - "io/ioutil" "net" - "net/http" "testing" "time" - "git.torproject.org/pluggable-transports/snowflake.git/common/util" . "github.com/smartystreets/goconvey/convey" ) -type MockTransport struct { - statusOverride int - body []byte -} - -// Just returns a response with fake SDP answer. -func (m *MockTransport) RoundTrip(req *http.Request) (*http.Response, error) { - s := ioutil.NopCloser(bytes.NewReader(m.body)) - r := &http.Response{ - StatusCode: m.statusOverride, - Body: s, - } - return r, nil -} - type FakeDialer struct { max int } @@ -172,11 +153,10 @@ func TestSnowflakeClient(t *testing.T) { Convey("Dialers", t, func() { Convey("Can construct WebRTCDialer.", func() { - broker := &BrokerChannel{front: "test"} + broker := &BrokerChannel{} d := NewWebRTCDialer(broker, nil, 1) So(d, ShouldNotBeNil) So(d.BrokerChannel, ShouldNotBeNil) - So(d.BrokerChannel.front, ShouldEqual, "test") }) SkipConvey("WebRTCDialer can Catch a snowflake.", func() { broker := &BrokerChannel{} @@ -187,77 +167,6 @@ func TestSnowflakeClient(t *testing.T) { }) }) - Convey("Rendezvous", t, func() { - transport := &MockTransport{ - http.StatusOK, - []byte(`{"answer": "{\"type\":\"answer\",\"sdp\":\"fake\"}" }`), - } - fakeOffer, err := util.DeserializeSessionDescription(`{"type":"offer","sdp":"test"}`) - if err != nil { - panic(err) - } - - Convey("Construct BrokerChannel with no front domain", func() { - b, err := NewBrokerChannel("http://test.broker", "", transport, false) - So(b.url, ShouldNotBeNil) - So(err, ShouldBeNil) - So(b.url.Host, ShouldResemble, "test.broker") - So(b.front, ShouldResemble, "") - So(b.transport, ShouldNotBeNil) - }) - - Convey("Construct BrokerChannel *with* front domain", func() { - b, err := NewBrokerChannel("http://test.broker", "front", transport, false) - So(b.url, ShouldNotBeNil) - So(err, ShouldBeNil) - So(b.url.Host, ShouldResemble, "test.broker") - So(b.front, ShouldResemble, "front") - So(b.transport, ShouldNotBeNil) - }) - - Convey("BrokerChannel.Negotiate responds with answer", func() { - b, err := NewBrokerChannel("http://test.broker", "", transport, false) - So(err, ShouldBeNil) - answer, err := b.Negotiate(fakeOffer) - So(err, ShouldBeNil) - So(answer, ShouldNotBeNil) - So(answer.SDP, ShouldResemble, "fake") - }) - - Convey("BrokerChannel.Negotiate fails", func() { - b, err := NewBrokerChannel("http://test.broker", "", - &MockTransport{http.StatusOK, []byte(`{"error": "no snowflake proxies currently available"}`)}, - false) - So(err, ShouldBeNil) - answer, err := b.Negotiate(fakeOffer) - So(err, ShouldNotBeNil) - So(answer, ShouldBeNil) - }) - - Convey("BrokerChannel.Negotiate fails with unexpected error", func() { - b, err := NewBrokerChannel("http://test.broker", "", - &MockTransport{http.StatusInternalServerError, []byte("\n")}, - false) - So(err, ShouldBeNil) - answer, err := b.Negotiate(fakeOffer) - So(err, ShouldNotBeNil) - So(answer, ShouldBeNil) - So(err.Error(), ShouldResemble, BrokerErrorUnexpected) - }) - - Convey("BrokerChannel.Negotiate fails with large read", func() { - b, err := NewBrokerChannel("http://test.broker", "", - &MockTransport{http.StatusOK, make([]byte, readLimit+1)}, - false) - So(err, ShouldBeNil) - answer, err := b.Negotiate(fakeOffer) - So(err, ShouldNotBeNil) - So(answer, ShouldBeNil) - So(err.Error(), ShouldResemble, "unexpected EOF") - }) - - }) - } func TestWebRTCPeer(t *testing.T) { diff --git a/client/lib/rendezvous.go b/client/lib/rendezvous.go index caa4ae4..8568120 100644 --- a/client/lib/rendezvous.go +++ b/client/lib/rendezvous.go @@ -9,13 +9,9 @@ package lib import ( - "bytes" "errors" - "io" - "io/ioutil" "log" "net/http" - "net/url" "sync" "time" @@ -30,11 +26,21 @@ const ( readLimit = 100000 //Maximum number of bytes to be read from an HTTP response ) -// Signalling Channel to the Broker. +// rendezvousMethod represents a way of communicating with the broker: sending +// an encoded client poll request (SDP offer) and receiving an encoded client +// poll response (SDP answer) in return. rendezvousMethod is used by +// BrokerChannel, which is in charge of encoding and decoding, and all other +// tasks that are independent of the rendezvous method. +type rendezvousMethod interface { + Exchange([]byte) ([]byte, error) +} + +// BrokerChannel contains a rendezvousMethod, as well as data that is not +// specific to any rendezvousMethod. BrokerChannel has the responsibility of +// encoding and decoding SDP offers and answers; rendezvousMethod is responsible +// for the exchange of encoded information. type BrokerChannel struct { - url *url.URL - front string // Optional front domain to replace url.Host in requests. - transport http.RoundTripper // Used to make all requests. + rendezvous rendezvousMethod keepLocalAddresses bool NATType string lock sync.Mutex @@ -54,31 +60,21 @@ func CreateBrokerTransport() http.RoundTripper { // |broker| is the full URL of the facilitating program which assigns proxies // to clients, and |front| is the option fronting domain. func NewBrokerChannel(broker string, front string, transport http.RoundTripper, keepLocalAddresses bool) (*BrokerChannel, error) { - targetURL, err := url.Parse(broker) - if err != nil { - return nil, err - } log.Println("Rendezvous using Broker at:", broker) if front != "" { log.Println("Domain fronting using:", front) } - bc := new(BrokerChannel) - bc.url = targetURL - bc.front = front - bc.transport = transport - bc.keepLocalAddresses = keepLocalAddresses - bc.NATType = nat.NATUnknown - return bc, nil -} -func limitedRead(r io.Reader, limit int64) ([]byte, error) { - p, err := ioutil.ReadAll(&io.LimitedReader{R: r, N: limit + 1}) + rendezvous, err := newHTTPRendezvous(broker, front, transport) if err != nil { - return p, err - } else if int64(len(p)) == limit+1 { - return p[0:limit], io.ErrUnexpectedEOF + return nil, err } - return p, err + + return &BrokerChannel{ + rendezvous: rendezvous, + keepLocalAddresses: keepLocalAddresses, + NATType: nat.NATUnknown, + }, nil } // Roundtrip HTTP POST using WebRTC SessionDescriptions. @@ -87,8 +83,6 @@ func limitedRead(r io.Reader, limit int64) ([]byte, error) { // with an SDP answer from a designated remote WebRTC peer. func (bc *BrokerChannel) Negotiate(offer *webrtc.SessionDescription) ( *webrtc.SessionDescription, error) { - log.Println("Negotiating via BrokerChannel...\nTarget URL: ", - bc.url.Host, "\nFront URL: ", bc.front) // Ideally, we could specify an `RTCIceTransportPolicy` that would handle // this for us. However, "public" was removed from the draft spec. // See https://developer.mozilla.org/en-US/docs/Web/API/RTCConfiguration#RTCIceTra… @@ -103,57 +97,34 @@ func (bc *BrokerChannel) Negotiate(offer *webrtc.SessionDescription) ( return nil, err } - // Encode client poll request + // Encode the client poll request. bc.lock.Lock() req := &messages.ClientPollRequest{ Offer: offerSDP, NAT: bc.NATType, } - body, err := req.EncodePollRequest() + encReq, err := req.EncodePollRequest() bc.lock.Unlock() if err != nil { return nil, err } - data := bytes.NewReader([]byte(body)) - // Suffix with broker's client registration handler. - clientURL := bc.url.ResolveReference(&url.URL{Path: "client"}) - request, err := http.NewRequest("POST", clientURL.String(), data) - if nil != err { + // Do the exchange using our rendezvousMethod. + encResp, err := bc.rendezvous.Exchange(encReq) + if err != nil { return nil, err } - if bc.front != "" { - // Do domain fronting. Replace the domain in the URL's with the - // front, and store the original domain the HTTP Host header. - request.Host = request.URL.Host - request.URL.Host = bc.front - } - resp, err := bc.transport.RoundTrip(request) - if nil != err { + log.Printf("Received answer: %s", string(encResp)) + + // Decode the client poll response. + resp, err := messages.DecodeClientPollResponse(encResp) + if err != nil { return nil, err } - defer resp.Body.Close() - log.Printf("BrokerChannel Response:\n%s\n\n", resp.Status) - - switch resp.StatusCode { - case http.StatusOK: - body, err := limitedRead(resp.Body, readLimit) - if nil != err { - return nil, err - } - log.Printf("Received answer: %s", string(body)) - - resp, err := messages.DecodeClientPollResponse(body) - if err != nil { - return nil, err - } - if resp.Error != "" { - return nil, errors.New(resp.Error) - } - return util.DeserializeSessionDescription(resp.Answer) - default: - return nil, errors.New(BrokerErrorUnexpected) + if resp.Error != "" { + return nil, errors.New(resp.Error) } + return util.DeserializeSessionDescription(resp.Answer) } func (bc *BrokerChannel) SetNATType(NATType string) { diff --git a/client/lib/rendezvous_http.go b/client/lib/rendezvous_http.go new file mode 100644 index 0000000..01219cb --- /dev/null +++ b/client/lib/rendezvous_http.go @@ -0,0 +1,77 @@ +package lib + +import ( + "bytes" + "errors" + "io" + "io/ioutil" + "log" + "net/http" + "net/url" +) + +// httpRendezvous is a rendezvousMethod that communicates with the .../client +// route of the broker over HTTP or HTTPS, with optional domain fronting. +type httpRendezvous struct { + brokerURL *url.URL + front string // Optional front domain to replace url.Host in requests. + transport http.RoundTripper // Used to make all requests. +} + +// newHTTPRendezvous creates a new httpRendezvous that contacts the broker at +// the given URL, with an optional front domain. transport is the +// http.RoundTripper used to make all requests. +func newHTTPRendezvous(broker, front string, transport http.RoundTripper) (*httpRendezvous, error) { + brokerURL, err := url.Parse(broker) + if err != nil { + return nil, err + } + return &httpRendezvous{ + brokerURL: brokerURL, + front: front, + transport: transport, + }, nil +} + +func (r *httpRendezvous) Exchange(encPollReq []byte) ([]byte, error) { + log.Println("Negotiating via HTTP rendezvous...") + log.Println("Target URL: ", r.brokerURL.Host) + log.Println("Front URL: ", r.front) + + // Suffix the path with the broker's client registration handler. + reqURL := r.brokerURL.ResolveReference(&url.URL{Path: "client"}) + req, err := http.NewRequest("POST", reqURL.String(), bytes.NewReader(encPollReq)) + if err != nil { + return nil, err + } + + if r.front != "" { + // Do domain fronting. Replace the domain in the URL's with the + // front, and store the original domain the HTTP Host header. + req.Host = req.URL.Host + req.URL.Host = r.front + } + + resp, err := r.transport.RoundTrip(req) + if err != nil { + return nil, err + } + defer resp.Body.Close() + + log.Printf("HTTP rendezvous response: %s", resp.Status) + if resp.StatusCode != http.StatusOK { + return nil, errors.New(BrokerErrorUnexpected) + } + + return limitedRead(resp.Body, readLimit) +} + +func limitedRead(r io.Reader, limit int64) ([]byte, error) { + p, err := ioutil.ReadAll(&io.LimitedReader{R: r, N: limit + 1}) + if err != nil { + return p, err + } else if int64(len(p)) == limit+1 { + return p[0:limit], io.ErrUnexpectedEOF + } + return p, err +} diff --git a/client/lib/rendezvous_test.go b/client/lib/rendezvous_test.go new file mode 100644 index 0000000..c263e37 --- /dev/null +++ b/client/lib/rendezvous_test.go @@ -0,0 +1,145 @@ +package lib + +import ( + "bytes" + "errors" + "fmt" + "io" + "io/ioutil" + "net/http" + "testing" + + "git.torproject.org/pluggable-transports/snowflake.git/common/messages" + "git.torproject.org/pluggable-transports/snowflake.git/common/nat" + . "github.com/smartystreets/goconvey/convey" +) + +// mockTransport's RoundTrip method returns a response with a fake status and +// body. +type mockTransport struct { + statusCode int + body []byte +} + +func (t *mockTransport) RoundTrip(req *http.Request) (*http.Response, error) { + return &http.Response{ + Status: fmt.Sprintf("%d %s", t.statusCode, http.StatusText(t.statusCode)), + StatusCode: t.statusCode, + Body: ioutil.NopCloser(bytes.NewReader(t.body)), + }, nil +} + +// errorTransport's RoundTrip method returns an error. +type errorTransport struct { + err error +} + +func (t errorTransport) RoundTrip(req *http.Request) (*http.Response, error) { + return nil, t.err +} + +// makeEncPollReq returns an encoded client poll request containing a given +// offer. +func makeEncPollReq(offer string) []byte { + encPollReq, err := (&messages.ClientPollRequest{ + Offer: offer, + NAT: nat.NATUnknown, + }).EncodePollRequest() + if err != nil { + panic(err) + } + return encPollReq +} + +// makeEncPollResp returns an encoded client poll response with given answer and +// error strings. +func makeEncPollResp(answer, errorStr string) []byte { + encPollResp, err := (&messages.ClientPollResponse{ + Answer: answer, + Error: errorStr, + }).EncodePollResponse() + if err != nil { + panic(err) + } + return encPollResp +} + +func TestHTTPRendezvous(t *testing.T) { + Convey("HTTP rendezvous", t, func() { + Convey("Construct httpRendezvous with no front domain", func() { + transport := &mockTransport{http.StatusOK, []byte{}} + rend, err := newHTTPRendezvous("http://test.broker", "", transport) + So(err, ShouldBeNil) + So(rend.brokerURL, ShouldNotBeNil) + So(rend.brokerURL.Host, ShouldResemble, "test.broker") + So(rend.front, ShouldResemble, "") + So(rend.transport, ShouldEqual, transport) + }) + + Convey("Construct httpRendezvous *with* front domain", func() { + transport := &mockTransport{http.StatusOK, []byte{}} + rend, err := newHTTPRendezvous("http://test.broker", "front", transport) + So(err, ShouldBeNil) + So(rend.brokerURL, ShouldNotBeNil) + So(rend.brokerURL.Host, ShouldResemble, "test.broker") + So(rend.front, ShouldResemble, "front") + So(rend.transport, ShouldEqual, transport) + }) + + fakeEncPollReq := makeEncPollReq(`{"type":"offer","sdp":"test"}`) + + Convey("httpRendezvous.Exchange responds with answer", func() { + fakeEncPollResp := makeEncPollResp( + `{"answer": "{\"type\":\"answer\",\"sdp\":\"fake\"}" }`, + "", + ) + rend, err := newHTTPRendezvous("http://test.broker", "", + &mockTransport{http.StatusOK, fakeEncPollResp}) + So(err, ShouldBeNil) + answer, err := rend.Exchange(fakeEncPollReq) + So(err, ShouldBeNil) + So(answer, ShouldResemble, fakeEncPollResp) + }) + + Convey("httpRendezvous.Exchange responds with no answer", func() { + fakeEncPollResp := makeEncPollResp( + "", + `{"error": "no snowflake proxies currently available"}`, + ) + rend, err := newHTTPRendezvous("http://test.broker", "", + &mockTransport{http.StatusOK, fakeEncPollResp}) + So(err, ShouldBeNil) + answer, err := rend.Exchange(fakeEncPollReq) + So(err, ShouldBeNil) + So(answer, ShouldResemble, fakeEncPollResp) + }) + + Convey("httpRendezvous.Exchange fails with unexpected HTTP status code", func() { + rend, err := newHTTPRendezvous("http://test.broker", "", + &mockTransport{http.StatusInternalServerError, []byte{}}) + So(err, ShouldBeNil) + answer, err := rend.Exchange(fakeEncPollReq) + So(err, ShouldNotBeNil) + So(answer, ShouldBeNil) + So(err.Error(), ShouldResemble, BrokerErrorUnexpected) + }) + + Convey("httpRendezvous.Exchange fails with error", func() { + transportErr := errors.New("error") + rend, err := newHTTPRendezvous("http://test.broker", "", + &errorTransport{err: transportErr}) + So(err, ShouldBeNil) + answer, err := rend.Exchange(fakeEncPollReq) + So(err, ShouldEqual, transportErr) + So(answer, ShouldBeNil) + }) + + Convey("httpRendezvous.Exchange fails with large read", func() { + rend, err := newHTTPRendezvous("http://test.broker", "", + &mockTransport{http.StatusOK, make([]byte, readLimit+1)}) + So(err, ShouldBeNil) + _, err = rend.Exchange(fakeEncPollReq) + So(err, ShouldEqual, io.ErrUnexpectedEOF) + }) + }) +}

1 0

[snowflake/main] amp package.
by dcf＠torproject.org 05 Aug '21

05 Aug '21

commit c9e0dd287f30b2acb0145a7efc326c881792138a Author: David Fifield <david(a)bamsoftware.com> Date: Sun Jul 18 15:22:03 2021 -0600 amp package. This package contains a CacheURL function that modifies a URL to be accessed through an AMP cache, and the "AMP armor" data encoding scheme for encoding data into the AMP subset of HTML. --- common/amp/armor_decoder.go | 136 +++++++++++++++++++ common/amp/armor_encoder.go | 176 ++++++++++++++++++++++++ common/amp/armor_test.go | 227 +++++++++++++++++++++++++++++++ common/amp/cache.go | 178 ++++++++++++++++++++++++ common/amp/cache_test.go | 320 ++++++++++++++++++++++++++++++++++++++++++++ common/amp/doc.go | 88 ++++++++++++ common/amp/path.go | 44 ++++++ common/amp/path_test.go | 54 ++++++++ 8 files changed, 1223 insertions(+) diff --git a/common/amp/armor_decoder.go b/common/amp/armor_decoder.go new file mode 100644 index 0000000..fed44a6 --- /dev/null +++ b/common/amp/armor_decoder.go @@ -0,0 +1,136 @@ +package amp + +import ( + "bufio" + "bytes" + "encoding/base64" + "fmt" + "io" + + "golang.org/x/net/html" +) + +// ErrUnknownVersion is the error returned when the first character inside the +// element encoding (but outside the base64 encoding) is not '0'. +type ErrUnknownVersion byte + +func (err ErrUnknownVersion) Error() string { + return fmt.Sprintf("unknown armor version indicator %+q", byte(err)) +} + +func isASCIIWhitespace(b byte) bool { + switch b { + // https://infra.spec.whatwg.org/#ascii-whitespace + case '\x09', '\x0a', '\x0c', '\x0d', '\x20': + return true + default: + return false + } +} + +func splitASCIIWhitespace(data []byte, atEOF bool) (advance int, token []byte, err error) { + var i, j int + // Skip initial whitespace. + for i = 0; i < len(data); i++ { + if !isASCIIWhitespace(data[i]) { + break + } + } + // Look for next whitespace. + for j = i; j < len(data); j++ { + if isASCIIWhitespace(data[j]) { + return j + 1, data[i:j], nil + } + } + // We reached the end of data without finding more whitespace. Only + // consider it a token if we are at EOF. + if atEOF && i < j { + return j, data[i:j], nil + } + // Otherwise, request more data. + return i, nil, nil +} + +func decodeToWriter(w io.Writer, r io.Reader) (int64, error) { + tokenizer := html.NewTokenizer(r) + // Set a memory limit on token sizes, otherwise the tokenizer will + // buffer text indefinitely if it is not broken up by other token types. + tokenizer.SetMaxBuf(elementSizeLimit) + active := false + total := int64(0) + for { + tt := tokenizer.Next() + switch tt { + case html.ErrorToken: + err := tokenizer.Err() + if err == io.EOF { + err = nil + } + if err == nil && active { + return total, fmt.Errorf("missing </pre> tag") + } + return total, err + case html.TextToken: + if active { + // Re-join the separate chunks of text and + // feed them to the decoder. + scanner := bufio.NewScanner(bytes.NewReader(tokenizer.Text())) + scanner.Split(splitASCIIWhitespace) + for scanner.Scan() { + n, err := w.Write(scanner.Bytes()) + total += int64(n) + if err != nil { + return total, err + } + } + if err := scanner.Err(); err != nil { + return total, err + } + } + case html.StartTagToken: + tn, _ := tokenizer.TagName() + if string(tn) == "pre" { + if active { + // nesting not allowed + return total, fmt.Errorf("unexpected %s", tokenizer.Token()) + } + active = true + } + case html.EndTagToken: + tn, _ := tokenizer.TagName() + if string(tn) == "pre" { + if !active { + // stray end tag + return total, fmt.Errorf("unexpected %s", tokenizer.Token()) + } + active = false + } + } + } +} + +// NewArmorDecoder returns a new AMP armor decoder. +func NewArmorDecoder(r io.Reader) (io.Reader, error) { + pr, pw := io.Pipe() + go func() { + _, err := decodeToWriter(pw, r) + pw.CloseWithError(err) + }() + + // The first byte inside the element encoding is a server–client + // protocol version indicator. + var version [1]byte + _, err := pr.Read(version[:]) + if err != nil { + pr.CloseWithError(err) + return nil, err + } + switch version[0] { + case '0': + return base64.NewDecoder(base64.StdEncoding, pr), nil + default: + err := ErrUnknownVersion(version[0]) + pr.CloseWithError(err) + return nil, err + } +} diff --git a/common/amp/armor_encoder.go b/common/amp/armor_encoder.go new file mode 100644 index 0000000..5d6b0ae --- /dev/null +++ b/common/amp/armor_encoder.go @@ -0,0 +1,176 @@ +package amp + +import ( + "encoding/base64" + "io" +) + +// https://amp.dev/boilerplate/ +// https://amp.dev/documentation/guides-and-tutorials/learn/spec/amp-boilerpla… +// https://amp.dev/documentation/guides-and-tutorials/learn/spec/amphtml/?form… +const ( + boilerplateStart = `<!doctype html> +<html amp> +<head> +<meta charset="utf-8"> +<script async src="https://cdn.ampproject.org/v0.js"></script> +<link rel="canonical" href="#"> +<meta name="viewport" content="width=device-width"> +<style amp-boilerplate>body{-webkit-animation:-amp-start 8s steps(1,end) 0s 1 normal both;-moz-animation:-amp-start 8s steps(1,end) 0s 1 normal both;-ms-animation:-amp-start 8s steps(1,end) 0s 1 normal both;animation:-amp-start 8s steps(1,end) 0s 1 normal both}@-webkit-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-moz-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-ms-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-o-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}</style><noscript><style amp-boilerplate>body{-webkit-animation:none;-moz-animation:none;-ms-animation:none;animation:none}</style></noscript> +</head> +<body> +` + boilerplateEnd = `</body> +</html>` +) + +const ( + // We restrict the amount of text may go inside an HTML element, in + // order to limit the amount a decoder may have to buffer. + elementSizeLimit = 32 * 1024 + + // The payload is conceptually a long base64-encoded string, but we + // break the string into short chunks separated by whitespace. This is + // to protect against modification by AMP caches, which reportedly may + // truncate long words in text: + // https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowfl… + bytesPerChunk = 32 + + // We set the number of chunks per element so as to stay under + // elementSizeLimit. Here, we assume that there is 1 byte of whitespace + // after each chunk (with an additional whitespace byte at the beginning + // of the element). + chunksPerElement = (elementSizeLimit - 1) / (bytesPerChunk + 1) +) + +// The AMP armor encoder is a chain of a base64 encoder (base64.NewEncoder) and +// an HTML element encoder (elementEncoder). A top-level encoder (armorEncoder) +// coordinates these two, and handles prepending and appending the AMP +// boilerplate. armorEncoder's Write method writes data into the base64 encoder, +// where it makes its way through the chain. + +// NewArmorEncoder returns a new AMP armor encoder. Anything written to the +// returned io.WriteCloser will be encoded and written to w. The caller must +// call Close to flush any partially written data and output the AMP boilerplate +// trailer. +func NewArmorEncoder(w io.Writer) (io.WriteCloser, error) { + // Immediately write the AMP boilerplate header. + _, err := w.Write([]byte(boilerplateStart)) + if err != nil { + return nil, err + } + + element := &elementEncoder{w: w} + // Write a server–client protocol version indicator, outside the base64 + // layer. + _, err = element.Write([]byte{'0'}) + if err != nil { + return nil, err + } + + base64 := base64.NewEncoder(base64.StdEncoding, element) + return &armorEncoder{ + w: w, + element: element, + base64: base64, + }, nil +} + +type armorEncoder struct { + base64 io.WriteCloser + element *elementEncoder + w io.Writer +} + +func (enc *armorEncoder) Write(p []byte) (int, error) { + // Write into the chain base64 | element | w. + return enc.base64.Write(p) +} + +func (enc *armorEncoder) Close() error { + // Close the base64 encoder first, to flush out any buffered data and + // the final padding. + err := enc.base64.Close() + if err != nil { + return err + } + + // Next, close the element encoder, to close any open elements. + err = enc.element.Close() + if err != nil { + return err + } + + // Finally, output the AMP boilerplate trailer. + _, err = enc.w.Write([]byte(boilerplateEnd)) + if err != nil { + return err + } + + return nil +} + +// elementEncoder arranges written data into pre elements, with the text within +// separated into chunks. It does no HTML encoding, so data written must not +// contain any bytes that are meaningful in HTML. +type elementEncoder struct { + w io.Writer + chunkCounter int + elementCounter int +} + +func (enc *elementEncoder) Write(p []byte) (n int, err error) { + total := 0 + for len(p) > 0 { + if enc.elementCounter == 0 && enc.chunkCounter == 0 { + _, err := enc.w.Write([]byte("<pre>\n")) + if err != nil { + return total, err + } + } + + n := bytesPerChunk - enc.chunkCounter + if n > len(p) { + n = len(p) + } + nn, err := enc.w.Write(p[:n]) + if err != nil { + return total, err + } + total += nn + p = p[n:] + + enc.chunkCounter += n + if enc.chunkCounter >= bytesPerChunk { + enc.chunkCounter = 0 + enc.elementCounter += 1 + nn, err = enc.w.Write([]byte("\n")) + if err != nil { + return total, err + } + total += nn + } + + if enc.elementCounter >= chunksPerElement { + enc.elementCounter = 0 + nn, err = enc.w.Write([]byte("</pre>\n")) + if err != nil { + return total, err + } + total += nn + } + } + return total, nil +} + +func (enc *elementEncoder) Close() error { + var err error + if !(enc.elementCounter == 0 && enc.chunkCounter == 0) { + if enc.chunkCounter == 0 { + _, err = enc.w.Write([]byte("</pre>\n")) + } else { + _, err = enc.w.Write([]byte("\n</pre>\n")) + } + } + return err +} diff --git a/common/amp/armor_test.go b/common/amp/armor_test.go new file mode 100644 index 0000000..594ae65 --- /dev/null +++ b/common/amp/armor_test.go @@ -0,0 +1,227 @@ +package amp + +import ( + "crypto/rand" + "io" + "io/ioutil" + "strings" + "testing" +) + +func armorDecodeToString(src string) (string, error) { + dec, err := NewArmorDecoder(strings.NewReader(src)) + if err != nil { + return "", err + } + p, err := ioutil.ReadAll(dec) + return string(p), err +} + +func TestArmorDecoder(t *testing.T) { + for _, test := range []struct { + input string + expectedOutput string + expectedErr bool + }{ + {` +<pre> +0 +</pre> +`, + "", + false, + }, + {` +<pre> +0aGVsbG8gd29ybGQK +</pre> +`, + "hello world\n", + false, + }, + // bad version indicator + {` +<pre> +1aGVsbG8gd29ybGQK +</pre> +`, + "", + true, + }, + // text outside <pre> elements + {` +0aGVsbG8gd29ybGQK +blah blah blah +<pre> +0aGVsbG8gd29ybGQK +</pre> +0aGVsbG8gd29ybGQK +blah blah blah +`, + "hello world\n", + false, + }, + {` +<pre> +0QUJDREV +GR0hJSkt +MTU5PUFF +SU1RVVld +</pre> +junk +<pre> +YWVowMTI +zNDU2Nzg +5Cg += +</pre> +<pre> += +</pre> +`, + "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789\n", + false, + }, + // no <pre> elements, hence no version indicator + {` +aGVsbG8gd29ybGQK +blah blah blah +aGVsbG8gd29ybGQK +aGVsbG8gd29ybGQK +blah blah blah +`, + "", + true, + }, + // empty <pre> elements, hence no version indicator + {` +aGVsbG8gd29ybGQK +blah blah blah +<pre> </pre> +aGVsbG8gd29ybGQK +aGVsbG8gd29ybGQK<pre></pre> +blah blah blah +`, + "", + true, + }, + // other elements inside <pre> + { + "blah <pre>0aGVsb<p>G8gd29</p>ybGQK</pre>", + "hello world\n", + false, + }, + // HTML comment + { + "blah ", + "", + true, + }, + // all kinds of ASCII whitespace + { + "blah <pre>\x200\x09aG\x0aV\x0csb\x0dG8\x20gd29ybGQK</pre>", + "hello world\n", + false, + }, + + // bad padding + {` +<pre> +0QUJDREV +GR0hJSkt +MTU5PUFF +SU1RVVld +</pre> +junk +<pre> +YWVowMTI +zNDU2Nzg +5Cg += +</pre> +`, + "", + true, + }, + /* + // per-chunk base64 + // test disabled because Go stdlib handles this incorrectly: + // https://github.com/golang/go/issues/31626 + { + "<pre>QQ==</pre><pre>Qg==</pre>", + "", + true, + }, + */ + // missing </pre> + { + "blah <pre></pre><pre>0aGVsbG8gd29ybGQK", + "", + true, + }, + // nested <pre> + { + "blah <pre>0aGVsb<pre>G8gd29</pre>ybGQK</pre>", + "", + true, + }, + } { + output, err := armorDecodeToString(test.input) + if test.expectedErr && err == nil { + t.Errorf("%+q → (%+q, %v), expected error", test.input, output, err) + continue + } + if !test.expectedErr && err != nil { + t.Errorf("%+q → (%+q, %v), expected no error", test.input, output, err) + continue + } + if !test.expectedErr && output != test.expectedOutput { + t.Errorf("%+q → (%+q, %v), expected (%+q, %v)", + test.input, output, err, test.expectedOutput, nil) + continue + } + } +} + +func armorRoundTrip(s string) (string, error) { + var encoded strings.Builder + enc, err := NewArmorEncoder(&encoded) + if err != nil { + return "", err + } + _, err = io.Copy(enc, strings.NewReader(s)) + if err != nil { + return "", err + } + err = enc.Close() + if err != nil { + return "", err + } + return armorDecodeToString(encoded.String()) +} + +func TestArmorRoundTrip(t *testing.T) { + lengths := make([]int, 0) + // Test short strings and lengths around elementSizeLimit thresholds. + for i := 0; i < bytesPerChunk*2; i++ { + lengths = append(lengths, i) + } + for i := -10; i < +10; i++ { + lengths = append(lengths, elementSizeLimit+i) + lengths = append(lengths, 2*elementSizeLimit+i) + } + for _, n := range lengths { + buf := make([]byte, n) + rand.Read(buf) + input := string(buf) + output, err := armorRoundTrip(input) + if err != nil { + t.Errorf("length %d → error %v", n, err) + continue + } + if output != input { + t.Errorf("length %d → %+q", n, output) + continue + } + } +} diff --git a/common/amp/cache.go b/common/amp/cache.go new file mode 100644 index 0000000..102993f --- /dev/null +++ b/common/amp/cache.go @@ -0,0 +1,178 @@ +package amp + +import ( + "crypto/sha256" + "encoding/base32" + "fmt" + "net" + "net/url" + "path" + "strings" + + "golang.org/x/net/idna" +) + +// domainPrefixBasic does the basic domain prefix conversion. Does not do any +// IDNA mapping, such as https://www.unicode.org/reports/tr46/. +// +// https://amp.dev/documentation/guides-and-tutorials/learn/amp-caches-and-cor… +func domainPrefixBasic(domain string) (string, error) { + // 1. Punycode Decode the publisher domain. + prefix, err := idna.ToUnicode(domain) + if err != nil { + return "", err + } + + // 2. Replace any "-" (hyphen) character in the output of step 1 with + // "--" (two hyphens). + prefix = strings.Replace(prefix, "-", "--", -1) + + // 3. Replace any "." (dot) character in the output of step 2 with "-" + // (hyphen). + prefix = strings.Replace(prefix, ".", "-", -1) + + // 4. If the output of step 3 has a "-" (hyphen) at both positions 3 and + // 4, then to the output of step 3, add a prefix of "0-" and add a + // suffix of "-0". + if len(prefix) >= 4 && prefix[2] == '-' && prefix[3] == '-' { + prefix = "0-" + prefix + "-0" + } + + // 5. Punycode Encode the output of step 3. + return idna.ToASCII(prefix) +} + +// Lower-case base32 without padding. +var fallbackBase32Encoding = base32.NewEncoding("abcdefghijklmnopqrstuvwxyz234567").WithPadding(base32.NoPadding) + +// domainPrefixFallback does the fallback domain prefix conversion. The returned +// base32 domain uses lower-case letters. +// +// https://amp.dev/documentation/guides-and-tutorials/learn/amp-caches-and-cor… +func domainPrefixFallback(domain string) string { + // The algorithm specification does not say what, exactly, we are to + // take the SHA-256 of. domain is notionally an abstract Unicode + // string, not a byte sequence. While + // https://github.com/ampproject/amp-toolbox/blob/84cb3057e5f6c54d64369ddd285d… + // says "Take the SHA256 of the punycode view of the domain," in reality + // it hashes the UTF-8 encoding of the domain, without Punycode: + // https://github.com/ampproject/amp-toolbox/blob/84cb3057e5f6c54d64369ddd285d… + // https://github.com/ampproject/amp-toolbox/blob/84cb3057e5f6c54d64369ddd285d… + // We do the same here, hashing the raw bytes of domain, presumed to be + // UTF-8. + + // 1. Hash the publisher's domain using SHA256. + h := sha256.Sum256([]byte(domain)) + + // 2. Base32 Escape the output of step 1. + // 3. Remove the last 4 characters from the output of step 2, which are + // always "=" (equals) characters. + return fallbackBase32Encoding.EncodeToString(h[:]) +} + +// domainPrefix computes the domain prefix of an AMP cache URL. +// +// https://amp.dev/documentation/guides-and-tutorials/learn/amp-caches-and-cor… +func domainPrefix(domain string) string { + // https://amp.dev/documentation/guides-and-tutorials/learn/amp-caches-and-cor… + // 1. Run the Basic Algorithm. If the output is a valid DNS label, + // [append the Cache domain suffix and] return. Otherwise continue to + // step 2. + prefix, err := domainPrefixBasic(domain) + // "A domain prefix is not a valid DNS label if it is longer than 63 + // characters" + if err == nil && len(prefix) <= 63 { + return prefix + } + // 2. Run the Fallback Algorithm. [Append the Cache domain suffix and] + // return. + return domainPrefixFallback(domain) +} + +// CacheURL computes the AMP cache URL for the publisher URL pubURL, using the +// AMP cache at cacheURL. contentType is a string such as "c" or "i" that +// indicates what type of serving the AMP cache is to perform. The Scheme of +// pubURL must be "http" or "https". The Port of pubURL, if any, must match the +// default for the scheme. cacheURL may not have RawQuery, Fragment, or +// RawFragment set, because the resulting URL's query and fragment are taken +// from the publisher URL. +// +// https://amp.dev/documentation/guides-and-tutorials/learn/amp-caches-and-cor… +func CacheURL(pubURL, cacheURL *url.URL, contentType string) (*url.URL, error) { + // The cache URL subdomain, including the domain prefix corresponding to + // the publisher URL's domain. + resultHost := domainPrefix(pubURL.Hostname()) + "." + cacheURL.Hostname() + if cacheURL.Port() != "" { + resultHost = net.JoinHostPort(resultHost, cacheURL.Port()) + } + + // https://amp.dev/documentation/guides-and-tutorials/learn/amp-caches-and-cor… + // The first part of the path is the cache URL's own path, if any. + pathComponents := []string{cacheURL.EscapedPath()} + // The next path component is the content type. We cannot encode an + // empty content type, because it would result in consecutive path + // separators, which would semantically combine into a single separator. + if contentType == "" { + return nil, fmt.Errorf("invalid content type %+q", contentType) + } + pathComponents = append(pathComponents, url.PathEscape(contentType)) + // Then, we add an "s" path component, if the publisher URL scheme is + // "https". + switch pubURL.Scheme { + case "http": + // Do nothing. + case "https": + pathComponents = append(pathComponents, "s") + default: + return nil, fmt.Errorf("invalid scheme %+q in publisher URL", pubURL.Scheme) + } + // The next path component is the publisher URL's host. The AMP cache + // URL format specification is not clear about whether other + // subcomponents of the authority (namely userinfo and port) may appear + // here. We adopt a policy of forbidding userinfo, and requiring that + // the port be the default for the scheme (and then we omit the port + // entirely from the returned URL). + if pubURL.User != nil { + return nil, fmt.Errorf("publisher URL may not contain userinfo") + } + if port := pubURL.Port(); port != "" { + if !((pubURL.Scheme == "http" && port == "80") || (pubURL.Scheme == "https" && port == "443")) { + return nil, fmt.Errorf("publisher URL port %+q is not the default for scheme %+q", port, pubURL.Scheme) + } + } + // As with the content type, we cannot encode an empty host, because + // that would result in an empty path component. + if pubURL.Hostname() == "" { + return nil, fmt.Errorf("invalid host %+q in publisher URL", pubURL.Hostname()) + } + pathComponents = append(pathComponents, url.PathEscape(pubURL.Hostname())) + // Finally, we append the remainder of the original escaped path from + // the publisher URL. + pathComponents = append(pathComponents, pubURL.EscapedPath()) + + resultRawPath := path.Join(pathComponents...) + resultPath, err := url.PathUnescape(resultRawPath) + if err != nil { + return nil, err + } + + // The query and fragment of the returned URL always come from pubURL. + // Any query or fragment of cacheURL would be ignored. Return an error + // if either is set. + if cacheURL.RawQuery != "" { + return nil, fmt.Errorf("cache URL may not contain a query") + } + if cacheURL.Fragment != "" { + return nil, fmt.Errorf("cache URL may not contain a fragment") + } + + return &url.URL{ + Scheme: cacheURL.Scheme, + User: cacheURL.User, + Host: resultHost, + Path: resultPath, + RawPath: resultRawPath, + RawQuery: pubURL.RawQuery, + Fragment: pubURL.Fragment, + }, nil +} diff --git a/common/amp/cache_test.go b/common/amp/cache_test.go new file mode 100644 index 0000000..45950fd --- /dev/null +++ b/common/amp/cache_test.go @@ -0,0 +1,320 @@ +package amp + +import ( + "bytes" + "net/url" + "testing" + + "golang.org/x/net/idna" +) + +func TestDomainPrefixBasic(t *testing.T) { + // Tests expecting no error. + for _, test := range []struct { + domain, expected string + }{ + {"", ""}, + {"xn--", ""}, + {"...", "---"}, + + // Should not apply mappings such as case folding and + // normalization. + {"b\u00fccher.de", "xn--bcher-de-65a"}, + {"B\u00fccher.de", "xn--Bcher-de-65a"}, + {"bu\u0308cher.de", "xn--bucher-de-hkf"}, + + // Check some that differ between IDNA 2003 and IDNA 2008. + // https://unicode.org/reports/tr46/#Deviations + // https://util.unicode.org/UnicodeJsps/idna.jsp + {"faß.de", "xn--fa-de-mqa"}, + {"βόλοσ.com", "xn---com-4ld8c2a6a8e"}, + + // Lengths of 63 and 64. 64 is too long for a DNS label, but + // domainPrefixBasic is not expected to check for that. + {"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}, + {"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}, + + // https://amp.dev/documentation/guides-and-tutorials/learn/amp-caches-and-cor… + {"example.com", "example-com"}, + {"foo.example.com", "foo-example-com"}, + {"foo-example.com", "foo--example-com"}, + {"xn--57hw060o.com", "xn---com-p33b41770a"}, + {"\u26a1\U0001f60a.com", "xn---com-p33b41770a"}, + {"en-us.example.com", "0-en--us-example-com-0"}, + } { + output, err := domainPrefixBasic(test.domain) + if err != nil || output != test.expected { + t.Errorf("%+q → (%+q, %v), expected (%+q, %v)", + test.domain, output, err, test.expected, nil) + } + } + + // Tests expecting an error. + for _, domain := range []string{ + "xn---", + } { + output, err := domainPrefixBasic(domain) + if err == nil || output != "" { + t.Errorf("%+q → (%+q, %v), expected (%+q, non-nil)", + domain, output, err, "") + } + } +} + +func TestDomainPrefixFallback(t *testing.T) { + for _, test := range []struct { + domain, expected string + }{ + { + "", + "4oymiquy7qobjgx36tejs35zeqt24qpemsnzgtfeswmrw6csxbkq", + }, + { + "example.com", + "un42n5xov642kxrxrqiyanhcoupgql5lt4wtbkyt2ijflbwodfdq", + }, + + // These checked against the output of + // https://github.com/ampproject/amp-toolbox/tree/84cb3057e5f6c54d64369ddd285d…, + // using the widget at + // https://amp.dev/documentation/guides-and-tutorials/learn/amp-caches-and-cor…. + { + "000000000000000000000000000000000000000000000000000000000000.com", + "stejanx4hsijaoj4secyecy4nvqodk56kw72whwcmvdbtucibf5a", + }, + { + "00000000000000000000000000000000000000000000000000000000000a.com", + "jdcvbsorpnc3hcjrhst56nfm6ymdpovlawdbm2efyxpvlt4cpbya", + }, + { + "00000000000000000000000000000000000000000000000000000000000\u03bb.com", + "qhzqeumjkfpcpuic3vqruyjswcr7y7gcm3crqyhhywvn3xrhchfa", + }, + } { + output := domainPrefixFallback(test.domain) + if output != test.expected { + t.Errorf("%+q → %+q, expected %+q", + test.domain, output, test.expected) + } + } +} + +// Checks that domainPrefix chooses domainPrefixBasic or domainPrefixFallback as +// appropriate; i.e., always returns string that is a valid DNS label and is +// IDNA-decodable. +func TestDomainPrefix(t *testing.T) { + // A validating IDNA profile, which checks label length and that the + // label contains only certain ASCII characters. It does not do the + // ValidateLabels check, because that depends on the input having + // certain properties. + profile := idna.New( + idna.VerifyDNSLength(true), + idna.StrictDomainName(true), + ) + for _, domain := range []string{ + "example.com", + "\u0314example.com", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", // 63 bytes + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", // 64 bytes + "xn--57hw060o.com", + "a b c", + } { + output := domainPrefix(domain) + if bytes.IndexByte([]byte(output), '.') != -1 { + t.Errorf("%+q → %+q contains a dot", domain, output) + } + _, err := profile.ToUnicode(output) + if err != nil { + t.Errorf("%+q → error %v", domain, err) + } + } +} + +func mustParseURL(rawurl string) *url.URL { + u, err := url.Parse(rawurl) + if err != nil { + panic(err) + } + return u +} + +func TestCacheURL(t *testing.T) { + // Tests expecting no error. + for _, test := range []struct { + pub string + cache string + contentType string + expected string + }{ + // With or without trailing slash on pubURL. + { + "http://example.com/", + "https://amp.cache/", + "c", + "https://example-com.amp.cache/c/example.com", + }, + { + "http://example.com", + "https://amp.cache/", + "c", + "https://example-com.amp.cache/c/example.com", + }, + // https pubURL. + { + "https://example.com/", + "https://amp.cache/", + "c", + "https://example-com.amp.cache/c/s/example.com", + }, + // The content type should be escaped if necessary. + { + "http://example.com/", + "https://amp.cache/", + "/", + "https://example-com.amp.cache/%2F/example.com", + }, + // Retain pubURL path, query, and fragment, including escaping. + { + "http://example.com/my%2Fpath/index.html?a=1#fragment", + "https://amp.cache/", + "c", + "https://example-com.amp.cache/c/example.com/my%2Fpath/index.html?a=1#fragme…", + }, + // Retain scheme, userinfo, port, and path of cacheURL, escaping + // whatever is necessary. + { + "http://example.com", + "http://cache%2Fuser:cache%40pass@amp.cache:123/with/../../path/..%2f../", + "c", + "http://cache%2Fuser:cache%40pass@example-com.amp.cache:123/path/..%2f../c/e…", + }, + // Port numbers in pubURL are allowed, if they're the default + // for scheme. + { + "http://example.com:80/", + "https://amp.cache/", + "c", + "https://example-com.amp.cache/c/example.com", + }, + { + "https://example.com:443/", + "https://amp.cache/", + "c", + "https://example-com.amp.cache/c/s/example.com", + }, + // "?" at the end of cacheURL is okay, as long as the query is + // empty. + { + "http://example.com/", + "https://amp.cache/?", + "c", + "https://example-com.amp.cache/c/example.com", + }, + + // https://developers.google.com/amp/cache/overview#example-requesting-documen… + { + "https://example.com/amp_document.html", + "https://cdn.ampproject.org/", + "c", + "https://example-com.cdn.ampproject.org/c/s/example.com/amp_document.html", + }, + // https://developers.google.com/amp/cache/overview#example-requesting-image-u… + { + "http://example.com/logo.png", + "https://cdn.ampproject.org/", + "i", + "https://example-com.cdn.ampproject.org/i/example.com/logo.png", + }, + // https://developers.google.com/amp/cache/overview#query-parameter-example + { + "https://example.com/g?value=Hello%20World", + "https://cdn.ampproject.org/", + "c", + "https://example-com.cdn.ampproject.org/c/s/example.com/g?value=Hello%20World", + }, + } { + pubURL := mustParseURL(test.pub) + cacheURL := mustParseURL(test.cache) + outputURL, err := CacheURL(pubURL, cacheURL, test.contentType) + if err != nil { + t.Errorf("%+q %+q %+q → error %v", + test.pub, test.cache, test.contentType, err) + continue + } + if outputURL.String() != test.expected { + t.Errorf("%+q %+q %+q → %+q, expected %+q", + test.pub, test.cache, test.contentType, outputURL, test.expected) + continue + } + } + + // Tests expecting an error. + for _, test := range []struct { + pub string + cache string + contentType string + }{ + // Empty content type. + { + "http://example.com/", + "https://amp.cache/", + "", + }, + // Empty host. + { + "http:///index.html", + "https://amp.cache/", + "c", + }, + // Empty scheme. + { + "//example.com/", + "https://amp.cache/", + "c", + }, + // Unrecognized scheme. + { + "ftp://example.com/", + "https://amp.cache/", + "c", + }, + // Wrong port number for scheme. + { + "http://example.com:443/", + "https://amp.cache/", + "c", + }, + // userinfo in pubURL. + { + "http://user@example.com/", + "https://amp.cache/", + "c", + }, + { + "http://user:pass@example.com/", + "https://amp.cache/", + "c", + }, + // cacheURL may not contain a query. + { + "http://example.com/", + "https://amp.cache/?a=1", + "c", + }, + // cacheURL may not contain a fragment. + { + "http://example.com/", + "https://amp.cache/#fragment", + "c", + }, + } { + pubURL := mustParseURL(test.pub) + cacheURL := mustParseURL(test.cache) + outputURL, err := CacheURL(pubURL, cacheURL, test.contentType) + if err == nil { + t.Errorf("%+q %+q %+q → %+q, expected error", + test.pub, test.cache, test.contentType, outputURL) + continue + } + } +} diff --git a/common/amp/doc.go b/common/amp/doc.go new file mode 100644 index 0000000..1387114 --- /dev/null +++ b/common/amp/doc.go @@ -0,0 +1,88 @@ +/* +Package amp provides functions for working with the AMP (Accelerated Mobile +Pages) subset of HTML, and conveying binary data through an AMP cache. + +AMP cache + +The CacheURL function takes a plain URL and converts it to be accessed through a +given AMP cache. + +The EncodePath and DecodePath functions provide a way to encode data into the +suffix of a URL path. AMP caches do not support HTTP POST, but encoding data +into a URL path with GET is an alternative means of sending data to the server. +The format of an encoded path is: + 0<0 or more bytes, including slash>/<base64 of data> +That is: +* "0", a format version number, which controls the interpretation of the rest of +the path. Only the first byte matters as a version indicator (not the whole +first path component). +* Any number of slash or non-slash bytes. These may be used as padding or to +prevent cache collisions in the AMP cache. +* A final slash. +* base64 encoding of the data, using the URL-safe alphabet (which does not +include slash). + +For example, an encoding of the string "This is path-encoded data." is the +following. The "lgWHcwhXFjUm" following the format version number is random +padding that will be ignored on decoding. + 0lgWHcwhXFjUm/VGhpcyBpcyBwYXRoLWVuY29kZWQgZGF0YS4 + +It is the caller's responsibility to add or remove any directory path prefix +before calling EncodePath or DecodePath. + +AMP armor + +AMP armor is a data encoding scheme that that satisfies the requirements of the +AMP (Accelerated Mobile Pages) subset of HTML, and survives modification by an +AMP cache. For the requirements of AMP HTML, see +https://amp.dev/documentation/guides-and-tutorials/learn/spec/amphtml/. +For modifications that may be made by an AMP cache, see +https://github.com/ampproject/amphtml/blob/main/docs/spec/amp-cache-modifications.md. + +The encoding is based on ones created by Ivan Markin. See codec/amp/ in +https://github.com/nogoegst/amper and discussion at +https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/25985. + +The encoding algorithm works as follows. Base64-encode the input. Prepend the +input with the byte '0'; this is a protocol version indicator that the decoder +can use to determine how to interpret the bytes that follow. Split the base64 +into fixed-size chunks separated by whitespace. Take up to 1024 chunks at a +time, and wrap them in a pre element. Then, situate the markup so far within the +body of the AMP HTML boilerplate. The decoding algorithm is to scan the HTML for +pre elements, split their text contents on whitespace and concatenate, then +base64 decode. The base64 encoding uses the standard alphabet, with normal "=" +padding (https://tools.ietf.org/html/rfc4648#section-4) + +The reason for splitting the base64 into chunks is that AMP caches reportedly +truncate long strings that are not broken by whitespace: +https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/25985#note_2592348. +The characters that may separate the chunks are the ASCII whitespace characters +(https://infra.spec.whatwg.org/#ascii-whitespace) "\x09", "\x0a", "\x0c", +"\x0d", and "\x20". The reason for separating the chunks into pre elements is to +limit the amount of text a decoder may have to buffer while parsing the HTML. +Each pre element may contain at most 64 KB of text. pre elements may not be +nested. + +Example + +The following is the result of encoding the string +"This was encoded with AMP armor.": + + <!doctype html> + <html amp> + <head> + <meta charset="utf-8"> + <script async src="https://cdn.ampproject.org/v0.js"></script> + <link rel="canonical" href="#"> + <meta name="viewport" content="width=device-width"> + <style amp-boilerplate>body{-webkit-animation:-amp-start 8s steps(1,end) 0s 1 normal both;-moz-animation:-amp-start 8s steps(1,end) 0s 1 normal both;-ms-animation:-amp-start 8s steps(1,end) 0s 1 normal both;animation:-amp-start 8s steps(1,end) 0s 1 normal both}@-webkit-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-moz-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-ms-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-o-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}</style><noscript><style amp-boilerplate>body{-webkit-animation:none;-moz-animation:none;-ms-animation:none;animation:none}</style></noscript> + </head> + <body> + <pre> + 0VGhpcyB3YXMgZW5jb2RlZCB3aXRoIEF + NUCBhcm1vci4= + </pre> + </body> + </html> +*/ +package amp diff --git a/common/amp/path.go b/common/amp/path.go new file mode 100644 index 0000000..5903694 --- /dev/null +++ b/common/amp/path.go @@ -0,0 +1,44 @@ +package amp + +import ( + "crypto/rand" + "encoding/base64" + "fmt" + "strings" +) + +// EncodePath encodes data in a way that is suitable for the suffix of an AMP +// cache URL. +func EncodePath(data []byte) string { + var cacheBreaker [9]byte + _, err := rand.Read(cacheBreaker[:]) + if err != nil { + panic(err) + } + b64 := base64.RawURLEncoding.EncodeToString + return "0" + b64(cacheBreaker[:]) + "/" + b64(data) +} + +// DecodePath decodes data from a path suffix as encoded by EncodePath. The path +// must have already been trimmed of any directory prefix (as might be present +// in, e.g., an HTTP request). That is, the first character of path should be +// the "0" message format indicator. +func DecodePath(path string) ([]byte, error) { + if len(path) < 1 { + return nil, fmt.Errorf("missing format indicator") + } + version := path[0] + rest := path[1:] + switch version { + case '0': + // Ignore everything else up to and including the final slash + // (there must be at least one slash). + i := strings.LastIndexByte(rest, '/') + if i == -1 { + return nil, fmt.Errorf("missing data") + } + return base64.RawURLEncoding.DecodeString(rest[i+1:]) + default: + return nil, fmt.Errorf("unknown format indicator %q", version) + } +} diff --git a/common/amp/path_test.go b/common/amp/path_test.go new file mode 100644 index 0000000..20e4ccf --- /dev/null +++ b/common/amp/path_test.go @@ -0,0 +1,54 @@ +package amp + +import ( + "testing" +) + +func TestDecodePath(t *testing.T) { + for _, test := range []struct { + path string + expectedData string + expectedErrStr string + }{ + {"", "", "missing format indicator"}, + {"0", "", "missing data"}, + {"0foobar", "", "missing data"}, + {"/0/YWJj", "", "unknown format indicator '/'"}, + + {"0/", "", ""}, + {"0foobar/", "", ""}, + {"0/YWJj", "abc", ""}, + {"0///YWJj", "abc", ""}, + {"0foobar/YWJj", "abc", ""}, + {"0/foobar/YWJj", "abc", ""}, + } { + data, err := DecodePath(test.path) + if test.expectedErrStr != "" { + if err == nil || err.Error() != test.expectedErrStr { + t.Errorf("%+q expected error %+q, got %+q", + test.path, test.expectedErrStr, err) + } + } else if err != nil { + t.Errorf("%+q expected no error, got %+q", test.path, err) + } else if string(data) != test.expectedData { + t.Errorf("%+q expected data %+q, got %+q", + test.path, test.expectedData, data) + } + } +} + +func TestPathRoundTrip(t *testing.T) { + for _, data := range []string{ + "", + "\x00", + "/", + "hello world", + } { + decoded, err := DecodePath(EncodePath([]byte(data))) + if err != nil { + t.Errorf("%+q roundtripped with error %v", data, err) + } else if string(decoded) != data { + t.Errorf("%+q roundtripped to %+q", data, decoded) + } + } +}

1 0

[snowflake/main] Skeleton of ampCacheRendezvous.
by dcf＠torproject.org 05 Aug '21

05 Aug '21

commit c13810192d243690dab4c0e890a1f50273a22ca1 Author: David Fifield <david(a)bamsoftware.com> Date: Sun Jul 18 14:18:32 2021 -0600 Skeleton of ampCacheRendezvous. Currently the same as httpRendezvous, but activated using the -ampcache command-line option. --- client/lib/rendezvous.go | 13 ++++++- client/lib/rendezvous_ampcache.go | 78 +++++++++++++++++++++++++++++++++++++++ client/lib/snowflake.go | 7 ++-- client/snowflake.go | 3 +- 4 files changed, 95 insertions(+), 6 deletions(-) diff --git a/client/lib/rendezvous.go b/client/lib/rendezvous.go index 8568120..8af638f 100644 --- a/client/lib/rendezvous.go +++ b/client/lib/rendezvous.go @@ -59,13 +59,22 @@ func CreateBrokerTransport() http.RoundTripper { // Construct a new BrokerChannel, where: // |broker| is the full URL of the facilitating program which assigns proxies // to clients, and |front| is the option fronting domain. -func NewBrokerChannel(broker string, front string, transport http.RoundTripper, keepLocalAddresses bool) (*BrokerChannel, error) { +func NewBrokerChannel(broker, ampCache, front string, transport http.RoundTripper, keepLocalAddresses bool) (*BrokerChannel, error) { log.Println("Rendezvous using Broker at:", broker) + if ampCache != "" { + log.Println("Through AMP cache at:", ampCache) + } if front != "" { log.Println("Domain fronting using:", front) } - rendezvous, err := newHTTPRendezvous(broker, front, transport) + var rendezvous rendezvousMethod + var err error + if ampCache != "" { + rendezvous, err = newAMPCacheRendezvous(broker, ampCache, front, transport) + } else { + rendezvous, err = newHTTPRendezvous(broker, front, transport) + } if err != nil { return nil, err } diff --git a/client/lib/rendezvous_ampcache.go b/client/lib/rendezvous_ampcache.go new file mode 100644 index 0000000..89745f4 --- /dev/null +++ b/client/lib/rendezvous_ampcache.go @@ -0,0 +1,78 @@ +package lib + +import ( + "bytes" + "errors" + "log" + "net/http" + "net/url" +) + +// ampCacheRendezvous is a rendezvousMethod that communicates with the +// .../amp/client route of the broker, optionally over an AMP cache proxy, and +// with optional domain fronting. +type ampCacheRendezvous struct { + brokerURL *url.URL + cacheURL *url.URL // Optional AMP cache URL. + front string // Optional front domain to replace url.Host in requests. + transport http.RoundTripper // Used to make all requests. +} + +// newAMPCacheRendezvous creates a new ampCacheRendezvous that contacts the +// broker at the given URL, optionally proxying through an AMP cache, and with +// an optional front domain. transport is the http.RoundTripper used to make all +// requests. +func newAMPCacheRendezvous(broker, cache, front string, transport http.RoundTripper) (*ampCacheRendezvous, error) { + brokerURL, err := url.Parse(broker) + if err != nil { + return nil, err + } + var cacheURL *url.URL + if cache != "" { + var err error + cacheURL, err = url.Parse(cache) + if err != nil { + return nil, err + } + } + return &ampCacheRendezvous{ + brokerURL: brokerURL, + cacheURL: cacheURL, + front: front, + transport: transport, + }, nil +} + +func (r *ampCacheRendezvous) Exchange(encPollReq []byte) ([]byte, error) { + log.Println("Negotiating via AMP cache rendezvous...") + log.Println("Broker URL:", r.brokerURL) + log.Println("AMP cache URL:", r.cacheURL) + log.Println("Front domain:", r.front) + + // Suffix the path with the broker's client registration handler. + reqURL := r.brokerURL.ResolveReference(&url.URL{Path: "client"}) + req, err := http.NewRequest("POST", reqURL.String(), bytes.NewReader(encPollReq)) + if err != nil { + return nil, err + } + + if r.front != "" { + // Do domain fronting. Replace the domain in the URL's with the + // front, and store the original domain the HTTP Host header. + req.Host = req.URL.Host + req.URL.Host = r.front + } + + resp, err := r.transport.RoundTrip(req) + if err != nil { + return nil, err + } + defer resp.Body.Close() + + log.Printf("AMP cache rendezvous response: %s", resp.Status) + if resp.StatusCode != http.StatusOK { + return nil, errors.New(BrokerErrorUnexpected) + } + + return limitedRead(resp.Body, readLimit) +} diff --git a/client/lib/snowflake.go b/client/lib/snowflake.go index f643c0a..0fc7671 100644 --- a/client/lib/snowflake.go +++ b/client/lib/snowflake.go @@ -39,7 +39,8 @@ type Transport struct { // iceAddresses are the STUN/TURN urls needed for WebRTC negotiation // keepLocalAddresses is a flag to enable sending local network addresses (for testing purposes) // max is the maximum number of snowflakes the client should gather for each SOCKS connection -func NewSnowflakeClient(brokerURL, frontDomain string, iceAddresses []string, keepLocalAddresses bool, max int) (*Transport, error) { +func NewSnowflakeClient(brokerURL, ampCacheURL, frontDomain string, + iceAddresses []string, keepLocalAddresses bool, max int) (*Transport, error) { log.Println("\n\n\n --- Starting Snowflake Client ---") @@ -57,9 +58,9 @@ func NewSnowflakeClient(brokerURL, frontDomain string, iceAddresses []string, ke log.Printf("url: %v", strings.Join(server.URLs, " ")) } - // Use potentially domain-fronting broker to rendezvous. + // Rendezvous with broker using the given parameters. broker, err := NewBrokerChannel( - brokerURL, frontDomain, CreateBrokerTransport(), + brokerURL, ampCacheURL, frontDomain, CreateBrokerTransport(), keepLocalAddresses) if err != nil { return nil, err diff --git a/client/snowflake.go b/client/snowflake.go index af9c2e4..ef06a2d 100644 --- a/client/snowflake.go +++ b/client/snowflake.go @@ -94,6 +94,7 @@ func main() { iceServersCommas := flag.String("ice", "", "comma-separated list of ICE servers") brokerURL := flag.String("url", "", "URL of signaling broker") frontDomain := flag.String("front", "", "front domain") + ampCacheURL := flag.String("ampcache", "", "URL of AMP cache to use as a proxy for signaling") logFilename := flag.String("log", "", "name of log file") logToStateDir := flag.Bool("log-to-state-dir", false, "resolve the log file relative to tor's pt state dir") keepLocalAddresses := flag.Bool("keep-local-addresses", false, "keep local LAN address ICE candidates") @@ -140,7 +141,7 @@ func main() { iceAddresses := strings.Split(strings.TrimSpace(*iceServersCommas), ",") - transport, err := sf.NewSnowflakeClient(*brokerURL, *frontDomain, iceAddresses, + transport, err := sf.NewSnowflakeClient(*brokerURL, *ampCacheURL, *frontDomain, iceAddresses, *keepLocalAddresses || *oldKeepLocalAddresses, *max) if err != nil { log.Fatal("Failed to start snowflake transport: ", err)

1 0

[snowflake/main] Use a URL with a Host component in BrokerChannel tests.
by dcf＠torproject.org 05 Aug '21

05 Aug '21

commit 191510c416db6b0229e62cc2b869aaf3cee907fa Author: David Fifield <david(a)bamsoftware.com> Date: Sun Jul 18 11:44:43 2021 -0600 Use a URL with a Host component in BrokerChannel tests. The tests were using a broker URL of "test.broker" (i.e., a schema-less, host-less, relative path), and running assertions on the value of b.url.Path. This is strange, especially in tests regarding domain fronting, where we care about b.url.Host, not b.url.Path. This commit changes the broker URL to "http://test.broker" and changes tests to check b.url.Host. I also added an additional assertion for an empty b.Host in the non-domain-fronted case. --- client/lib/lib_test.go | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/client/lib/lib_test.go b/client/lib/lib_test.go index c31a1a6..03c53dd 100644 --- a/client/lib/lib_test.go +++ b/client/lib/lib_test.go @@ -198,24 +198,25 @@ func TestSnowflakeClient(t *testing.T) { } Convey("Construct BrokerChannel with no front domain", func() { - b, err := NewBrokerChannel("test.broker", "", transport, false) + b, err := NewBrokerChannel("http://test.broker", "", transport, false) So(b.url, ShouldNotBeNil) So(err, ShouldBeNil) - So(b.url.Path, ShouldResemble, "test.broker") + So(b.Host, ShouldResemble, "") + So(b.url.Host, ShouldResemble, "test.broker") So(b.transport, ShouldNotBeNil) }) Convey("Construct BrokerChannel *with* front domain", func() { - b, err := NewBrokerChannel("test.broker", "front", transport, false) + b, err := NewBrokerChannel("http://test.broker", "front", transport, false) So(b.url, ShouldNotBeNil) So(err, ShouldBeNil) - So(b.url.Path, ShouldResemble, "test.broker") + So(b.Host, ShouldResemble, "test.broker") So(b.url.Host, ShouldResemble, "front") So(b.transport, ShouldNotBeNil) }) Convey("BrokerChannel.Negotiate responds with answer", func() { - b, err := NewBrokerChannel("test.broker", "", transport, false) + b, err := NewBrokerChannel("http://test.broker", "", transport, false) So(err, ShouldBeNil) answer, err := b.Negotiate(fakeOffer) So(err, ShouldBeNil) @@ -224,7 +225,7 @@ func TestSnowflakeClient(t *testing.T) { }) Convey("BrokerChannel.Negotiate fails", func() { - b, err := NewBrokerChannel("test.broker", "", + b, err := NewBrokerChannel("http://test.broker", "", &MockTransport{http.StatusOK, []byte(`{"error": "no snowflake proxies currently available"}`)}, false) So(err, ShouldBeNil) @@ -234,7 +235,7 @@ func TestSnowflakeClient(t *testing.T) { }) Convey("BrokerChannel.Negotiate fails with unexpected error", func() { - b, err := NewBrokerChannel("test.broker", "", + b, err := NewBrokerChannel("http://test.broker", "", &MockTransport{http.StatusInternalServerError, []byte("\n")}, false) So(err, ShouldBeNil) @@ -245,7 +246,7 @@ func TestSnowflakeClient(t *testing.T) { }) Convey("BrokerChannel.Negotiate fails with large read", func() { - b, err := NewBrokerChannel("test.broker", "", + b, err := NewBrokerChannel("http://test.broker", "", &MockTransport{http.StatusOK, make([]byte, readLimit+1)}, false) So(err, ShouldBeNil)

1 0

[snowflake/main] Change the representation of domain fronting in HTTP rendezvous.
by dcf＠torproject.org 05 Aug '21

05 Aug '21

commit 55f4814dfb5c196bb66416d4f3ba367498602489 Author: David Fifield <david(a)bamsoftware.com> Date: Sun Jul 18 10:39:51 2021 -0600 Change the representation of domain fronting in HTTP rendezvous. Formerly, BrokerChannel represented the broker URL and possible domain fronting as bc.url *url.URL bc.Host string That is, bc.url is the URL of the server which we contact directly, and bc.Host is the Host header to use in the request. With no domain fronting, bc.url points directly at the broker itself, and bc.Host is blank. With domain fronting, we do the following reshuffling: if front != "" { bc.Host = bc.url.Host bc.url.Host = front } That is, we alter bc.url to reflect that the server to which we send requests directly is the CDN, not the broker, and store the broker's own URL in the HTTP Host header. The above representation was always confusing to me, because in my mental model, we are always conceptually communicating with the broker; but we may optionally be using a CDN proxy in the middle. The new representation is bc.url *url.URL bc.front string bc.url is the URL of the broker itself, and never changes. bc.front is the optional CDN front domain, and likewise never changes after initialization. When domain fronting is in use, we do the swap in the http.Request struct, not in BrokerChannel itself: if bc.front != "" { request.Host = request.URL.Host request.URL.Host = bc.front } Compare to the representation in meek-client: https://gitweb.torproject.org/pluggable-transports/meek.git/tree/meek-clien… var options struct { URL string Front string } https://gitweb.torproject.org/pluggable-transports/meek.git/tree/meek-clien… if ok { // if front is set info.Host = info.URL.Host info.URL.Host = front } --- client/lib/lib_test.go | 12 ++++++------ client/lib/rendezvous.go | 23 +++++++++++------------ 2 files changed, 17 insertions(+), 18 deletions(-) diff --git a/client/lib/lib_test.go b/client/lib/lib_test.go index 03c53dd..9087eed 100644 --- a/client/lib/lib_test.go +++ b/client/lib/lib_test.go @@ -172,14 +172,14 @@ func TestSnowflakeClient(t *testing.T) { Convey("Dialers", t, func() { Convey("Can construct WebRTCDialer.", func() { - broker := &BrokerChannel{Host: "test"} + broker := &BrokerChannel{front: "test"} d := NewWebRTCDialer(broker, nil, 1) So(d, ShouldNotBeNil) So(d.BrokerChannel, ShouldNotBeNil) - So(d.BrokerChannel.Host, ShouldEqual, "test") + So(d.BrokerChannel.front, ShouldEqual, "test") }) SkipConvey("WebRTCDialer can Catch a snowflake.", func() { - broker := &BrokerChannel{Host: "test"} + broker := &BrokerChannel{} d := NewWebRTCDialer(broker, nil, 1) conn, err := d.Catch() So(conn, ShouldBeNil) @@ -201,8 +201,8 @@ func TestSnowflakeClient(t *testing.T) { b, err := NewBrokerChannel("http://test.broker", "", transport, false) So(b.url, ShouldNotBeNil) So(err, ShouldBeNil) - So(b.Host, ShouldResemble, "") So(b.url.Host, ShouldResemble, "test.broker") + So(b.front, ShouldResemble, "") So(b.transport, ShouldNotBeNil) }) @@ -210,8 +210,8 @@ func TestSnowflakeClient(t *testing.T) { b, err := NewBrokerChannel("http://test.broker", "front", transport, false) So(b.url, ShouldNotBeNil) So(err, ShouldBeNil) - So(b.Host, ShouldResemble, "test.broker") - So(b.url.Host, ShouldResemble, "front") + So(b.url.Host, ShouldResemble, "test.broker") + So(b.front, ShouldResemble, "front") So(b.transport, ShouldNotBeNil) }) diff --git a/client/lib/rendezvous.go b/client/lib/rendezvous.go index b89f432..caa4ae4 100644 --- a/client/lib/rendezvous.go +++ b/client/lib/rendezvous.go @@ -32,10 +32,8 @@ const ( // Signalling Channel to the Broker. type BrokerChannel struct { - // The Host header to put in the HTTP request (optional and may be - // different from the host name in URL). - Host string url *url.URL + front string // Optional front domain to replace url.Host in requests. transport http.RoundTripper // Used to make all requests. keepLocalAddresses bool NATType string @@ -61,14 +59,12 @@ func NewBrokerChannel(broker string, front string, transport http.RoundTripper, return nil, err } log.Println("Rendezvous using Broker at:", broker) - bc := new(BrokerChannel) - bc.url = targetURL - if front != "" { // Optional front domain. + if front != "" { log.Println("Domain fronting using:", front) - bc.Host = bc.url.Host - bc.url.Host = front } - + bc := new(BrokerChannel) + bc.url = targetURL + bc.front = front bc.transport = transport bc.keepLocalAddresses = keepLocalAddresses bc.NATType = nat.NATUnknown @@ -92,7 +88,7 @@ func limitedRead(r io.Reader, limit int64) ([]byte, error) { func (bc *BrokerChannel) Negotiate(offer *webrtc.SessionDescription) ( *webrtc.SessionDescription, error) { log.Println("Negotiating via BrokerChannel...\nTarget URL: ", - bc.Host, "\nFront URL: ", bc.url.Host) + bc.url.Host, "\nFront URL: ", bc.front) // Ideally, we could specify an `RTCIceTransportPolicy` that would handle // this for us. However, "public" was removed from the draft spec. // See https://developer.mozilla.org/en-US/docs/Web/API/RTCConfiguration#RTCIceTra… @@ -126,8 +122,11 @@ func (bc *BrokerChannel) Negotiate(offer *webrtc.SessionDescription) ( if nil != err { return nil, err } - if "" != bc.Host { // Set true host if necessary. - request.Host = bc.Host + if bc.front != "" { + // Do domain fronting. Replace the domain in the URL's with the + // front, and store the original domain the HTTP Host header. + request.Host = request.URL.Host + request.URL.Host = bc.front } resp, err := bc.transport.RoundTrip(request) if nil != err {

1 0

[tor-browser-build/maint-10.5] Tor Browser 10.5.4 release preparations
by boklm＠torproject.org 05 Aug '21

05 Aug '21

commit 9b3892532ac96593997afbf16ca8642cd84a51a6 Author: Matthew Finkel <sysrqb(a)torproject.org> Date: Wed Aug 4 18:15:11 2021 +0000 Tor Browser 10.5.4 release preparations Version bumps and Changelog update --- projects/firefox/config | 2 +- projects/go/config | 4 ++-- projects/tor-browser/Bundle-Data/Docs/ChangeLog.txt | 11 +++++++++++ projects/tor-browser/config | 4 ++-- rbm.conf | 4 ++-- 5 files changed, 18 insertions(+), 7 deletions(-) diff --git a/projects/firefox/config b/projects/firefox/config index d62d9ab..1837c5c 100644 --- a/projects/firefox/config +++ b/projects/firefox/config @@ -8,7 +8,7 @@ git_submodule: 1 gpg_keyring: torbutton.gpg var: - firefox_platform_version: 78.12.0 + firefox_platform_version: 78.13.0 firefox_version: '[% c("var/firefox_platform_version") %]esr' torbrowser_branch: 10.5 branding_directory: 'browser/branding/alpha' diff --git a/projects/go/config b/projects/go/config index 8a67ae8..9572cd2 100644 --- a/projects/go/config +++ b/projects/go/config @@ -1,5 +1,5 @@ # vim: filetype=yaml sw=2 -version: 1.15.13 +version: 1.15.14 filename: '[% project %]-[% c("version") %]-[% c("var/build_id") %].tar.gz' var: @@ -118,7 +118,7 @@ input_files: enable: '[% ! c("var/linux") %]' - URL: 'https://golang.org/dl/go[% c("version") %].src.tar.gz' name: go - sha256sum: 99069e7223479cce4553f84f874b9345f6f4045f27cf5089489b546da619a244 + sha256sum: 60a4a5c48d63d0a13eca8849009b624629ff429c8bc5d1a6a8c3c4da9f34e70a - URL: 'https://golang.org/dl/go[% c("var/go14_version") %].src.tar.gz' name: go14 sha256sum: 9947fc705b0b841b5938c48b22dc33e9647ec0752bae66e50278df4f23f64959 diff --git a/projects/tor-browser/Bundle-Data/Docs/ChangeLog.txt b/projects/tor-browser/Bundle-Data/Docs/ChangeLog.txt index 5eca79f..bc768d0 100644 --- a/projects/tor-browser/Bundle-Data/Docs/ChangeLog.txt +++ b/projects/tor-browser/Bundle-Data/Docs/ChangeLog.txt @@ -1,3 +1,14 @@ +Tor Browser 10.5.4 -- August 10 2021 + * Windows + OS X + Linux + * Update Firefox to 78.13.0esr + * Update NoScript to 11.2.11 + * Bug 40041: Remove V2 Deprecation banner on about:tor for desktop [torbutton] + * Bug 40506: Saved Logins not available in 10.5 [tor-browser] + * Bug 40524: Update DuckDuckGo onion site URL in search preferences and onboarding [tor-browser] + * Build System + * Windows + OS X + Linux + * Update Go to 1.15.14 + Tor Browser 10.5.3 -- July 17 2021 * Android * Update HTTPS Everywhere to 2021.7.13 diff --git a/projects/tor-browser/config b/projects/tor-browser/config index 1045d90..4091361 100644 --- a/projects/tor-browser/config +++ b/projects/tor-browser/config @@ -78,9 +78,9 @@ input_files: enable: '[% ! c("var/android") %]' - filename: Bundle-Data enable: '[% ! c("var/android") %]' - - URL: https://addons.cdn.mozilla.net/user-media/addons/722/noscript_security_suit… + - URL: https://addons.cdn.mozilla.net/user-media/addons/722/noscript_security_suit… name: noscript - sha256sum: 830a25dad07327ae220b4740ea834b0abe715e9ef3dabc326bf7fef2c5af1efb + sha256sum: b833e81823986646dbc473ebbee987bd47757fbe79c9d1720150f08ba6ca9ba9 - filename: 'RelativeLink/start-tor-browser.desktop' enable: '[% c("var/linux") %]' - filename: 'RelativeLink/execdesktop' diff --git a/rbm.conf b/rbm.conf index 5972405..95ec20a 100644 --- a/rbm.conf +++ b/rbm.conf @@ -57,11 +57,11 @@ buildconf: git_signtag_opt: '-s' var: - torbrowser_version: '10.5.3' + torbrowser_version: '10.5.4' torbrowser_build: 'build1' torbrowser_incremental_from: - - 10.0.18 - 10.5 + - 10.5.2 project_name: tor-browser multi_lingual: 0 build_mar: 1

1 0

[translation/tbmanual-contentspot] https://gitweb.torproject.org/translation.git/commit/?h=tbmanual-contentspot
by translation＠torproject.org 05 Aug '21

05 Aug '21

commit 63034831c1f439abc437e0a0366976dc0c0e646a Author: Translation commit bot <translation(a)torproject.org> Date: Thu Aug 5 20:46:45 2021 +0000 https://gitweb.torproject.org/translation.git/commit/?h=tbmanual-contentspot --- contents+ar.po | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/contents+ar.po b/contents+ar.po index e54719d355..e913387c6b 100644 --- a/contents+ar.po +++ b/contents+ar.po @@ -397,6 +397,9 @@ msgid "" "[gettor@torproject.org](mailto:gettor@torproject.org) with the words " "\"windows zh\" in it." msgstr "" +"على سبيل المثال، للحصول على روابط لتنزيل متصفح Tor باللغة الصينية لنظام " +"التشغيل ويندوز، أرسل بريدًا إلكترونيًا إلى [gettor(a)torproject.org](mailto: " +"gettor(a)torproject.org) بداخله الكلمات \"windows zh\"." #: https//tb-manual.torproject.org/installation/ #: (content/installation/contents+en.lrtopic.title) @@ -629,7 +632,7 @@ msgstr "" #: (content/running-tor-browser/contents+en.lrtopic.body) msgid "" "Once clicked, a status bar will appear, showing Tor's connection progress." -msgstr "" +msgstr "بمجرد النقر عليه، سيظهر شريط حالة يعرض تقدم اتصال Tor." #: https//tb-manual.torproject.org/running-tor-browser/ #: (content/running-tor-browser/contents+en.lrtopic.body) @@ -654,6 +657,8 @@ msgid "" "<img class=\"col-md-6\" src=\"../../static/images/configure.png\" " "alt=\"Click 'Tor Network Settings' to adjust network settings.\">" msgstr "" +"<img class=\"col-md-6\" src=\"../../static/images/configure.png\" " +"alt=\"Click 'Tor Network Settings' to adjust network settings.\">" #: https//tb-manual.torproject.org/running-tor-browser/ #: (content/running-tor-browser/contents+en.lrtopic.body) @@ -701,7 +706,7 @@ msgstr "" #: https//tb-manual.torproject.org/running-tor-browser/ #: (content/running-tor-browser/contents+en.lrtopic.body) msgid "### OTHER OPTIONS" -msgstr "" +msgstr "### خيارات أخرى" #: https//tb-manual.torproject.org/running-tor-browser/ #: (content/running-tor-browser/contents+en.lrtopic.body) @@ -719,7 +724,7 @@ msgstr "إذا أمكن ، اطلب من مسؤول الشبكة لديك الم #: https//tb-manual.torproject.org/running-tor-browser/ #: (content/running-tor-browser/contents+en.lrtopic.body) msgid "If your connection does not use a proxy, click \"Connect\"." -msgstr "" +msgstr "إذا كان اتصالك لا يستخدم وكيلاً، فانقر على \"اتصال\"." #: https//tb-manual.torproject.org/running-tor-browser/ #: (content/running-tor-browser/contents+en.lrtopic.body) @@ -1095,6 +1100,8 @@ msgid "" "<img class=\"col-md-6\" src=\"../../static/images/request-a-bridge.png\" " "alt=\"Request a bridge from torproject.org\">" msgstr "" +"<img class=\"col-md-6\" src=\"../../static/images/request-a-bridge.png\" " +"alt=\"Request a bridge from torproject.org\">" #: https//tb-manual.torproject.org/bridges/ #: (content/bridges/contents+en.lrtopic.body) @@ -1128,6 +1135,8 @@ msgid "" "<img class=\"col-md-6\" src=\"../../static/images/tor-launcher-custom-" "bridges.png\" alt=\"Enter custom bridge addresses\">" msgstr "" +"<img class=\"col-md-6\" src=\"../../static/images/tor-launcher-custom-" +"bridges.png\" alt=\"Enter custom bridge addresses\">" #: https//tb-manual.torproject.org/bridges/ #: (content/bridges/contents+en.lrtopic.body) @@ -1617,6 +1626,8 @@ msgid "" "<img class=\"col-md-6\" src=\"../../static/images/client-auth.png\" " "alt=\"Client Authorization\">" msgstr "" +"<img class=\"col-md-6\" src=\"../../static/images/client-auth.png\" " +"alt=\"Client Authorization\">" #: https//tb-manual.torproject.org/onion-services/ #: (content/onion-services/contents+en.lrtopic.body)

1 0