Shelikhoo brought up an interesting point at today's meeting. Snowflake
uses WebRTC data channels on the client–proxy link. Data channels are
SCTP in DTLS. They are by default fully reliable and in-order, like TCP,
and that is the way we currently use them. But SCTP and in turn WebRTC
data channels also have "partial reliability" and "unordered" options
that might be a better fit for our uses.
http://meetbot.debian.net/tor-meeting/2023/tor-meeting.2023-03-16-15.57.log…
16:25:29 <shelikhoo> webrtc(SCTP to be specific) can be set to not retransmit packet
16:25:34 <shelikhoo> and deliver packet out of order
16:25:43 <shelikhoo> to application
16:26:14 <dcf1> I did not know about that. Do we use it that way in Snowflake?
16:27:28 <shelikhoo> it is called "unreliable mode" https://pkg.go.dev/github.com/pion/webrtc/v3#DataChannel.MaxRetransmits
16:29:28 <dcf1> Ok. I might have missed an issue, is that mode used in Snowflake? Or could it be?
16:30:27 <shelikhoo> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
16:30:54 <shelikhoo> https://pkg.go.dev/github.com/pion/webrtc/v3#DataChannelInit
16:31:18 <shelikhoo> Maybe no...? but we can enable "unreliable mode"
## Background
The specification of WebRTC data channels, RFC 8831, requires the SCTP
implementation to support limiting transmissions by either time or
number.
https://www.rfc-editor.org/rfc/rfc8831
> SCTP, as specified in [RFC4960] with the partial reliability extension
> (PR-SCTP) defined in [RFC3758] and the additional policies defined in
> [RFC7496], provides multiple streams natively with reliable, and the
> relevant partially reliable, delivery modes for user messages.
> ...
> The partial reliability extension defined in [RFC3758] MUST be
> supported. In addition to the timed reliability PR-SCTP policy defined
> in [RFC3758], the limited retransmission policy defined in [RFC7496]
> MUST be supported.
Orthogonal to reliability, the SCTP implementation must support ordered
or unordered delivery of messages.
https://www.rfc-editor.org/rfc/rfc8831#name-sctp-protocol-consideration
> This SCTP stack and its upper layer MUST support the usage of multiple
> SCTP streams. A user message can be sent ordered or unordered and with
> partial or full reliability.
Removing retransmissions and ordering restrictions makes a data channel
into a datagram delivery service that preserves message boundaries, but
not ordering, and does not guarantee delivery.
https://www.rfc-editor.org/rfc/rfc8831#name-sctp-protocol-consideration
> Limiting the number of retransmissions to zero, combined with
> unordered delivery, provides a UDP-like service where each user
> message is sent exactly once and delivered in the order received.
The WebRTC Data Channel Establishment Protocol's DATA_CHANNEL_OPEN
message is where the ordering and retransmission limits are specified.
(Note partial reliability and ordering are two separate concepts, and
that you can only limit retransmissions by time or number, not both.)
https://www.rfc-editor.org/rfc/rfc8832.html#name-data_channel_open-message
> DATA_CHANNEL_RELIABLE (0x00):
> The data channel provides a reliable in-order bidirectional
> communication.
> DATA_CHANNEL_RELIABLE_UNORDERED (0x80):
> The data channel provides a reliable unordered bidirectional
> communication.
> DATA_CHANNEL_PARTIAL_RELIABLE_REXMIT (0x01):
> The data channel provides a partially reliable in-order
> bidirectional communication. User messages will not be retransmitted
> more times than specified in the Reliability Parameter.
> DATA_CHANNEL_PARTIAL_RELIABLE_REXMIT_UNORDERED (0x81):
> The data channel provides a partially reliable unordered
> bidirectional communication. User messages will not be retransmitted
> more times than specified in the Reliability Parameter.
> DATA_CHANNEL_PARTIAL_RELIABLE_TIMED (0x02):
> The data channel provides a partially reliable in-order
> bidirectional communication. User messages might not be transmitted
> or retransmitted after a specified lifetime given in milliseconds in
> the Reliability Parameter. This lifetime starts when providing the
> user message to the protocol stack.
> DATA_CHANNEL_PARTIAL_RELIABLE_TIMED_UNORDERED (0x82):
> The data channel provides a partially reliable unordered
> bidirectional communication. User messages might not be transmitted
> or retransmitted after a specified lifetime given in milliseconds in
> the Reliability Parameter. This lifetime starts when providing the
> user message to the protocol stack.
Though in SCTP the unordered flag (U) may vary per DATA chunk, data
channels can only be entirely ordered or entirely unordered.
https://www.rfc-editor.org/rfc/rfc9260#name-ordered-and-unordered-deliv
The JavaScript API exposes these settings in the
RTCPeerConnection.createDataChannel method:
https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/createDa…
> `ordered` (Optional)
> Indicates whether or not messages sent on the RTCDataChannel are
> required to arrive at their destination in the same order in which
> they were sent (`true`), or if they're allowed to arrive
> out-of-order (`false`). Default: `true`.
> `maxPacketLifeTime` (Optional)
> The maximum number of milliseconds that attempts to transfer a
> message may take in unreliable mode. While this value is a 16-bit
> unsigned number, each user agent may clamp it to whatever maximum it
> deems appropriate. Default: `null`.
> `maxRetransmits` (Optional)
> The maximum number of times the user agent should attempt to
> retransmit a message which fails the first time in unreliable mode.
> While this value is a 16-bit unsigned number, each user agent may
> clamp it to whatever maximum it deems appropriate. Default: `null`.
Similar options are exposed in Pion PeerConnection.CreateDataChannel and
the DataChannelInit struct:
https://pkg.go.dev/github.com/pion/webrtc/v3#DataChannelInit
> // Ordered indicates if data is allowed to be delivered out of order.
> // The default value of true, guarantees that data will be delivered
> // in order.
> Ordered *bool
> // MaxPacketLifeTime limits the time (in milliseconds) during which the
> // channel will transmit or retransmit data if not acknowledged. This
> // value may be clamped if it exceeds the maximum value supported.
> MaxPacketLifeTime *uint16
> // MaxRetransmits limits the number of times a channel will retransmit
> // data if not successfully delivered. This value may be clamped if it
> // exceeds the maximum value supported.
> MaxRetransmits *uint16
## Possibilities for Snowflake
Currently in Snowflake, we treat the data channel as if it were a TCP
stream: we ignore message boundaries and concatenate all messages in
order. What we transmit over this stream, though, is a sequence of
discrete KCP packets (and possible padding), delimited using length
prefixes:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
The boundaries of encapsulated KCP packets do not necessarily correspond
to SCTP/data channel message boundaries.
We may be able to increase efficiency by using data channel messages
directly, one KCP packet per message. Because KCP has its own
retransmission and reordering facilities, we could completely turn off
reliability and ordering at the data channel layer.
One of the reasons we use the packet encapsulation we use now is to
permit shaping the size of network packets. But matching data channel
messages 1:1 with KCP packet does not necessarily negatively affect
traffic shaping capability. We could put a simple length prefix at the
head of each message that indicates the effective payload length, with
the rest being padding. If the traffic schedule calls for a payload
smaller than an immediately available packet, an all-padding message can
be sent.
Such a change would require changes at the proxy. Proxies would need a
second mode to take message boundaries directly from the data channel,
rather than concatenating messages and decoding message boundaries from
the joined-up stream. Proxies could backward-compatibly forward the
messages to the bridge over WebSocket, applying the packet encapsulation
at the proxy rather than the client. Or (requiring a change at the
bridge too), we could use WebRTC on the proxy–bridge link and similarly
send packets with unordered and unreliable data channels.
Hi! I was wondering if ANY of the dark or deep web browsers featured on the Chrome/Google Play store are trustworthy in and of themselves, and if Tor can be accessed with them, or if any use Tor automatically. I have a Chromebook, and I know that Tor does not work with it as you have yet to develop a version for it. Why is it that I think that it will be impossible to have Tor work safely with Chrome/Chromebooks; because Google is EVIL?! :-) Who do you trust for DNS? What is the world's most private and secure computer? I remember Australia having some cool guys build a n independent computer system, but these would not be available outside of Oz. Comments?
On Feb 24, 2023 at 7:00 AM, "anti-censorship-team-request(a)lists.torproject.org " <> wrote:
--------------------------IronVest---------------------------
Preview: Send anti-censorship-team mailing list submissions to anti-ce -->
SPAM? CLICK to BLOCK:
https://ironvest.com/app/0/#emails/r8196gs9ev5p@opayq.com/toggle
This email is Masked using Ironvest - it was sent from duck.com to
r8196gs9ev5p(a)opayq.com (your reply stays Masked). To protect your privacy, do
not forward this message, or add new recipients like CCs or BCCs.
Thanks for being an IronVest customer!
--------------------------IronVest---------------------------
Send anti-censorship-team mailing list submissions to
anti-censorship-team(a)lists.torproject.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.torproject.org/cgi-bin/mailman/listinfo/anti-censorship-team
or, via email, send a message with subject or body 'help' to
anti-censorship-team-request(a)lists.torproject.org
You can reach the person managing the list at
anti-censorship-team-owner(a)lists.torproject.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of anti-censorship-team digest..."
Today's Topics:
1. onbasca and rdsys (meskio)
----------------------------------------------------------------------
Message: 1
Date: Fri, 24 Feb 2023 12:47:19 +0100
From: meskio <meskio(a)torproject.org>
To: anti-censorship-team <anti-censorship-team(a)lists.torproject.org>
Subject: [anti-censorship-team] onbasca and rdsys
Message-ID: <167723923930.1535.1605523355422205109@localhost>
Content-Type: text/plain; charset="utf-8"
Hello,
Onbasca[0], a relay bandwidth, has just added support for testing bridges. I
have modified rdsys to use it[1]. onbasca produces a bandwidth ratio, on how
fast is a bridge compared to the rest of bridges (1 is the median), and rdsys
decides if distribute a bridge if the ratio is higher than a configured
threshold (right now 0.75) or if is jet untested. rdsys does ignore the onbasca
results if there is less then 50% of bridges with high enough ratio to be
distributed.
As this was needed by a sponsor I have already deployed it in production. I
wrote a survival guide:
https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Onbasca-Surv…
I tested it and I don't expect much problems, but as I will be AFK next week if
rdsys gives any problem is easy to roll back to the previous version of rdsys
without onbasca support. I left the previous binary of rdsys backend in
~/bin/rdsys-backend.old, so rolling back will be running as rdsys user in
polyanthum:
systemctl --user stop rdsys-backend
mv ~/bin/rdsys-backend.old ~/bin/rdsys-backend
systemdl --user start rdsys-backend
[0] https://gitlab.torproject.org/tpo/network-health/onbasca/
[1]
https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/merge_requests/76
--
meskio | https://meskio.net/
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
My contact info: https://meskio.net/crypto.txt
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Nos vamos a Croatan.
We have discussed how Snowflake uses WebRTC data channels (DTLS), while
some other WebRTC apps use media channels (DTLS-STRP), and how that
could be used in a classifier. See for example Section 2 of
https://censorbib.nymity.ch/#Fifield2016b. We have also entertained the
possibility of encoding Snowflake data into an audio/video stream, à la
FreeWave (https://censorbib.nymity.ch/#Houmansadr2013a) or CovertCast
(https://censorbib.nymity.ch/#McPherson2016a), in order to use media
channels in Snowflake.
I came across an API that may permit us to use media channel, without
the trouble of an audio/video codec. WebRTC Encoded Transform (formerly
Insertable Streams, I think) provides a way to get at and manipulate the
raw bytes of media frames.
https://w3c.github.io/webrtc-encoded-transform/
> This API defines an API surface for manipulating the bits on
> MediaStreamTracks being sent via an RTCPeerConnection.
The API defines RTCEncodedVideoFrame and RTCEncodedAudioFrame types that
give you an ArrayBuffer of the encoded media data:
https://w3c.github.io/webrtc-encoded-transform/#RTCEncodedVideoFrame-interf…
// New interfaces to define encoded video and audio frames. Will eventually
// re-use or extend the equivalent defined in WebCodecs.
[Exposed=(Window,DedicatedWorker)]
interface RTCEncodedVideoFrame {
readonly attribute RTCEncodedVideoFrameType type;
readonly attribute unsigned long timestamp;
attribute ArrayBuffer data;
RTCEncodedVideoFrameMetadata getMetadata();
};
https://w3c.github.io/webrtc-encoded-transform/#RTCEncodedAudioFrame-interf…
[Exposed=(Window,DedicatedWorker)]
interface RTCEncodedAudioFrame {
readonly attribute unsigned long timestamp;
attribute ArrayBuffer data;
RTCEncodedAudioFrameMetadata getMetadata();
};
One of the intended use cases of Encoded Transform is a end-to-end
encryption when then media streams pass through a middlebox that removes
and re-adds hop-by-hop DTLS-STRP layers:
https://webrtchacks.com/true-end-to-end-encryption-with-webrtc-insertable-s…https://jitsi.org/security/
> You can turn on end-to-end encryption (e2ee) as long as you are using
> Jitsi Meet on a browser with support for insertable streams.
The explainer document has a sample of inverting all the bits in an
encoded media frame:
https://github.com/w3c/webrtc-encoded-transform/blob/5c7ab84f4ce338f299172f…
// Receiver transform
function createReceiverTransform() {
return new TransformStream({
start() {},
flush() {},
async transform(encodedFrame, controller) {
// Reconstruct the original frame.
const view = new DataView(encodedFrame.data);
// Ignore the last 4 bytes
const newData = new ArrayBuffer(encodedFrame.data.byteLength - 4);
const newView = new DataView(newData);
// Negate all bits in the incoming frame, ignoring the
// last 4 bytes
for (let i = 0; i < encodedFrame.data.byteLength - 4; ++i)
newView.setInt8(i, ~view.getInt8(i));
encodedFrame.data = newData;
controller.enqueue(encodedFrame);
}
});
}
I am not clear on the difference between Encoded Transform, Insertable
Streams, and WebCodecs, but here are some possible browser support
tables:
https://chromestatus.com/feature/6321945865879552https://caniuse.com/?search=webcodechttps://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_API
Hello,
Onbasca[0], a relay bandwidth, has just added support for testing bridges. I
have modified rdsys to use it[1]. onbasca produces a bandwidth ratio, on how
fast is a bridge compared to the rest of bridges (1 is the median), and rdsys
decides if distribute a bridge if the ratio is higher than a configured
threshold (right now 0.75) or if is jet untested. rdsys does ignore the onbasca
results if there is less then 50% of bridges with high enough ratio to be
distributed.
As this was needed by a sponsor I have already deployed it in production. I
wrote a survival guide:
https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Onbasca-Surv…
I tested it and I don't expect much problems, but as I will be AFK next week if
rdsys gives any problem is easy to roll back to the previous version of rdsys
without onbasca support. I left the previous binary of rdsys backend in
~/bin/rdsys-backend.old, so rolling back will be running as rdsys user in
polyanthum:
systemctl --user stop rdsys-backend
mv ~/bin/rdsys-backend.old ~/bin/rdsys-backend
systemdl --user start rdsys-backend
[0] https://gitlab.torproject.org/tpo/network-health/onbasca/
[1] https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/merge_requests/76
--
meskio | https://meskio.net/
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
My contact info: https://meskio.net/crypto.txt
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Nos vamos a Croatan.
Hi Micah! Is the georgetown bridge is flaky as it seems? We have alerts
set up for the default bridges and it seems like yours alerts a lot
and the others alert rarely. It always comes back after a little while,
so this isn't a "hey your bridge is down" mail so much as a "what's up
with your bridge" mail.
I wonder if it is an issue with the network it's running on; or if it's an
issue with the bridge itself, e.g. maybe there is something in Tor's logs
that could help diagnose; or if it's something wonky about our alerts.
(I am cc'ing anti-censorship-team so others can coordinate and stay
in sync; note that this is a public list with archives.)
Thanks!
--Roger
----- Forwarded message from Monit <anti-censorship-monitor(a)nymity.ch> -----
Date: Thu, 19 Jan 2023 12:26:36 GMT
From: Monit <anti-censorship-monitor(a)nymity.ch>
To: anti-censorship-alerts(a)lists.torproject.org
Subject: [anti-censorship-alerts] monit alert -- Connection failed
default-bridge-georgetown
X-Mailer: Monit 5.26.0
Connection failed Service default-bridge-georgetown
Date: Thu, 19 Jan 2023 13:26:35
Action: alert
Host: server01
Description: failed protocol test [DEFAULT] at [209.148.46.65]:443 [TCP/IP] -- Connection timed out
_______________________________________________
anti-censorship-alerts mailing list
anti-censorship-alerts(a)lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/anti-censorship-alerts
----- End forwarded message -----
Greetings,
I hope this is an appropriate mailing list to discuss a technical issue
with Tor's Snowflake project. Please redirect me to the right place if not.
I am the original author and maintainer of the open source project,
Stuntman. Stuntman is an implementation of the STUN protocol, which
includes the STUN server. More details at www.stunprotocol.org. In short,
a STUN server helps bootstream direct "p2p" connections such as WebRTC
sessions or similar VOIP scenarios by allowing internet devices to
self-discover their own public IP address and obtain a (UDP) port for
communicating with another node.
I also run a public instance of a STUN server with the code at
stun.stunprotocol.org. It's been up and running for about 10 years
now. It's hosted on AWS. In recent years, the hosting bills for this
server have started to get on the high side, even with reserved instances.
The number of STUN queries it processes per day is now on the order of
hundreds of millions. The stunprotocol.org domain receives nearly a million
DNS queries on Route 53 daily. What used to cost a trivial number of
dollars to run is now starting to reach $1000 in annual service costs.
This isn't paid for by a corporation or well funded internet organization.
I pay this out of my personal pocket.
It's been a mystery what has been driving the increasing traffic to the
server - especially redundant requests from the same IPs. I was inspecting
the DNS logs the other day and started to investigate the nodes sending out
redundant DNS requests repetitively. Trying to understand why these nodes
wouldn't leverage DNS caching. And to my surprise, one of the IPs was
running a web server that presented a TOR landing page. That led me to
discover this discussion online:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
And a quick inspection of the Snowflake code leads me to find that
stun.stunprotocol.org is the default STUN server for Snowflake proxy and
listed throughout the documentation as well.
While the Snowflake project has good intentions, it doesn't appear to take
my hosting costs into consideration. I'm hoping we can have a good
discussion on the following:
1) How many snowflake clients and proxies are active and how many STUN
requests are each generating towards stunprotocol.org? Do we think the
entire worldwide usage of Snowflake could be responsible for millions of
STUN queries to stunprotocol.org per day?
2) Expected number of DNS queries (it's a 3-day TTL on these DNS entries,
so it blows my mind that there are so many redundant requests). Does Pion
or any other part of the Snowflake code tend to go direct to the namespace
server itself?
3) Removing stun.stunprotocol.org as the default STUN server.
OR...
4) Alternatively, I'm always open to accepting donations to help run the
service costs of stunprotocol.org. I'm definitely not getting rich running
this thing.
Thanks,
John Selbie
Linus and I have been putting in a lot of time on the Snowflake bridge
over the past week or so to improve performance. The increase in users
after the blocking of Tor in Russia last September (the one that led us
to the multi-tor architecture) was large, but this recent increase is
many times larger. We've cleared out the major bottlenecks and since two
days ago days the bridge is finally meeting the additional demand. But
it's close: during the busiest times of day the CPU and RAM resources
are nearly 100% consumed, and the need for horizontal scaling still
exists.
There have been a lot of changes, both major and minor. You can find
listings at:
https://gitlab.torproject.org/tpo/network-health/metrics/timeline/-/blob/f5…https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…
The most important optimizations have been:
- Reduce allocations in WebSocket reads and writes.
- Use more than one KCP state machine.
- Conserve ephemeral ports in localhost communication.
- Disable nftables connection tracking.
I attached a graph of bandwidth on the bridge. You can see that between
24 Sep and 02 Oct the daily peaks were unnaturally flattened. The daily
shutdowns in Iran caused a paradoxical *increase* in bandwidth, probably
because it relieved congestion in the system for the fewer remaining
users. In the past two days the shape of the graph looks more natural,
and a shutdown decreases bandwidth as you would expect. At its peak,
bandwidth is above 4 Gbps. The daily lows are higher than the highest
highs of two weeks ago.
## Reduce allocations in writing packets
The code forreading and writing encapsulated packets from WebSocket
connections was doing unnecessary memory allocations. Some implicit,
like a 32 KB buffer being created by io.Copy, and some explicit, like
the intentional packet copies being made in the QueuePacketConn type.
Reducing allocations makes the garbage collector run less often, and
there's also a small benefit from reduced buffer copies.
https://gitlab.torproject.org/dcf/snowflake/-/commit/42ea5ff60ce9b3a0dff305…https://gitlab.torproject.org/dcf/snowflake/-/commit/4d4fad30c429bba9062d1b…https://gitlab.torproject.org/dcf/snowflake/-/commit/57c9aa3477513daf433476…
## Use more than one KCP state machine
Though most part of snowflake-server are multi-threaded and can scale
across many CPUs, the central KCP packet scheduler was limited to one
CPU. Because we have a session identity (the client ID) separate from
any KCP-specific identity, it's not hard to partition client packets
across separate KCP instances by a hash of the client ID. Expanding the
number of KCPs from 1 to 2 was enough to relive this bottleneck.
https://gitlab.torproject.org/dcf/snowflake/-/commit/17dc8cad8299eae76c4019…
## Conserve ephemeral ports in localhost communication
The pluggable transports model relies heavily on localhost TCP sockets.
The number of users had increased enough that it was sometimes
exhausting the range of port numbers usable for distinct localhost
4-tuples. The kernel's errno for this situation is EADDRNOTAVAIL when
you try to connect; it manifests variously in different programs as
"cannot assign requested address" or "no free ports," generally leading
to a terminated connection. We mitigated this problem by having
different programs use different localhost source IP addresses (e.g.
127.0.1.0/24, 127.0.2.0/24, etc.) to expand the space of distinct
4-tuples. In a couple of cases this was done in a hacky way that will
need to be revisited, by hardcoding a source address range in the source
code.
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…https://gitlab.torproject.org/dcf/snowflake/-/commit/d8183ff9680ac9f92e888a…https://gitlab.torproject.org/dcf/extor-static-cookie/-/commit/0d9078d6aad8…
## Disable nftables connection tracking
We took care of it before it became a problem, but we found it necessary
to disable connection tracking in the firewall. The number of tracked
connections was getting close to the limit, and past the limit, packets
just get dropped.
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snow…