<html><head></head><body><div class="ydp558bb730yahoo-style-wrap" style="font-family: Helvetica Neue, Helvetica, Arial, sans-serif; font-size: 16px;"><div><div><font color="#26282a" size="2"><span>David,</span></font></div></div><div><font color="#26282a" size="2"><span><br></span></font></div><div><font color="#26282a" size="2"><span>Excellent documentation of your loadbalanced Snowflake endeavors!</span></font></div><div id="ydp558bb730yahoo_quoted_3205270459" class="ydp558bb730yahoo_quoted" style="font-size: 16px;"><div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div><br></div>
<div>> The DNS record for the Snowflake bridge was switched to a temporary staging server, running the load balancing setup, at 2022-01-25 17:41:00. We were debugging some initial problems until 2022-01-25 18:47:00. You can read about it here:<br></div><div><br clear="none">> <a shape="rect" href="https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40095#note_2772325" rel="nofollow" target="_blank">https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40095#note_2772325</a><br clear="none"><br>It's nice to see that the Snowflake daemon offers a native configuration option for LimitNOFile. I ran into a similar issue with my initial loadbalanced Tor Relay Nodes that was solved at the O/S level using ulimit. It would be nice if torrc had a similar option.</div><div><br></div><div>From your documentation, it sounds like you're running everything on the same machine? When expanding to additional machines, similar to the file limit issue, you'll have to expand the usable ports as well.</div><div><br></div><div>I'd like to see more of your HAProxy configuration. Do you not have to use transparent proxy mode with Snowflake instances as you do with Tor Relay instances? I hadn't realized HAProxy had a client timeout. Thank you for that tidbit. And thank you for referencing my comments as well.<br><br clear="none">> Snowflake sessions are now using the staging bridge, except for those that started before the change happened and haven't finished yet, and perhaps some proxies that still have the IP address of the production bridge in their DNS cache. I am not sure yet what will happen with metrics, but we'll see after a few days.</div><div><br></div><div><span>Currently, as I only use IPv4, I can't offer much insight as to the lack of IPv6 connections being reported (that's what my logs report, too). </span>Your Heartbeat messages are looking good with a symmetric balance of connections and data. They look very similar to my Heartbeat logs; except, you can tell you offer more computing power, which is great to see extrapolated! I've found that the Heartbeat logs are key to knowing the health of your loadbalanced Tor implementation. You might consider setting up syslog with a Snowflake filter to aggregate your Snowflake logs for easier readability.<br clear="none"><br>Regarding metrics.torproject.org... I expect you'll see that written-bytes and <span>read-bytes </span>only reflect that of a single Snowflake instance. However, your consensus weight will reflect the aggregate of all Snowflake instances.</div><div><br clear="none">> On the matter of onion key rotation, I had the idea of making the onion key files read-only. Roger did some source code investigation and said that it might work to prevent onion key rotation, with some minor side effects. I plan to give the idea a try on a different bridge. The possible side effects are that tor will continue trying and failing to rotate the onion key every hour, and "force a router descriptor rebuild, so it will try to publish a new descriptor each hour."</div><div><br></div><div>I'm interested to hear how the prospective read-only file fix plays out. However, from my observations, I would assume that connects will eventually start bleeding off any instances that fail to update the key. <span>We really need a long-term solution to this issue for this style of deployment.</span></div><div><br></div><div>Keep up the Great Work!</div><div><br></div><div>Respectfully,</div><div><br></div><div><br></div><div>Gary<br><div class="ydp558bb730yqt8425828846" id="ydp558bb730yqtfd78737"><span style="color: rgb(29, 34, 40); font-size: 16px;">—</span><br style="color: rgb(29, 34, 40); font-size: 16px;"><span style="color: rgb(29, 34, 40); font-size: 16px;">This Message Originated by the Sun.</span><br style="color: rgb(29, 34, 40); font-size: 16px;"><span style="color: rgb(29, 34, 40); font-size: 16px;">iBigBlue 63W Solar Array (~12 Hour Charge)</span><br style="color: rgb(29, 34, 40); font-size: 16px;"><span style="color: rgb(29, 34, 40); font-size: 16px;">+ 2 x Charmast 26800mAh Power Banks</span><br style="color: rgb(29, 34, 40); font-size: 16px;"><span style="color: rgb(29, 34, 40); font-size: 16px;">= iPhone XS Max 512GB (~2 Weeks Charged)</span><br clear="none"></div></div>
</div>
</div></div></body></html>