disruption in email services
Hi everyone, We've had reports of increasing delivery failure rates at other providers, particularly gmail, from multiple parties in the course of the last few weeks / months. It might be worth falling back to an alternative provider if you have access to one, for your @torproject.org email, otherwise you might miss some email. Inversely, if you have a hard requirement to reach @gmail.com people right now, you may want to use another provider or use a gmail account directly. We're working on emergency mitigations for this problem. I should come up with a proposal tomorrow and work should start next week. In the meantime, status.torproject.org has been updated. Feel free to circulate outside of this space, since many people here might not be able to read this message in the first place, of course. a. -- Antoine Beaupré torproject.org system administration
On 30 Nov (15:12:52), Antoine Beaupré wrote:
Hi everyone,
We've had reports of increasing delivery failure rates at other providers, particularly gmail, from multiple parties in the course of the last few weeks / months.
It might be worth falling back to an alternative provider if you have access to one, for your @torproject.org email, otherwise you might miss some email. Inversely, if you have a hard requirement to reach @gmail.com people right now, you may want to use another provider or use a gmail account directly.
We're working on emergency mitigations for this problem. I should come up with a proposal tomorrow and work should start next week.
In the meantime, status.torproject.org has been updated. Feel free to circulate outside of this space, since many people here might not be able to read this message in the first place, of course.
SPF record ;) ;) :D Cheers! David -- l2kj7J1p2QON/ZOIT8ZzZhbo5MU1g3dQ6yKqBtMmxhM=
Status update, day three: We now have "soft" SPF and DMARC records on all mail servers and DKIM signatures on the three major mail servers. This will probably impact users currently sending mail from gmail and riseup, as their reputation will suffer from not being in the allow lists. If this is a problem, an `include:riseup.net include:google.com` mechanism could be added to the top-level SPF policy. Next step is to finish the DKIM deployment (#40989) and then make the DMARC and SPF records "hard", which will happen once we are more confident this will cause more good than harm. Work on the mail exchanger may also start soon, alongside other mitigations for problems we may encounter from here on. Do let us know here if you encounter problems or improvements. As usual, you can follow our work in GitLab in: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40981 Comment there or file new issues in: https://gitlab.torproject.org/tpo/tpa/team/-/issues/new If all fails, contact us at: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/support Onwards! -- Antoine Beaupré torproject.org system administration
Status update, week two: TL;DR: situation mostly fixed. Next work in 2023. Things seem to have gone back to normal. In fact, I've been a little hesitant in sending this email, because I'm worried that people will suddenly remember all those issues they've been having with email forever and remind me how everything *else* is broken, but dang, it looks like we kind of made it? As far as I can tell, emails send through the submission server have stopped bouncing (#40640 closed!), authentication problems for messages actually delivered at gmail have stopped (#40765 closed!) and other outright gmail bounces seem to have also stopped (#40959 closed!). That is a *major* improvement, in such a short time span. There are still issues with our mail services, of course... In particular, email forwards are still in somewhat of a gray zone. They are *technically* not allowed, especially for senders with a hard SPF policy, but that was already a problem before. We *may* see delivery problems for mails sent *internally*, but, again, probably not worse off than what we were before. And, for the record, we are well within my time estimates for the labor on this. We still have a *lot* of work to do to complete (even a plan for) the mail infrastructure, but I'm going to call this one as fixed for now, as the next steps require further thinking, and even more major architectural changes. So I'm going to reserve those for that non-proverbial next year. In the meantime, I'll work on another one of my famous proposal. Apparently, ChatGPT can generate those for me now, so maybe I'll give that a shot.. ;) "Make a proposal for torproject.org mail services in the style of anarcat's TPA-RFC" please? A. -- Antoine Beaupré torproject.org system administration
A huge thank you for jumping up and making changes when we noticed a downturn in delivery on our fundraising emails during a critical time. Many rounds of applause and beverages on me to you and the TPA team the next time we meet in person--thank you!! Al On 12/15/22 12:32 PM, Antoine Beaupré wrote:
Status update, week two: TL;DR: situation mostly fixed. Next work in 2023.
Things seem to have gone back to normal. In fact, I've been a little hesitant in sending this email, because I'm worried that people will suddenly remember all those issues they've been having with email forever and remind me how everything *else* is broken, but dang, it looks like we kind of made it?
As far as I can tell, emails send through the submission server have stopped bouncing (#40640 closed!), authentication problems for messages actually delivered at gmail have stopped (#40765 closed!) and other outright gmail bounces seem to have also stopped (#40959 closed!).
That is a *major* improvement, in such a short time span. There are still issues with our mail services, of course... In particular, email forwards are still in somewhat of a gray zone. They are *technically* not allowed, especially for senders with a hard SPF policy, but that was already a problem before. We *may* see delivery problems for mails sent *internally*, but, again, probably not worse off than what we were before.
And, for the record, we are well within my time estimates for the labor on this. We still have a *lot* of work to do to complete (even a plan for) the mail infrastructure, but I'm going to call this one as fixed for now, as the next steps require further thinking, and even more major architectural changes. So I'm going to reserve those for that non-proverbial next year.
In the meantime, I'll work on another one of my famous proposal. Apparently, ChatGPT can generate those for me now, so maybe I'll give that a shot.. ;)
"Make a proposal for torproject.org mail services in the style of anarcat's TPA-RFC" please?
A.
_______________________________________________ tor-project mailing list tor-project@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-project
Thank you, Thank you and Thank you TPA team :) On Thu, Dec 15, 2022 at 5:35 PM Al Smith <smith@torproject.org> wrote:
A huge thank you for jumping up and making changes when we noticed a downturn in delivery on our fundraising emails during a critical time. Many rounds of applause and beverages on me to you and the TPA team the next time we meet in person--thank you!!
Al
Status update, week two: TL;DR: situation mostly fixed. Next work in
On 12/15/22 12:32 PM, Antoine Beaupré wrote: 2023.
Things seem to have gone back to normal. In fact, I've been a little hesitant in sending this email, because I'm worried that people will suddenly remember all those issues they've been having with email forever and remind me how everything *else* is broken, but dang, it looks like we kind of made it?
As far as I can tell, emails send through the submission server have stopped bouncing (#40640 closed!), authentication problems for messages actually delivered at gmail have stopped (#40765 closed!) and other outright gmail bounces seem to have also stopped (#40959 closed!).
That is a *major* improvement, in such a short time span. There are still issues with our mail services, of course... In particular, email forwards are still in somewhat of a gray zone. They are *technically* not allowed, especially for senders with a hard SPF policy, but that was already a problem before. We *may* see delivery problems for mails sent *internally*, but, again, probably not worse off than what we were before.
And, for the record, we are well within my time estimates for the labor on this. We still have a *lot* of work to do to complete (even a plan for) the mail infrastructure, but I'm going to call this one as fixed for now, as the next steps require further thinking, and even more major architectural changes. So I'm going to reserve those for that non-proverbial next year.
In the meantime, I'll work on another one of my famous proposal. Apparently, ChatGPT can generate those for me now, so maybe I'll give that a shot.. ;)
"Make a proposal for torproject.org mail services in the style of anarcat's TPA-RFC" please?
A.
_______________________________________________ tor-project mailing list tor-project@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-project
_______________________________________________ tor-project mailing list tor-project@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-project
participants (4)
-
Al Smith -
Antoine Beaupré -
David Goulet -
isabela fernandes