Hello!
I'm writing to inform you that TPA's been moving on something that should make it possible to eventually merge both of the prometheus / grafana servers: we're moving towards per-user passwords that need to be set through LDAP and deprecating the shared passwords.
I would invite everyone who needs to continue accessing prometheus and grafana (either on prometheus1.tpo or prometheus2.tpo) to head over to https://db.torproject.org/ , login and set yourself a password in the "Web password" field. That field was previously hidden but it was recently added to the form.
The password will take some time to get synchronized to the servers, so allow some 1 to 2 hours before you test out your new credentials. When ready, head over to https://grafana2.torproject.org/ and use your ldap username with your new web password to confirm that you're able to login there.
The shared passwords that are currently in use are bound to be removed on April 17th.
If you're having issues with setting up and/or using your own credentials to access prometheus and grafana, please contact TPA and we'll take a look with you into what's happening.
Cheers!
additional note for TPA members: we now have a fallback password that's present in our password manager. it should let us access the monitoring sites even if ldap has a disruption. you can try that one as well.
On 2025-04-01 16:41:55, Gabriel Filion wrote:
[...]
Some precisions here, if I may. :)
The password will take some time to get synchronized to the servers, so allow some 1 to 2 hours before you test out your new credentials. When ready, head over to https://grafana2.torproject.org/ and use your ldap username with your new web password to confirm that you're able to login there.
The delay is normally between 5 to 15 minutes:
https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/ldap#know-when-will...
The shared passwords that are currently in use are bound to be removed on April 17th.
Specifically, here we're talking about the "metrics" Grafana (and Prometheus!) user you folks have been sharing around with each other. I don't know exactly how far it was shared, but you need to, now, let those people know that this password will stop working soon.
Also be aware, as announced in TPA-RFC-33, that this effectively gives access to all users with an LDAP account to the Prometheus server. When we last discussed this, it was okayed by people here, but it's still time to review that policy. It would be more complicated for us because we'd need to grant access on a username basis (like "hiro can login") but it's possible.
[...]
additional note for TPA members: we now have a fallback password that's present in our password manager. it should let us access the monitoring sites even if ldap has a disruption. you can try that one as well.
For TPA folks, that's in services/prometheus.torproject.org. You *should* use your "web password" to operate prometheus on a daily basis however: for now, you can write that in your .netrc file (like we do for the KGB bot password), but I have plans to hook that up to pass(1) instead.
Let me know if you think that should be expedited.
a.
On 2025-04-01 17:15:58, Antoine Beaupré wrote:
On 2025-04-01 16:41:55, Gabriel Filion wrote:
[...]
Some precisions here, if I may. :)
Another thing that came up is that we didn't realize this, but the all-mighty "metrics" user has admin privileges! Your new user in Grafana will *not* have such privileges. We have not thought about that aspect *at all*, so I guess, for now, we'll just transitively grant you admin access on the org?
So far, i've given geko access, if others need access, let us know and we'll promote your user. We can't proactively do so, you need to login with your user first.
A.
Hi!,
On 4/2/25 16:35, Antoine Beaupré via network-health wrote:
On 2025-04-01 17:15:58, Antoine Beaupré wrote:
On 2025-04-01 16:41:55, Gabriel Filion wrote:
[...]
Some precisions here, if I may. :)
Another thing that came up is that we didn't realize this, but the all-mighty "metrics" user has admin privileges! Your new user in Grafana will *not* have such privileges. We have not thought about that aspect *at all*, so I guess, for now, we'll just transitively grant you admin access on the org?
So far, i've given geko access, if others need access, let us know and we'll promote your user. We can't proactively do so, you need to login with your user first.
I just successfully logged in with my ldap password into grafana2.tpo \o/, but... don't see the the "New" button to create a dashboard (i can still see it logging in as metrics user). Would it be possible to have that right without being admin, at least for the network-health dashboards directory?
Thanks for your work on this!, juga
Hi again,
On 4/7/25 15:56, juga via network-health wrote:
Hi!,
On 4/2/25 16:35, Antoine Beaupré via network-health wrote:
On 2025-04-01 17:15:58, Antoine Beaupré wrote:
On 2025-04-01 16:41:55, Gabriel Filion wrote:
[...]
Some precisions here, if I may. :)
Another thing that came up is that we didn't realize this, but the all-mighty "metrics" user has admin privileges! Your new user in Grafana will *not* have such privileges. We have not thought about that aspect *at all*, so I guess, for now, we'll just transitively grant you admin access on the org?
So far, i've given geko access, if others need access, let us know and we'll promote your user. We can't proactively do so, you need to login with your user first.
I just successfully logged in with my ldap password into grafana2.tpo \o/, but... don't see the the "New" button to create a dashboard (i can still see it logging in as metrics user). Would it be possible to have that right without being admin, at least for the network-health dashboards directory?
err... since i can still login with the metrics user, i changed my role from viewer to editor and that's enough to see the new button :) Not sure that's fine as a permanent setup (dunno anything about grafana permissions).
Thanks!, juga
Hello again!
This is just a reminder that the shared accounts for grafana (`tor-guest` for grafana1 and `metrics` for grafana2) are bound to be removed two days from now, on thursday april 17th
If you have not yet setup your "web password" and logged in to grafana1.tpo (mostly TPA for now) and/or grafana2.tpo (all teams in this email), I would encourage you to do so soon. Of course it's possible to do this after the cutoff date, it just means that in the mean time your access through the shared account will stop working this thursday.
Once your first login on grafana has happened, give me a sign and I will add your user to the appropriate team in grafana to fix your permissions. Your user needs to exist in grafana before the permissions can be set. you can either add a comment to https://gitlab.torproject.org/tpo/tpa/team/-/issues/41636 , send me an email, or contact me on IRC.
Cheers!
On 2025-04-01 16:41, Gabriel Filion wrote:
Hello!
I'm writing to inform you that TPA's been moving on something that should make it possible to eventually merge both of the prometheus / grafana servers: we're moving towards per-user passwords that need to be set through LDAP and deprecating the shared passwords.
I would invite everyone who needs to continue accessing prometheus and grafana (either on prometheus1.tpo or prometheus2.tpo) to head over to https://db.torproject.org/ , login and set yourself a password in the "Web password" field. That field was previously hidden but it was recently added to the form.
The password will take some time to get synchronized to the servers, so allow some 1 to 2 hours before you test out your new credentials. When ready, head over to https://grafana2.torproject.org/ and use your ldap username with your new web password to confirm that you're able to login there.
The shared passwords that are currently in use are bound to be removed on April 17th.
If you're having issues with setting up and/or using your own credentials to access prometheus and grafana, please contact TPA and we'll take a look with you into what's happening.
Cheers!
additional note for TPA members: we now have a fallback password that's present in our password manager. it should let us access the monitoring sites even if ldap has a disruption. you can try that one as well.
Hello!
This is my final email about this, then I can stop bothering everyone about this.
Since the cutoff date was planned for today, I've just removed the shared passwords for grafana, prometheus and karma.
All access should now be made using your "web password" from your LDAP account.
Now that a real change happened, the trouble might start happening. Do give TPA a sign if you have lost access to grafana/prometheus/karma and you need help to gain that access back.
TPA: note that grafana's admin password used to be different but it now defaults back to the same one used by prometheus. I'll remove the obsolete one from the password manager to remove confusion.
On 2025-04-15 11:36, Gabriel Filion wrote:
Hello again!
This is just a reminder that the shared accounts for grafana (`tor- guest` for grafana1 and `metrics` for grafana2) are bound to be removed two days from now, on thursday april 17th
If you have not yet setup your "web password" and logged in to grafana1.tpo (mostly TPA for now) and/or grafana2.tpo (all teams in this email), I would encourage you to do so soon. Of course it's possible to do this after the cutoff date, it just means that in the mean time your access through the shared account will stop working this thursday.
Once your first login on grafana has happened, give me a sign and I will add your user to the appropriate team in grafana to fix your permissions. Your user needs to exist in grafana before the permissions can be set. you can either add a comment to https://gitlab.torproject.org/tpo/tpa/ team/-/issues/41636 , send me an email, or contact me on IRC.
Cheers!
On 2025-04-01 16:41, Gabriel Filion wrote:
Hello!
I'm writing to inform you that TPA's been moving on something that should make it possible to eventually merge both of the prometheus / grafana servers: we're moving towards per-user passwords that need to be set through LDAP and deprecating the shared passwords.
I would invite everyone who needs to continue accessing prometheus and grafana (either on prometheus1.tpo or prometheus2.tpo) to head over to https://db.torproject.org/ , login and set yourself a password in the "Web password" field. That field was previously hidden but it was recently added to the form.
The password will take some time to get synchronized to the servers, so allow some 1 to 2 hours before you test out your new credentials. When ready, head over to https://grafana2.torproject.org/ and use your ldap username with your new web password to confirm that you're able to login there.
The shared passwords that are currently in use are bound to be removed on April 17th.
If you're having issues with setting up and/or using your own credentials to access prometheus and grafana, please contact TPA and we'll take a look with you into what's happening.
Cheers!
additional note for TPA members: we now have a fallback password that's present in our password manager. it should let us access the monitoring sites even if ldap has a disruption. you can try that one as well.
network-health@lists.torproject.org
-
Antoine Beaupré -
Gabriel Filion -
juga