On 2022-06-18 17:56:14, Jim Newsome wrote:
On 6/16/22 08:55, Antoine Beaupré wrote:
<snip> > In general, you shouldn't really *trust* GitLab or GitLab CI for > anything else than running tests. Builds should be verified out of band > with reproducible builds. You can reproduce a local GitLab CI > environment by installing gitlab-runner and executing jobs locally, > without having to trust the entire GitLab installation or foreign > runners. As a reminder, it is your responsibility to ensure the > integrity of your code and artifacts, see those links for a further > discussion: > > https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/81 > https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab#git-repository-integrity-solutions > https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/git#security-concerns <snip>
We also had some discussion about reproducing gitlab-CI builds in https://gitlab.torproject.org/tpo/core/tor/-/issues/40615.
Interesting!
While it's fairly straightforward to install a gitlab-runner and execute locally, as far as I can tell a malicious GitLab installation could still send a modified "script" (post-processed .gitlab-ci.yml) or repo checkout down to the runner. Maybe there's some way to audit this, but I couldn't find an obvious one. Maybe configuring the runner to log at debug level would record enough? https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the...
Thtat's not what I mean. I don't mean installing your own runner locally and hooking it up with GitLab. I mean installing the gitlab-runner package (only!) and *not* hooking it up in GitLab.
Instead, you run the job completely locally, without involving GitLab at all. That's done with the `gitlab-runner exec` command:
https://docs.gitlab.com/runner/commands/#gitlab-runner-exec
We have docs about this here:
https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/ci#running-a-job-...
This removes a large part of the attack surface because GitLab is taken out of the equation. It reduces the stack to:
* your local computer and operating system * your git repository * git * gitlab-runner * the executor (e.g. Docker) and its image
It's still pretty darn large, but it's better than before. :)
For that issue I ended up hacking together a small python script that processes the .gitlab-ci.yml into something to feed directly through Docker. It's currently a bit hacky and specialized for the Debian tor package build. I think it could be generalized further to be reusable if that's of interest (maybe using Docker Compose to orchestrate jobs within a pipeline), but am still thinking about whether there's a better way... https://gitlab.torproject.org/jnewsome/reproduce-tor-debian-build/-/blob/mai...
Note that @eighthave has done a similar thing for F-Droid, you might want to collaborate.
I think the improvement of that over the above is that you remove the "gitlab-runner" part of the attack surface. It's a pretty large attack surface because the runners are a surprisingly large amount of code, but I wonder if it's worth the trouble...
What's the threat model here specifically? Backdoored gitlab-runner code?
Right now my top candidate we haven't tried yet is to install a full local GitLab in addition to a local gitlab-runner; maybe using their published Docker images https://docs.gitlab.com/ee/install/docker.html. This seems like the least engineering effort (~none) but a bit more work for every individual wanting to do such a local build.
Other organisations run *two* GitLab instances for that purpose, by the way. GitLab.com included, from what I understand.
Keeping as much logic out of the .gitlab-ci.yml as possible so that the gitlab yml is trivial to manually reproduce outside of gitlab (e.g. run `./build.sh`) is probably ideal, though gives up some gitlab functionality.
What functionality are you thinking of here?
IIUC this is the approach we're using for the tor tarballs. https://gitlab.torproject.org/tpo/core/tor-ci-reproducible
Thanks for the input! :)
a.