[tor-project] GitLab Runner updates

[Antoine Beaupré]

On 2022-06-18 17:56:14, Jim Newsome wrote:
> On 6/16/22 08:55, Antoine Beaupré wrote:
> <snip>
>> In general, you shouldn't really *trust* GitLab or GitLab CI for
>> anything else than running tests. Builds should be verified out of band
>> with reproducible builds. You can reproduce a local GitLab CI
>> environment by installing gitlab-runner and executing jobs locally,
>> without having to trust the entire GitLab installation or foreign
>> runners. As a reminder, it is your responsibility to ensure the
>> integrity of your code and artifacts, see those links for a further
>> discussion:
>> 
>> https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/81
>> https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab#git-repository-integrity-solutions
>> https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/git#security-concerns
> <snip>
>
> We also had some discussion about reproducing gitlab-CI builds in 
> https://gitlab.torproject.org/tpo/core/tor/-/issues/40615.

Interesting!

> While it's fairly straightforward to install a gitlab-runner and execute 
> locally, as far as I can tell a malicious GitLab installation could 
> still send a modified "script" (post-processed .gitlab-ci.yml) or repo 
> checkout down to the runner. Maybe there's some way to audit this, but I 
> couldn't find an obvious one. Maybe configuring the runner to log at 
> debug level would record enough? 
> https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section

Thtat's not what I mean. I don't mean installing your own runner locally
and hooking it up with GitLab. I mean installing the gitlab-runner
package (only!) and *not* hooking it up in GitLab.

Instead, you run the job completely locally, without involving GitLab at
all. That's done with the `gitlab-runner exec` command:

https://docs.gitlab.com/runner/commands/#gitlab-runner-exec

We have docs about this here:

https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/ci#running-a-job-locally

This removes a large part of the attack surface because GitLab is taken
out of the equation. It reduces the stack to:

 * your local computer and operating system
 * your git repository
 * git
 * gitlab-runner
 * the executor (e.g. Docker) and its image

It's still pretty darn large, but it's better than before. :)

> For that issue I ended up hacking together a small python script that 
> processes the .gitlab-ci.yml into something to feed directly through 
> Docker. It's currently a bit hacky and specialized for the Debian tor 
> package build. I think it could be generalized further to be reusable if 
> that's of interest (maybe using Docker Compose to orchestrate jobs 
> within a pipeline), but am still thinking about whether there's a better 
> way... 
> https://gitlab.torproject.org/jnewsome/reproduce-tor-debian-build/-/blob/main/reproduce_pipeline.py

Note that @eighthave has done a similar thing for F-Droid, you might
want to collaborate.

I think the improvement of that over the above is that you remove the
"gitlab-runner" part of the attack surface. It's a pretty large attack
surface because the runners are a surprisingly large amount of code, but
I wonder if it's worth the trouble...

What's the threat model here specifically? Backdoored gitlab-runner code?

> Right now my top candidate we haven't tried yet is to install a full 
> local GitLab in addition to a local gitlab-runner; maybe using their 
> published Docker images https://docs.gitlab.com/ee/install/docker.html. 
> This seems like the least engineering effort (~none) but a bit more work 
> for every individual wanting to do such a local build.

Other organisations run *two* GitLab instances for that purpose, by the
way. GitLab.com included, from what I understand.

> Keeping as much logic out of the .gitlab-ci.yml as possible so that the 
> gitlab yml is trivial to manually reproduce outside of gitlab (e.g. run 
> `./build.sh`) is probably ideal, though gives up some gitlab 
> functionality.

What functionality are you thinking of here?

> IIUC this is the approach we're using for the tor
> tarballs. https://gitlab.torproject.org/tpo/core/tor-ci-reproducible

Thanks for the input! :)

a.

-- 
Antoine Beaupré
torproject.org system administration