[tor-bugs] #23756 [Core Tor/Tor]: tor's .gitlab-ci.yml is doing mirroring? why?

Wed Oct 4 01:02:05 UTC 2017

#23756: tor's .gitlab-ci.yml is doing mirroring? why?
--------------------------+------------------------------------
 Reporter:  isis          |          Owner:  (none)
     Type:  defect        |         Status:  new
 Priority:  Medium        |      Milestone:  Tor: 0.3.1.x-final
Component:  Core Tor/Tor  |        Version:  Tor: 0.3.1.3-alpha
 Severity:  Normal        |     Resolution:
 Keywords:  tor-ci        |  Actual Points:
Parent ID:                |         Points:
 Reviewer:                |        Sponsor:
--------------------------+------------------------------------
Description changed by isis:

Old description:

> Currently in master we have the following stanza in our .gitlab-ci.yml
> (from #22891):
>
> {{{
> update:
>   script:
>     - "apt-get install -y --fix-missing git openssh-client"
>
>     # Run ssh-agent (inside the build environment)
>     - eval $(ssh-agent -s)
>
>     # Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent
> store
>     - ssh-add <("$DEPLOY_KEY")
>
>     # For Docker builds disable host key checking. Be aware that by
> adding that
>     # you are suspectible to man-in-the-middle attacks.
>     # WARNING: Use this only with the Docker executor, if you use it with
> shell
>     # you will overwrite your user's SSH config.
>     - mkdir -p ~/.ssh
>     - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking
> no\n\n" > ~/.ssh/config'
>     # In order to properly check the server's host key, assuming you
> created the
>     # SSH_SERVER_HOSTKEYS variable previously, uncomment the following
> two lines
>     # instead.
>     - mkdir -p ~/.ssh
>     - '[[ -f /.dockerenv ]] && echo "$SSH_SERVER_HOSTKEYS" >
> ~/.ssh/known_hosts'
>     - echo "merging from torgit"
>     - git config --global user.email "labadmin at oniongit.eu"
>     - git config --global user.name "gitadmin"
>     - "mkdir tor"
>     - "cd tor"
>     - git clone --bare https://git.torproject.org/tor.git
>     - git push --mirror git at oniongit.eu:network/tor.git
> }}}
>
> Why are we doing this? Can we put a cronjob on the oniongit.eu server
> instead? It's pretty weird and frankly unexpected that my personal fork
> of tor at https://gitlab.com/isis/tor is cloning the official tor repo
> and then trying to mirror it to oniongit.eu. It also has a bunch of other
> problems:
>
> * The `ssh-add` line [https://gitlab.com/isis/tor/-/jobs/34990901 is
> broken, causing CI to fail because it sits there forever waiting for a
> passphrase].
>
>   I was originally going to patch the `ssh-add` line to instead be `[[ -n
> "${DEPLOY_KEY}" -a -r "$DEPLOY_KEY" ]] && ssh-add "$DEPLOY_KEY" <<<""`
> but if I fix that, then all the rest of this script would run, so I'm
> rather glad it's failing on a more innocuous command.
>
> * Even if the `ssh-add` line weren't broken, this whole thing fails
> unless it's being run from a fork on oniongit.eu.
> * Why is it disabling SSH hostkey checking?!
> * Why is it making the `~/.ssh` directory twice?
> * Why is it assuming that environment variables are set? e.g. `$FOO`
> versus `${FOO}` or better `test -n ${FOO}`
> * Why is it unconditionally setting (global!) git config options? (I
> assume to disable the warning that git spits out when you don't have
> `$GIT_{AUTHOR,COMMITTER}_{NAME,EMAIL}` set, but why would a CI config set
> them globally instead of just setting the correct environment variables?)
> * Why are the mirror URLs hardcoded?
> * Why is the git username and email hardcoded?
> * Why is any of this even running when I push to
> https://gitlab.com/isis/tor?
> * Why is it unconditionally starting an ssh-agent?
>

> I'm sorry if this is all necessary and I'm just not understanding the
> setup, but it's all just extremely unexpected behaviour from what is
> supposed to be a CI config file. Further, it's not even doing the same
> testing as our .travis.yml, but I'll make another ticket for that issue.

New description:

 Currently in master we have the following stanza in our .gitlab-ci.yml
 (from #22891):

 {{{
 update:
   script:
     - "apt-get install -y --fix-missing git openssh-client"

     # Run ssh-agent (inside the build environment)
     - eval $(ssh-agent -s)

     # Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent
 store
     - ssh-add <("$DEPLOY_KEY")

     # For Docker builds disable host key checking. Be aware that by adding
 that
     # you are suspectible to man-in-the-middle attacks.
     # WARNING: Use this only with the Docker executor, if you use it with
 shell
     # you will overwrite your user's SSH config.
     - mkdir -p ~/.ssh
     - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking
 no\n\n" > ~/.ssh/config'
     # In order to properly check the server's host key, assuming you
 created the
     # SSH_SERVER_HOSTKEYS variable previously, uncomment the following two
 lines
     # instead.
     - mkdir -p ~/.ssh
     - '[[ -f /.dockerenv ]] && echo "$SSH_SERVER_HOSTKEYS" >
 ~/.ssh/known_hosts'
     - echo "merging from torgit"
     - git config --global user.email "labadmin at oniongit.eu"
     - git config --global user.name "gitadmin"
     - "mkdir tor"
     - "cd tor"
     - git clone --bare https://git.torproject.org/tor.git
     - git push --mirror git at oniongit.eu:network/tor.git
 }}}

 Why are we doing this? Can we put a cronjob on the oniongit.eu server
 instead? It's pretty weird and frankly unexpected that my personal fork of
 tor at https://gitlab.com/isis/tor is cloning the official tor repo and
 then trying to mirror it to oniongit.eu. It also has a bunch of other
 problems:

 * The `ssh-add` line [https://gitlab.com/isis/tor/-/jobs/34990901 is
 broken, causing CI to fail because it sits there forever waiting for a
 passphrase].

   I was originally going to patch the `ssh-add` line to instead be `[[ -n
 "${DEPLOY_KEY}" -a -r "$DEPLOY_KEY" ]] && ssh-add "$DEPLOY_KEY" <<<""` but
 if I fix that, then all the rest of this script would run, so I'm rather
 glad it's failing on a more innocuous command.

 * Even if the `ssh-add` line weren't broken, this whole thing fails unless
 it's being run from a fork on oniongit.eu.
 * Why is it disabling SSH hostkey checking?!
 * Why is it making the `~/.ssh` directory twice?
 * Why is it assuming that environment variables are set? e.g. `$FOO`
 versus `${FOO}` or better `test -n ${FOO}`
 * Why is it unconditionally setting (global!) git config options? (I
 assume to disable the warning that git spits out when you don't have
 `$GIT_{AUTHOR,COMMITTER}_{NAME,EMAIL}` set, but why would a CI config set
 them globally instead of just setting the correct environment variables?)
 * Why are the mirror URLs hardcoded?
 * Why is the git username and email hardcoded?
 * Why is any of this even running when I push to
 https://gitlab.com/isis/tor?
 * Why is any of this even running when I push anywhere?
 * Why is it unconditionally starting an ssh-agent?
 * Why is using the existence of a
 ([https://superuser.com/questions/1021834/what-are-dockerenv-and-
 dockerinit#1021925 deprecated!]) `/.dockerenv` file to determine if we're
 in a docker container?
 * Why is it assuming we're in the ''correct'' docker container, when lots
 of things, especially lots of CI systems, use docker?

 I'm sorry if this is all necessary and I'm just not understanding the
 setup, but it's all just extremely unexpected behaviour from what is
 supposed to be a CI config file. Further, it's not even doing the same
 testing as our .travis.yml, but I'll make another ticket for that issue.

--

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/23756#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online