GitLab CI preserve environment between build stages

Question

I am working on a python project and using miniconda to manage my environment. I am using GitLab for CI with the following runner configuration

stages:
  - b         


        

Answer 1

The independence of the jobs is a design feature. You might have noticed that GitLab's interface allows you to re-run a single job which wouldn't be possible if the jobs depended on each other.

I don't know what Miniconda exactly performs but if it builds a virtual environment in specific folders, you can use cache to preserve the content of those folders between the jobs. However, you cannot fully rely on it because the documentation states that...

The cache is provided on a best-effort basis, so don't expect that the cache will be always present. For implementation details, please check GitLab Runner.

Considering that your job absolutely depends on the environment being built, you would need a mechanism to detect whether the (cached) environment exists and re-create it only if needed.

I think you are taking good path trying to separate the environment setup and the jobs because it might save lots of time in case you decide one day to run different tests simultaneously (jobs at the same stage run in parallel).

Answer 2

You can put your common commands under before_script. It will be applied to all the of the child stages. In this way, you do not need to repeat your code.

You can use something like following:

stages:
  - build
  - test
  - deploy 

before_script:
  - if hash $HOME/miniconda/bin/conda 2>/dev/null; 
      then
         export PATH="$HOME/miniconda/bin:$PATH";
      else
        wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh;
        bash miniconda.sh -b -p $HOME/miniconda;
        export PATH="$HOME/miniconda/bin:$PATH";
      fi
    - conda update --yes conda

build:
  stage: build
  script:
    - << Your build script >>

test:
  stage: test
  script:
    - conda env create --quiet --force --file environment.yml
    - source activate myenv
    - nosetests --with-coverage --cover-erase --cover-package=mypackage --cover-html
    - pylint --reports=n tests/test_final.py
    - pep8 tests/test_final.py
    - grep pc_cov cover/index.html | egrep -o "[0-9]+\%" | awk '{ print "covered " $1;}'

deploy:
  stage: deploy
  before_script:
    - << Override to global before_script >>
    - << DO something else >>

Answer 3

You can use artifacts to pass files between build stages.

However if the shared parts are environment stuff (i.e. it's not code you've written) then you probably should use a cache.