Making a gitlab-ci for the pelican site

So I've had this pelican site for .. ehm.. $time ..

First .gitlab-ci.yml was easy, just use the basic stuff from examples, but then I needed some extra stuff.

Had to make a requirements.txt to add a few basic python packages (which pulls in an extra dusin).

requirements.txt:

pelican
markdown
typogrify
beautifulsoup4
blockdiag
nwdiag

So the CI needed some /changes/ to make things works.

First attempt was

image: python:2.7-alpine
pages:
  script:
  - pip install -r requirements.txt
  - pelican -s publishconf.py
  artifacts:
    paths:
    - public/

It gave me the basic functionality, but lacked some stuff to make everything work, so desperate to see a successfull gitlab-ci job I went for one with everything.. and then some.

image: "ubuntu:latest"

pages:
  script:
  - apt update >/dev/null
  - apt install -y apt-utils >/dev/null
  - apt install -y git >/dev/null
  - apt install -y python-pip >/dev/null
  - apt install -y graphviz >/dev/null
  - apt install -y python-blockdiag >/dev/null
  - pip install -r requirements.txt >/dev/null
  - pelican -s publishconf.py
  artifacts:
    paths:
    - public/

Unfortunately, this ment that buildtimes went from a nice 51 seconds, to a disheartening 4 minutes, so something had to happen.

I did a few experients trying to make it quicker, among other things I tweaked the server actually doing the CI bit, which cut it down to just below 4 minutes, no big gain there.

image: "python:2.7"

pages:
  before_script:
  - apt update >/dev/null 2>/dev/null
  - apt install -y graphviz >/dev/null 2>/dev/null
  - pip install -U pip -r requirements.txt >/dev/null 2>/dev/null
  script:
  - pelican -s publishconf.py
  only:
  - master
  artifacts:
    paths:
    - public/

That seems to be the best combination of flexibility and speed so far, it lets me easily add helper programs with apt and runs in less than a minute. I'm not sure if that's as good as it'll get but time will tell.

.. /some time passed/

And then, almost by accident, well, I actually tried setting up a Jekyll site and for that I found a .gitlab-ci.yml file that did caching of gem bundles. So in an attempt to cache python stuff, the current version of the .gitlab-ci.yml is now

image: "python:2.7"

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/pip-cache"

cache:
  paths:
    - "$CI_PROJECT_DIR/pip-cache"
  key: "$CI_PROJECT_ID"

pages:
  before_script:
  - apt update >/dev/null 2>/dev/null
  - apt install -y graphviz >/dev/null 2>/dev/null
  - pip install -U pip setuptools >/dev/null 2>/dev/null
  - pip install -r requirements.txt >/dev/null 2>/dev/null
  script:
  - pelican -s siteconf.py
  only:
  - master
  artifacts:
    paths:
    - public/

Which is very promising. It cut down the processing time from 58 to 47 seconds, that's close to a 20% improvement with a few generic lines of configuration. Kind of impressive.

The 'pelican' site generation itself takes around 3.4 seconds, so 92% of the time is spent setting up the environment, that is still a bit depressing.