Last week I decided to google “self-hosted CI/CD” and serendipitously discovered Laminar CI. I was looking for something lightweight to wrap around a bash build script, and Laminar CI was the perfect match!

Buckle up, this is a long entry!

The projects tagline describes it perfectly:

Continuous Integration the Light Way

My first idea was to automate building and publishing QMK firmware artefacts, so I can put in small fixes for my keyboards without having a complete development setup on my local machine.

Table of Contents

Laminar #

The Laminar documentation describes this much better than me below, but Laminar is very easy to understand, at least if you have some background in old-fashioned unix system administration:

• It’s bash-scripts¹, cronjobs, ENV variables and configuration files – all the way down!
• Laminar is just a small application to manage a queue of jobs, which in turn triggers said bash scripts

Create a file in the right location (file ending .run, with .init and .after for setting up/tearing down if needed), make it executable and you can schedule jobs. It is refreshingly simple.

Web UI #

There’s a helpful web ui for Laminar, which by default binds to *:8080. I’m using Caddy to serve this internally, my Caddyfile looks something like this:

{
    email me@example.com
    acme_ca https://ca.example.com/acme/acme/directory
}

ci.example.com {
    reverse_proxy 127.0.0.1:8080
}

It’s worth noting that this is a read-only ui, you have to trigger jobs elsewhere. No authentication either, so it might be worth hiding this behind some authentication proxy or not exposing this to the internet at all.

QMK build pipeline #

I store my fork of QMK in a local forgejo container instance. In the same subnet I also run a ‘worker’ node which runs webhook which triggers Laminar jobs when the right conditions are met.

That’s a lot of information in one image!

1. A commit to forgejo triggers a request from Forgejo to my worker node, running webhook
2. Said webhook has a few endpoints, which routes to webhook.sh with different parameters. webhooks.sh is basically a switch statement used to queue different laminar jobs (eg, call laminarc queue qmk_firmware DATA=/tmp/foobar.tmp on each commit in webhook input)
3. If triggering qmk_firmware, we first check if we have a laminar workspace, run qmk_firmware.init otherwise (which checks out my repository to a folder at a predictable location)
4. qmk_firmware runs a python script (qmk_firmware.py), which looks at the listed commit and sees if changes are related to a keyboard and returns what keyboards and layouts needs to be rebuilt. This isn’t a bash script as it required nested data structures (sets in a dict) which bash can’t handle
5. qmk_firmware queues the necessary build tasks as qmk_firmware_build jobs
6. qmk_firmware_build builds and publishes each queued combination back to forgejo

This pipeline is slightly complicated as I wanted to handle webhooks containing multiple commits on multiple keyboards and/or multiple layout changes. There’s also some hacky bits in qmk_firmware.py to handle configuration.

webhook #

I’m just using the Debian provided version of webhook.

sudo apt install webhook

The builtin systemd service will try to read /etc/webhook.conf, which we will create next.

/etc/webhook.conf #

This is where we create hooks which Forgejo can call. This is just a JSON or YAML² file, and there are a few examples in the project documentation. My YAML setup looks something like this:

---
- id: 8dd7cb39-adc8-45a6-99d6-0cd6efa135e6
  execute-command: /var/lib/laminar/webhook.sh
  command-working-directory: /var/lib/laminar
  pass-arguments-to-command:
    - source: payload
      name: repository.name
    - source: payload
      name: commits
  trigger-rule:
    match:
      type: value
      # My branch name here!
      value: refs/heads/trunk
      parameter:
        source: payload
        name: ref

The arguments (pass-arguments-to-command) are sent as input parameters for the triggered command (execute-command), so repository.name is the first parameter in the argument vector, commits is the second and so on.

So, if someone posts to the correct UUID (id) and the data is on the correct branch (trunk in the example) we will forward parts of the payload to our dispatch script webhook.sh. It should be noted that the above is without authentication!

You can find an example payload from Gitea here, as of 2024-11-04 it’s identical to Forgejos payload.

webhook.sh #

This script receives the payload from our webhook and dispatches the requests to build scripts. It’s essentially a switch statement.

#!/bin/bash -ex

unknown_handle() {
    echo "Unhandled function call ${*}"
    exit 123
}

# $1 & $2 corresponds to the entries in pass-arguments-to-command in
# webhook.conf, so $2 is a list of commits and $1 is the repository name.

qmk_firmware() {
    # Dump the list of commits to a temporary file, and send the file name to
    # the build function for later processing
    tmp=$(mktemp -p /var/lib/laminar/tmp)
    echo "${2}" > "${tmp}"

    laminarc queue qmk_firmware DATA="${tmp}"
}

foo_bar() {
    laminarc queue foo_bar
}

case "$1" in
"qmk_firmware")
    qmk_firmware "${@}"
    ;;
"foo_bar")
    foo_bar
    ;;
*)
    unknown_handle "${@}"
    ;;
esac

Laminar setup #

I’m just using the Debian provided version of Laminar, plus Python and python-venv and some minor dependencies:

sudo apt install laminar python3-venv jq mawk

By default Laminar uses an abstract socket for RPC connectivity, which is unauthenticated and more or less magic. You can use a normal socket or IP/port as well, but then you have to configure this in laminar.conf and for all clients as well. See the example laminar.conf for details!

Folder structure #

My repository looks something like this:

.
cfg
├── contexts
│   ├── default.conf
│   └── qmk_firmware.conf
└── jobs
    ├── foo_bar.run
    ├── qmk_firmware.after
    ├── qmk_firmware_build.after
    ├── qmk_firmware_build.conf
    ├── qmk_firmware_build.run
    ├── qmk_firmware.conf
    ├── qmk_firmware.init
    └── qmk_firmware.run
run
├── foo_bar
│   └── workspace
├── qmk_firmware
│   └── workspace
│       └── qmk_firmware
└── qmk_firmware_build
    └── workspace
secrets/
├── qmk_firmware_build
└── qmk_firmware_setup
tmp

cfg contains the build jobs and some configuration, run contains my workspaces and secrets contains…my secrets.

qmk_firmware #

qmk_firmware.init #

This is triggered before qmk_firmware.run if there is no workspace setup. I use this to checkout the repository in the workspace, setup a venv for QMK and finally installing & setting up qmk. Documentation on init jobs can be found here.

#!/bin/bash -ex

set +x
SECRETS_FILE=/var/lib/laminar/secrets/qmk_firmware_setup
if [ -f $SECRETS_FILE ]; then
        echo "Loading secrets from $SECRETS_FILE"
        . $SECRETS_FILE
else
        echo "No secrets found, please create $SECRETS_FILE"
        exit 1
fi

echo "Initializing workspace for qmk_firmware"

git clone "${GITEA_URL}/${QMK_OWNER}/qmk_firmware.git"

pushd qmk_firmware

echo "Setting up venv"

python3 -m venv venv && source venv/bin/activate
pip install qmk

echo "Setting up qmk"

qmk setup -y

qmk_firmware.run #

Initially this is where I built my firmware, and it was just a short bash script. Eventually I wanted to build all combinations of keyboards and layouts in the list of commits, which required me to switch from bash to Python³. Now it’s just a bash script that triggers a Python script, which in turn triggers laminar build jobs.

#!/usr/bin/env bash

pushd "$WORKSPACE" || exit 1

python3 /var/lib/laminar/qmk_firmware.py --file "$DATA"

qmk_firmware.py #

And the Python script (/var/lib/laminar/qmk_firmware.py):

#!/usr/bin/env python3

import re
import subprocess
import json
import sys
import argparse

ENV={
    "PATH": "/usr/bin:/bin:/usr/local/bin"
}

known_variants = {
    "4pplet/steezy60": "4pplet/steezy60/rev_b",
    "keebio/iris": "keebio/iris/rev2",
    "atreus": "atreus/astar"
}

def extract_filenames(data):
    changed_files = set()

    for commit in data:
        added = commit.get("added")
        modified = commit.get("modified")

        if added:
            changed_files.update(added)

        if modified:
            changed_files.update(modified)

    return changed_files

def get_details(filename):
    p = re.compile(r"keyboards/(?P<kb>[\w_/]+)keymaps/(?P<km>[\w_/]+)/")
    m = p.match(filename)

    if not m:
        return None

    if m.group("kb") and m.group("km"):
        return (m.group("kb"), m.group("km"))

    return None

def queue_compile_commands(seen, buildname="qmk_firmware_build"):
    for k, v in seen.items():
        _k = k.rstrip("/")
        kb = known_variants.get(_k) or _k

        for l in v:
            subprocess.run(["laminarc", "queue", buildname, f"KB={kb}",
                            f"KM={l}"], env=ENV)

def parse_json(input_file):
    data = None
    with open(input_file, "r") as fobj:
        data = json.loads(fobj.read())

    return extract_filenames(data)

def generate_combinations(changed_files):
    seen = {}
    for f in changed_files:
        n = get_details(f)
        if not n:
            continue

        km = seen.get(n[0]) or set()
        km.add(n[1])

        seen[n[0]] = km

    return seen


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--file", help="parse json indata")
    args = parser.parse_args()

    if args.file:
        changed_files = parse_json(args.file)
        seen_combinations = generate_combinations(changed_files)
        queue_compile_commands(seen=seen_combinations)
        sys.exit(0)

qmk_firmware.after #

We remove the temporary file (containing commit data) in qmk_firmware.after:

#!/usr/bin/env bash

rm $DATA

qmk_firmware.conf #

And qmk_firmware.conf:

TIMEOUT=300
CONTEXTS=qmk_firmware

qmk_firmware_build #

This job actually builds and uploads the firmware, and is split into a few parts as outlined below.

qmk_firmware_build.run #

This is triggered by qmk_firmware.py, with args KB=foo and KM=bar:

#!/usr/bin/env bash

source /var/lib/laminar/secrets/qmk_firmware_build

pushd "${QMK_SRC}" || exit 1

source venv/bin/activate

qmk compile -kb "${KB}" -km "${KM}"

This should result in a firmware file in $QMK_SRC. Next thing is to figure out it’s file name and upload it.

qmk_firmware_build.after #

If the build goes well (exit code 0), I have an after script which uploads the built artefact to Forgejo. We try to guess the expected file name based on known expections, and then my old Gitea upload script to upload the file back to Forgejo:

#!/usr/bin/env bash

if [ "${BASH_VERSINFO:-0}" -le 4 ]; then
    echo "bash v4 or greater is needed"
    exit 1
fi

declare -A filenames
filenames=(
    ["hhkb/ansi"]="hhkb_ansi_32u4_keymap.hex"
    ["4pplet/steezy60"]="4pplet_steezy60_rev_b_keymap.bin"
    ["converter/adb_usb"]="converter_adb_usb_rev1_keymap.hex"
    ["keebio/iris"]="keebio_iris_rev2_keymap.hex"
    ["gmmk/pro/rev1/iso"]="gmmk_pro_rev1_iso_keymap.bin"
    ["atreus"]="atreus_astar_keymap.hex"
)

firmware_name() {
    if [[ -v filenames[$1] ]]; then
        echo "${filenames[$1]//keymap/$2}"
    else
        echo "keyboard_keymap.hex" |
            sed -e "s|keyboard|$1|" -e "s|keymap|$2|" |
            tr / _
    fi
}

gitea-upload() {
    nver=$(curl -s  -u "${REPO_USER}:${REPO_PASS}" \
        "${GITEA_URL}/api/v1/packages/${QMK_OWNER}" |
        jq --arg fname "${1}" -c '.[] | select(.name == $fname) | .version' |
        sort |
        tail -n 1 |
        tr -d '"' |
        awk -F. -v OFS=. '{$NF += 1 ; print}')
    version="${nver:-1.0.0}"

    # Use basic auth
    curl -s -u "${REPO_USER}:${REPO_PASS}" --upload-file "${1}" \
        "${GITEA_URL}/api/packages/${QMK_OWNER}/generic/${1}/${version}/${1}"
}

set +x
declare -a SECRETS_FILES=("qmk_firmware_setup" "qmk_firmware_build")
for f in "${SECRETS_FILES[@]}"; do
    if [ -f "/var/lib/laminar/secrets/${f}" ]; then
        echo "Loading secrets from ${f}"
        . "/var/lib/laminar/secrets/${f}"
    else
        echo "No secrets found, please create ${f}"
        exit 1
    fi
done

pushd "$QMK_SRC" || exit 1

gitea-upload "$(firmware_name ${KB} ${KM})" || exit 2

qmk_firmware_build.conf #

And qmk_firmware_build.conf:

TIMEOUT=300
CONTEXTS=qmk_firmware

secrets #

My /var/lib/laminar/secrets/qmk_firmware_setup looks like this:

GITEA_URL="htpps://gitea.example.com"
QMK_OWNER="keebs"

And /var/lib/laminar/secrets/qmk_firmware_build:

REPO_USER="laminar"
REPO_PASS=""
QMK_SRC="/var/lib/laminar/run/qmk_firmware/workspace/qmk_firmware"

contexts #

We can use contexts to define how many executions of each job is allowed. We both have a default context, and one specific for my QMK pipeline as it can’t run concurrently. More information here.

The default context is normally implicit but needs to be explicit if you define more, below is it’s default settings:

# cfg/contexts/default.conf
EXECUTORS=6
JOBS=*

And for my QMK jobs:

# cfg/contexts/qmk_firmware.conf
EXECUTORS=1
JOBS=qmk_firmware,qmk_firmware_build

If you don’t have a catch-all context, any other Laminar jobs won’t execute unless you explicitly match them to a context. If my firmware files were big I’d probably split the upload task into it’s own job so I could jump on the next build task without waiting for the upload to finish, but now the firmware files are only a few kilobytes in size.

Conclusion #

That was a lot of text! But we’ve also setup a working CI/CD solution for QMK which is easily extentable for other scripts as well. I’m quite happy with the results.