CI/CD Pipeline

Complete reference for the Wilo CI/CD pipeline: GitLab configuration, pipeline stages, release process, and step-by-step deployment guides.

Overview

The pipeline runs on a self-hosted GitLab at git.surcontrol.com with a local runner. It has two separate flows:

Trigger	Jobs that run
Merge Request opened/updated	All quality:* gates (no push, no deploy)
Push to main	release only (quality already passed in the MR)
After release — manual button	deploy:staging

This prevents duplicate pipelines: when a branch has an open MR, direct push pipelines to that branch are suppressed.

Pipeline Stages

MR pipeline:
  quality:api → quality:app → quality:mobile → quality:email → quality:pdf → quality:docs → quality:docker

main pipeline:
  release → [manual] deploy:staging

Quality gates (MR only)

Job	Image	What it checks
quality:api	dotnet/sdk:10.0	Format, build (Release), all unit + arch tests, pending migration check
quality:app	oven/bun:1.3.12	oxlint, Vitest, Vite build
quality:mobile	oven/bun:1.3.12	oxlint, Vitest, Vite build
quality:email	oven/bun:1.3.12	TypeScript typecheck, oxlint, tests
quality:docs	oven/bun:1.3.12	VitePress build
quality:docker	docker:27 + dind	Build all 5 Dockerfiles (no push)

TIP

quality:docker is the slowest job (~4–6 min, dominated by the .NET SDK layer). It runs last among the quality stages to avoid blocking faster checks.

Known limitation

oxfmt --check is disabled in quality:app and quality:mobile due to a DataCloneError in Bun running inside Docker CI (tracked in Bun #25610, oxc #17801). Formatting is still enforced locally via the pre-commit lefthook.

Release stage (main only)

Runs semantic-release (configured in .releaserc.json at the repo root) to:

1. Analyze commits since the last tag using Conventional Commits
2. Determine the next version (patch / minor / major)
3. Update CHANGELOG.md
4. Bump version in package.json, Directory.Build.props, and all site package.json files via scripts/update-versions.sh
5. Create a GitLab Release with generated notes
6. Tag the commit (vX.Y.Z)
7. Run scripts/build-push-images.sh — builds and pushes 5 Docker images to the private registry

Images pushed on each release:

Image	Tag
docker.surcontrol.com/wilo/wilo-api	X.Y.Z + latest
docker.surcontrol.com/wilo/wilo-frontend	X.Y.Z + latest
docker.surcontrol.com/wilo/wilo-mobile	X.Y.Z + latest
docker.surcontrol.com/wilo/wilo-email	X.Y.Z + latest
docker.surcontrol.com/wilo/wilo-pdf	X.Y.Z + latest

Deploy stage (main — manual trigger)

After a release succeeds, a ▶ button appears in the GitLab pipeline UI for deploy:staging. Clicking it:

1. SSH into the staging server
2. Updates WILO_VERSION=X.Y.Z in the server's .env
3. Pulls the new images from the private registry
4. Restarts only the services whose image changed (docker compose up -d)

---

Step 1 — Configure GitLab CI Variables

Go to GitLab → your project → Settings → CI/CD → Variables.

Add the following variables. Mark them as Protected (only available on protected branches like main) and Masked where indicated.

Registry credentials

These are used by scripts/build-push-images.sh during the release stage to push images to docker.surcontrol.com.

Variable	Type	Masked	Protected	Value
DOCKER_REGISTRY_USER	Variable	No	Yes	Username for docker.surcontrol.com
DOCKER_REGISTRY_PASSWORD	Variable	Yes	Yes	Password for the registry user

SSH deploy credentials

Used by deploy:staging to SSH into the staging server.

Variable	Type	Masked	Protected	Value
DEPLOY_SSH_PRIVATE_KEY	Variable	Yes	Yes	Private SSH key (see below)
STAGING_SERVER_HOST	Variable	No	Yes	IP or hostname of the staging server

How to generate the SSH key pair:

# Generate a dedicated deploy key (no passphrase — CI must connect non-interactively)
ssh-keygen -t ed25519 -C "gitlab-ci-deploy" -f ~/.ssh/wilo_deploy -N ""

# Copy the public key to the staging server
ssh-copy-id -i ~/.ssh/wilo_deploy.pub deploy@YOUR_STAGING_HOST

# Copy the private key value to paste into GitLab
cat ~/.ssh/wilo_deploy

Paste the full content of ~/.ssh/wilo_deploy (including -----BEGIN... and -----END... lines) into the DEPLOY_SSH_PRIVATE_KEY variable.

WARNING

Never commit SSH private keys to the repository. They live only in GitLab CI Variables.

---

Step 2 — Prepare the Staging Server

SSH into your staging server as a user with sudo rights and run these steps once.

2.1 Install Docker

# Debian/Ubuntu
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
# Log out and back in for the group to take effect

2.2 Create the deploy user

The CI pipeline SSHes as deploy. This user only needs Docker access, not sudo.

sudo useradd -m -s /bin/bash deploy
sudo usermod -aG docker deploy

# Create the SSH authorized_keys for the deploy user
sudo mkdir -p /home/deploy/.ssh
sudo chmod 700 /home/deploy/.ssh

# Paste the PUBLIC key (wilo_deploy.pub content) here:
echo "ssh-ed25519 AAAA... gitlab-ci-deploy" | sudo tee /home/deploy/.ssh/authorized_keys
sudo chmod 600 /home/deploy/.ssh/authorized_keys
sudo chown -R deploy:deploy /home/deploy/.ssh

2.3 Create the deployment directory

sudo mkdir -p /opt/wilo
sudo chown deploy:deploy /opt/wilo

2.4 Log in to the private Docker registry

As the deploy user:

sudo -u deploy docker login docker.surcontrol.com
# Enter DOCKER_REGISTRY_USER and DOCKER_REGISTRY_PASSWORD

Docker saves credentials in ~deploy/.docker/config.json. The CI pipeline will reuse these credentials when running docker compose pull.

2.5 Create the initial .env

sudo -u deploy cp /opt/wilo/.env.example /opt/wilo/.env
# Or create it manually from scratch:
sudo -u deploy tee /opt/wilo/.env <<'EOF'
POSTGRES_USER=postgres
POSTGRES_PASSWORD=CHANGE_ME
POSTGRES_DB=wilo
REDIS_PASSWORD=CHANGE_ME
STORAGE_ACCESS_KEY=minio
STORAGE_SECRET_KEY=CHANGE_ME
JWT_SECRET_KEY=CHANGE_ME_32_CHARS_MINIMUM
JWT_ISSUER=wilo-api
JWT_AUDIENCE=wilo-app
SMTP_FROM=noreply@clubwilo.com
WILO_VERSION=dev
EOF

Replace each CHANGE_ME with a strong random value:

# Generate a strong password
openssl rand -hex 24

# Generate a JWT key (minimum 32 chars)
openssl rand -hex 32

DANGER

POSTGRES_PASSWORD, REDIS_PASSWORD, STORAGE_SECRET_KEY, and JWT_SECRET_KEY must be strong random values. Never use the defaults in production.

2.6 Copy the Docker Compose files

The staging server needs docker-compose.yml and docker-compose.registry.yml from the repository. Copy them to /opt/wilo/:

# From your local machine (or from the repo):
scp docker-compose.yml docker-compose.registry.yml deploy@YOUR_STAGING_HOST:/opt/wilo/

Or clone the repo as the deploy user and symlink the files — whichever fits your workflow. The important thing is that /opt/wilo/docker-compose.yml and /opt/wilo/docker-compose.registry.yml exist before the first deploy.

---

Step 3 — Trigger a Deployment

Option A — Automatic (after a release)

Deployments happen after a semantic release on main. The flow:

1. Merge a MR with a conventional commit into main
   • fix(api): ... → patch release (1.0.0 → 1.0.1)
   • feat(app): ... → minor release (1.0.0 → 1.1.0)
   • feat!: ... or BREAKING CHANGE → major release (1.0.0 → 2.0.0)
2. The release job runs semantic-release, determines the new version, builds and pushes images
3. In the GitLab pipeline view, the deploy:staging job appears with a ▶ play button
4. Click ▶ — the job SSHes into the server and deploys

Option B — Manual re-deploy without a new release

If you need to redeploy the current version without a code change:

# SSH into the staging server as the deploy user
ssh deploy@YOUR_STAGING_HOST

cd /opt/wilo

# Pull and restart with the current WILO_VERSION in .env
docker compose -f docker-compose.yml -f docker-compose.registry.yml pull
docker compose -f docker-compose.yml -f docker-compose.registry.yml up -d

Verifying a deployment

After the deploy job finishes (or after a manual deploy), verify on the server:

ssh deploy@YOUR_STAGING_HOST

# Check running containers and their image versions
docker compose -f /opt/wilo/docker-compose.yml -f /opt/wilo/docker-compose.registry.yml ps

# Check the deployed version in .env
grep WILO_VERSION /opt/wilo/.env

# Check API health
curl http://localhost:5000/health/live
curl http://localhost:5000/health/ready

# Follow logs
docker compose -f /opt/wilo/docker-compose.yml -f /opt/wilo/docker-compose.registry.yml logs -f wilo-api

---

Step 4 — Rollback

To roll back to a previous version, update WILO_VERSION on the server and restart:

ssh deploy@YOUR_STAGING_HOST

cd /opt/wilo

# Set the version to roll back to (must exist in the registry)
sed -i "s/^WILO_VERSION=.*/WILO_VERSION=1.2.3/" .env

# Pull that specific version and restart
docker compose -f docker-compose.yml -f docker-compose.registry.yml pull
docker compose -f docker-compose.yml -f docker-compose.registry.yml up -d

The docker-compose.registry.yml overlay sets pull_policy: always, so Docker will always pull the version specified in WILO_VERSION.

---

Accessing Internal Services in Production

In production, no database or internal service port is exposed to the Internet. Use SSH tunnels to access them from your local machine.

# PostgreSQL — connect with any Postgres client on localhost:15432
ssh -L 15432:postgresdb:5432 deploy@YOUR_STAGING_HOST -N

# Redis
ssh -L 16379:redis:6379 deploy@YOUR_STAGING_HOST -N

# ClickHouse HTTP
ssh -L 18123:clickhouse:8123 deploy@YOUR_STAGING_HOST -N

# MinIO Console — open http://localhost:19001 in browser
ssh -L 19001:minio:9001 deploy@YOUR_STAGING_HOST -N

# Aspire Dashboard — open http://localhost:18888 in browser
ssh -L 18888:aspire-dashboard:18888 deploy@YOUR_STAGING_HOST -N

Run without -N and add -f to background the tunnel:

ssh -fN -L 15432:postgresdb:5432 deploy@YOUR_STAGING_HOST
psql -h localhost -p 15432 -U postgres -d wilo

---

Production Deployment (Planned)

Production will follow the exact same pattern as staging. The deploy:production job will be added to .gitlab-ci.yml with:

• PRODUCTION_SERVER_HOST CI variable instead of STAGING_SERVER_HOST
• needs: [deploy:staging] to enforce staging validation before production
• A separate environment: production block for GitLab environment tracking

---

Pending Migration Detection

The quality:api job includes a step that verifies all EF Core DbContexts have an up-to-date migration for every model change.

dotnet ef migrations has-pending-model-changes

This command compares the current code model against the *ModelSnapshot.cs file — no database connection required. It uses each module's IDesignTimeDbContextFactory<T> implementation.

DbContexts checked:

DbContext	Module
IdentityDbContext	src/Wilo.Modules.Identity
MembersDbContext	src/Wilo.Modules.Members
NotificationsDbContext	src/Wilo.Modules.Notifications
SocialDbContext	src/Wilo.Modules.Social
WiloDbContext	src/Wilo.Infrastructure

If a DbContext has pending model changes, the CI job fails with the exact dotnet ef migrations add command needed to fix it.

Run locally:

# Requires a prior Release build
bun run api:build
bun run api:migrations:check

When adding a new module: add a check line for the new DbContext to scripts/check-migrations.sh.

---

Caching Strategy

Each quality job caches its dependency directory, keyed by branch name with a fallback to main. This means MRs use the warm main cache on first run.

Job	Cache key	Cached path
quality:api	dotnet-{branch}	.nuget/packages/
quality:app	bun-app-{branch}	sites/Wilo.App/node_modules/
quality:mobile	bun-mobile-{branch}	sites/Wilo.Mobile/node_modules/
quality:email	bun-email-{branch}	services/Wilo.Email/node_modules/
quality:docs	bun-docs-{branch}	webs/Wilo.Docs/node_modules/

quality:docker has no cache — Docker layer caching within a DinD service is ephemeral by design.

---

Troubleshooting

deploy:staging fails with "Host key verification failed"

The ssh-keyscan step in the job populates ~/.ssh/known_hosts automatically. If it fails, verify that STAGING_SERVER_HOST is set correctly and the server is reachable from the GitLab runner.

deploy:staging fails with "Permission denied (publickey)"

1. Verify the public key is in /home/deploy/.ssh/authorized_keys on the server
2. Verify DEPLOY_SSH_PRIVATE_KEY contains the matching private key (full content including header/footer lines)
3. Verify the variable is Protected and you are deploying from main

docker compose pull fails with "unauthorized"

The deploy user on the server must be logged into the private registry:

sudo -u deploy docker login docker.surcontrol.com

Images not updating after deploy

Check that WILO_VERSION in /opt/wilo/.env was updated to the new version and that docker-compose.registry.yml has pull_policy: always.

Release job runs but no new version is created

semantic-release only creates a release when there is at least one feat, fix, or BREAKING CHANGE commit since the last tag. Commits with types like chore, docs, style, test, or refactor do not trigger a release.