[EMB-115] Migrate package release from Cloudsmith to GAR #8

jellespijker · 2026-01-18T10:01:03Z

EMB-115 Migrate to Google Artifact Registry (GAR) with Workload Identity Federation and SemVer 2.0

📋 Summary

This PR modernizes the embedded firmware CI/CD infrastructure by migrating from Cloudsmith to Google Artifact Registry (GAR). Key highlights include the implementation of Workload Identity Federation (WIF) for secure, credential-free GCP authentication and a comprehensive Semantic Versioning 2.0 strategy.

📦 Semantic Versioning 2.0 Strategy

Version generation is now fully automated. The format follows the standard SemVer 2.0 structure:

📊 SemVer Workflow Diagram

This diagram illustrates how GitHub events trigger specific versioning and repository logic:

graph TB
    Start([GitHub Event]) --> EventType{Event Type?}
    
    EventType -->|Push to Branch| BranchType{Is Master Branch?}
    EventType -->|Push Tag| TagPush[Tag Push Event]
    EventType -->|Pull Request| PREvent[Pull Request Event]
    
    %% Master Branch Flow
    BranchType -->|Yes: main/master/stable| MasterFlow[Master Branch Merge]
    MasterFlow --> GetLatestTag1[Get Latest Tag via git describe]
    GetLatestTag1 --> BumpPatch1[Bump Patch Version]
    BumpPatch1 --> NightlyFormat[Format: X.Y.Z-alpha.0+YYYYMMDD.branch.sha]
    NightlyFormat --> NightlyRepo[Repository: nightly-builds]
    
    %% Regular Branch Flow
    BranchType -->|No: feature/bugfix| RegularBranch[Regular Branch Commit]
    RegularBranch --> GetLatestTag2[Get Latest Tag via git describe]
    GetLatestTag2 --> BumpPatch2[Bump Patch Version]
    BumpPatch2 --> AlphaFormat[Format: X.Y.Z-alpha.0+sha]
    AlphaFormat --> DevRepo1[Repository: packages-dev]
    
    %% Pull Request Flow
    PREvent --> GetLatestTag3[Get Latest Tag via git describe]
    GetLatestTag3 --> BumpPatch3[Bump Patch Version]
    BumpPatch3 --> PRFormat[Format: X.Y.Z-alpha.0+sha]
    PRFormat --> DevRepo2[Repository: packages-dev]
    
    %% Tag Flow
    TagPush --> ParseTag{Parse Tag}
    ParseTag -->|Valid SemVer Tag| CheckPreRelease{Contains Prerelease?}
    ParseTag -->|Invalid Format| FallbackVersion[Fallback to Auto-Generated]
    FallbackVersion --> GetLatestTag4[Get Latest Tag]
    GetLatestTag4 --> BumpPatch4[Bump Patch Version]
    BumpPatch4 --> DevRepo3[Repository: packages-dev]
    
    %% Tag with Prerelease
    CheckPreRelease -->|Yes: alpha/beta/rc| PreReleaseTag[X.Y.Z-prerelease.N]
    PreReleaseTag --> DevRepo4[Repository: packages-dev]
    
    %% Production Release
    CheckPreRelease -->|No: Production| ProductionTag[X.Y.Z]
    ProductionTag --> ProdRepo[Repository: packages-released]
    
    %% Styling
    classDef masterStyle fill:#4CAF50,stroke:#2E7D32,stroke-width:3px,color:#fff
    classDef devStyle fill:#2196F3,stroke:#1565C0,stroke-width:3px,color:#fff
    classDef prodStyle fill:#F44336,stroke:#C62828,stroke-width:3px,color:#fff
    classDef processStyle fill:#FF9800,stroke:#E65100,stroke-width:2px,color:#fff
    
    class NightlyRepo masterStyle
    class DevRepo1,DevRepo2,DevRepo3,DevRepo4 devStyle
    class ProdRepo prodStyle
    class MasterFlow,RegularBranch,PREvent,TagPush processStyle

🏗️ Version Format Breakdown

The version components provide the following information:

graph LR
    Version[1.2.3-beta.1+20260118.main.abc1234]
    Version --> Major[Major: 1]
    Version --> Minor[Minor: 2]
    Version --> Patch[Patch: 3]
    Version --> PreRelease[Prerelease: beta.1]
    Version --> BuildMeta[Build Metadata: 20260118.main.abc1234]
    
    Major --> MajorDesc[Breaking Changes<br/>Incompatible API]
    Minor --> MinorDesc[New Features<br/>Backward Compatible]
    Patch --> PatchDesc[Bug Fixes<br/>Backward Compatible]
    PreRelease --> PreDesc[Not Stable<br/>alpha → beta → rc]
    BuildMeta --> MetaDesc[Traceability<br/>Date, Branch, SHA]
    
    classDef componentStyle fill:#9C27B0,stroke:#6A1B9A,stroke-width:2px,color:#fff
    classDef descStyle fill:#E1BEE7,stroke:#9C27B0,stroke-width:1px,color:#333
    
    class Major,Minor,Patch,PreRelease,BuildMeta componentStyle
    class MajorDesc,MinorDesc,PatchDesc,PreDesc,MetaDesc descStyle

📦 Repository Distribution Strategy

Packages are automatically routed to the correct repository based on their maturity:

graph TB
    Commit([Code Commit/Tag]) --> Decision{What is it?}
    
    Decision -->|Master Merge| Nightly[Nightly Build]
    Decision -->|Alpha Tag| AlphaTag[Alpha Release]
    Decision -->|Beta Tag| BetaTag[Beta Release]
    Decision -->|RC Tag| RCTag[RC Release]
    Decision -->|Release Tag| ReleaseTag[Production Release]
    Decision -->|Other Commit| DevCommit[Development Commit]
    
    Nightly --> NightlyRepo[(nightly-builds<br/>X.Y.Z-alpha.0+YYYYMMDD.branch.sha)]
    AlphaTag --> DevRepo1[(packages-dev<br/>X.Y.Z-alpha.N)]
    BetaTag --> DevRepo2[(packages-dev<br/>X.Y.Z-beta.N)]
    RCTag --> DevRepo3[(packages-dev<br/>X.Y.Z-rc.N)]
    DevCommit --> DevRepo4[(packages-dev<br/>X.Y.Z-alpha.0+sha)]
    ReleaseTag --> ProdRepo[(packages-released<br/>X.Y.Z)]
    
    NightlyRepo --> NightlyUse[Daily Testing<br/>Continuous Integration]
    DevRepo1 --> DevUse1[Internal Testing]
    DevRepo2 --> DevUse2[QA Testing]
    DevRepo3 --> DevUse3[Pre-Production Validation]
    DevRepo4 --> DevUse4[Development Testing]
    ProdRepo --> ProdUse[Production Deployment<br/>Customer Releases]
    
    classDef nightlyStyle fill:#673AB7,stroke:#4527A0,stroke-width:3px,color:#fff
    classDef devStyle fill:#2196F3,stroke:#1565C0,stroke-width:3px,color:#fff
    classDef prodStyle fill:#4CAF50,stroke:#2E7D32,stroke-width:3px,color:#fff
    
    class NightlyRepo nightlyStyle
    class DevRepo1,DevRepo2,DevRepo3,DevRepo4 devStyle
    class ProdRepo prodStyle

1. Automatic Patch Bumping

To prevent version conflicts and remove manual overhead, patch versions are automatically incremented from the latest reachable tag.

graph LR
    A[Latest Tag: v1.2.3] --> B[Auto Bump Patch]
    B --> C[Next Version: 1.2.4]
    C --> D[Add Prerelease: 1.2.4-alpha.0]
    D --> E[Add Build Meta: +sha]
    E --> F[Final: 1.2.4-alpha.0+abc1234]
    
    style A fill:#FFE0B2,stroke:#E65100,stroke-width:2px
    style F fill:#C8E6C9,stroke:#2E7D32,stroke-width:2px

Why auto-bump?

Ensures every commit has a unique, incrementing version
No manual version management needed
Prevents version conflicts
Note: This is a naive bump - it always increments patch. For major/minor changes, create an explicit tag.

🔐 Security & Authentication

Workload Identity Federation (WIF)

The primary security improvement is the transition to Workload Identity Federation. This removes the need for long-lived Service Account keys in GitHub Secrets.

Warning

A GitHub PAT is currently retained only for checkouts involving recursive sub-modules. This will be phased once a GitHub App has been setup for generating short-lived tokens.

🚀 Platform Team: Setup Steps

Note

Developers: You do not need to perform these steps. This infrastructure is managed by the Platform/DevOps team aka @Ultichiel or @jellespijker .

🚀 Setup Steps

Step 1: Create Workload Identity Pool (5 min)

gcloud iam workload-identity-pools create github-pool \
  --location="global" \
  --display-name="GitHub Actions Pool" \
  --description="Workload Identity Pool for GitHub Actions workflows"

Save this output: Full pool resource name

Step 2: Create Workload Identity Provider (5 min)

gcloud iam workload-identity-pools providers create-oidc github-provider \
  --location="global" \
  --workload-identity-pool="github-pool" \
  --issuer-uri="https://token.actions.githubusercontent.com" \
  --attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository,attribute.repository_owner=assertion.repository_owner,attribute.actor=assertion.actor" \
  --attribute-condition="attribute.repository_owner=='Ultimaker'" \
  --display-name="GitHub OIDC Provider"

Save this output: Full provider resource name
Format: projects/893160625502/locations/global/workloadIdentityPools/github-pool/providers/github-provider

Step 3: Create Service Accounts (5 min)

# For Docker operations
gcloud iam service-accounts create docker-ci \
  --display-name="Docker CI/CD Service Account" \
  --description="Service account for Docker image operations in GitHub Actions"

# For APT package operations
gcloud iam service-accounts create apt-ci \
  --display-name="APT Package CI/CD Service Account" \
  --description="Service account for APT package operations in GitHub Actions"

Save these emails:

Docker: docker-ci@dev-embedded.iam.gserviceaccount.com
APT: apt-ci@dev-embedded.iam.gserviceaccount.com

Step 4: Create GAR Repositories (10 min)

# Docker repository for firmware images
gcloud artifacts repositories create firmware-images \
  --repository-format=docker \
  --location=europe-west1 \
  --description="Docker images for embedded firmware projects"

# APT repository for nightly builds
gcloud artifacts repositories create nightly-builds \
  --repository-format=apt \
  --location=europe-west1 \
  --description="Nightly alpha builds from master/main branches"

# APT repository for development packages
gcloud artifacts repositories create packages-dev \
  --repository-format=apt \
  --location=europe-west1 \
  --description="Pre-release packages (alpha, beta, rc)"

# APT repository for released packages
gcloud artifacts repositories create packages-released \
  --repository-format=apt \
  --location=europe-west1 \
  --description="Production release packages"

Step 5: Grant IAM Permissions (5 min)

# Docker service account permissions
gcloud artifacts repositories add-iam-policy-binding firmware-images \
  --location=europe-west1 \
  --member="serviceAccount:docker-ci@dev-embedded.iam.gserviceaccount.com" \
  --role="roles/artifactregistry.writer"

# APT service account permissions
for repo in nightly-builds packages-dev packages-released; do
  gcloud artifacts repositories add-iam-policy-binding "$repo" \
    --location=europe-west1 \
    --member="serviceAccount:apt-ci@dev-embedded.iam.gserviceaccount.com" \
    --role="roles/artifactregistry.writer"
done

Step 6: Bind Workload Identity - PER REPOSITORY (2 min each)

This must be done for EACH repository that will use the workflows.

Replace:

REPO_NAME with repository name (e.g., opinicus, um-kernel)

# Bind Docker service account to repository
gcloud iam service-accounts add-iam-policy-binding docker-ci@dev-embedded.iam.gserviceaccount.com \
  --role="roles/iam.workloadIdentityUser" \
  --member="principalSet://iam.googleapis.com/projects/893160625502/locations/global/workloadIdentityPools/github-pool/attribute.repository/Ultimaker/REPO_NAME"

# Bind APT service account to repository
gcloud iam service-accounts add-iam-policy-binding apt-ci@dev-embedded.iam.gserviceaccount.com \
  --role="roles/iam.workloadIdentityUser" \
  --member="principalSet://iam.googleapis.com/projects/893160625502/locations/global/workloadIdentityPools/github-pool/attribute.repository/Ultimaker/REPO_NAME"

Repositories to bind:

opinicus
(add others as needed)

Step 7: Configure GitHub Organization Variables (5 min)

Important

I'm currently using the WIF and GAR in the neoprep-staging GCP

"New organization variable" for each:

Variable Name	Value	Notes
`EMB_DOCKER_REGISTRY`	`europe-west1-docker.pkg.dev`	Registry URL
`EMB_GCP_PROJECT`	`dev-embedded`	From step 1
`EMB_GCP_LOCATION`	`europe-west1`	Same as repos
`EMB_DOCKER_REPOSITORY`	`firmware-images`	From step 4
`EMB_WI_PROVIDER`	`projects/893160625502/locations/global/workloadIdentityPools/github-pool/providers/github-provider`	From step 2
`EMB_DOCKER_SERVICE_ACCOUNT`	`docker-ci@dev-embedded.iam.gserviceaccount.com`	From step 3
`EMB_APT_SERVICE_ACCOUNT`	`apt-ci@dev-embedded.iam.gserviceaccount.com`	From step 3

Important: Set as organization variables, not repository variables, so all repos can access them.

Step 8: Enable Required APIs (2 min)

gcloud services enable \
  artifactregistry.googleapis.com \
  iamcredentials.googleapis.com \
  cloudresourcemanager.googleapis.com \
  sts.googleapis.com

✅ Verification

Verify Workload Identity Pool

gcloud iam workload-identity-pools describe github-pool --location=global

Verify Provider

gcloud iam workload-identity-pools providers describe github-provider \
  --workload-identity-pool=github-pool \
  --location=global

Verify Service Accounts

gcloud iam service-accounts list | grep -E 'docker-ci|apt-ci'

Verify GAR Repositories

gcloud artifacts repositories list --location=europe-west1

Verify Docker Permissions

gcloud artifacts repositories get-iam-policy firmware-images \
  --location=europe-west1 | grep docker-ci

Verify APT Permissions

gcloud artifacts repositories get-iam-policy packages-released \
  --location=europe-west1 | grep apt-ci

Verify Workload Identity Binding (per repository)

# Replace REPO_NAME with actual repository
gcloud iam service-accounts get-iam-policy docker-ci@dev-embedded.iam.gserviceaccount.com \
  --format=json | grep "Ultimaker/REPO_NAME"

💻 How to install packages locally or on a printer

Developers can pull Debian packages directly from GAR for local testing or printer installation.

1. Authenticate your machine

Ensure you have the Google Cloud CLI installed and are logged in:

gcloud auth login
gcloud auth application-default login

2. Configure the APT Helper

GAR requires an authentication helper to allow apt to communicate with Google's registries.

Install the helper: sudo apt install apt-transport-artifact-registry
Add the repository to your sources (example for packages-dev):

echo "deb [signed-by=/usr/share/keyrings/google-artifact-registry-active-keyring.gpg] https://europe-west1-apt.pkg.dev/projects/dev-embedded/packages-dev buster main" | sudo tee /etc/apt/sources.list.d/ultimaker.list

3. Install packages

Standard APT workflows now apply:

sudo apt update
sudo apt install your-firmware-package

✅ Verification Checklist

Workload Identity Pool/Provider active.
Organization-level variables configured in GitHub.
SemVer generation script tested with vX.Y.Z and pre-release tags.
Initial end-to-end test successful on opinicus.

This commit migrates the Debian package release process from Cloudsmith to Google Artifact Registry (GAR). The `release_pkg.yml` workflow has been updated to: - Authenticate with Google Cloud using Workload Identity Federation. - Use `gcloud artifacts apt upload` to publish packages. - Automatically extract package metadata from the `.deb` file. - Generate a detailed job summary with instructions on how to install the package from the new GAR repository. The now-unused `cloudsmith.Dockerfile` has been removed.

This commit refactors the CI versioning logic to fully support Semantic Versioning 2.0. The version generation logic has been extracted from the `prepare_env.yml` workflow into a dedicated, reusable, and testable shell script. Key changes include: - A new script `generate_semver_version.sh` that determines the `RELEASE_VERSION` and `RELEASE_REPO` based on the Git context (branch, tag, PR). - Support for `alpha`, `beta`, and `rc` pre-release tags, in addition to final release tags. - Automatic patch version bumping for development and nightly builds based on the latest Git tag. - Nightly builds for master branch merges now produce versions like `X.Y.Z-alpha.0+YYYYMMDD.branch.sha`. - Development builds for pull requests and feature branches generate versions like `X.Y.Z-alpha.0+sha`. - A comprehensive test script, `test_generate_semver_version.sh`, is introduced to validate the versioning logic across various scenarios. - The `prepare_env` workflow now includes a summary step that clearly documents the generated version, target repository, and the versioning strategy for better visibility in GitHub Actions.

The `release_pkg` workflow was hardcoded to find `.deb` packages in the `./dist` directory. This change removes that assumption, making the workflow search for packages in the current working directory. This allows for more flexibility in the preceding build job, which no longer needs to place artifacts in a `dist` folder. The documentation in the job summary has been updated accordingly.

Not needed anymore and only clutter the logs

This commit refactors the GitHub Actions workflows to use a new centralized build script located in the `embedded-workflows` repository. Key changes include: - Updating `build.yml`, `unit_test.yml`, `shellcheck.yml`, and `prepare_env.yml` to check out the `embedded-workflows` repository and call the new script. - Adding authentication steps for Google Artifact Registry (GAR) using Workload Identity Federation, removing the dependency on PATs for Docker operations. - Overhauling the `release_docker_img.yml` workflow to use modern Docker actions (`docker/metadata-action`, `docker/build-push-action`) for building and pushing images to GAR. - Introducing a new reusable workflow `docker_build.yml` for standardized Docker image builds. - Adding the centralized `build_for_ultimaker.sh` script.

This commit standardizes the checkout process for the `embedded-workflows` repository across all reusable workflows. - An `embedded_workflows_branch` input is added to the `release_pkg` workflow to align it with other workflows. - The default branch for checking out `embedded-workflows` is temporarily set to `EMB-115_migrate_to_gar` across all workflows to facilitate ongoing development. - The checkout step in `release_pkg` is cleaned up by removing an unnecessary PAT and fetch-depth. - A redundant "Security Best Practices" summary is removed from the `docker_build` workflow.

The `UMLM_ENCRYPTION_KEY` is now passed as a temporary environment variable directly to the `build_for_ultimaker.sh` script instead of being exported for the entire step. This improves security by limiting the scope of the secret. Additionally, `TODO` comments in the CI workflows have been standardized.

The static analysis Docker images (clang-tidy, clang-format, cppcheck) are hardened by creating and running as a non-privileged user. Additionally, the shellcheck script is updated to mount the source directory as read-only to prevent accidental modifications.

This reverts commit 8b8d996.

This commit removes the dependency on the `embedded-workflows` repository by deleting the centralized `build_for_ultimaker.sh` script and the `docker_build.yml` workflow. The CI workflows (`build.yml`, `release_pkg.yml`, `unit_test.yml`, `shellcheck.yml`, `prepare_env.yml`, and `release_docker_img.yml`) are updated to no longer check out the `embedded-workflows` repository. Instead, they now rely on local scripts and configurations. Key changes include: - Deleting `scripts/build_for_ultimaker.sh` and `.github/workflows/docker_build.yml`. - Updating workflow files to remove the checkout step for `embedded-workflows`. - Simplifying Docker authentication and build steps, removing logic related to Google Artifact Registry (GAR) in favor of direct Docker commands and GitHub Container Registry (`ghcr.io`) login where needed. - Adjusting build, test, and shellcheck jobs to call local scripts directly. - Cleaning up unused inputs and environment variables from the workflows.

jellespijker added 16 commits January 18, 2026 10:57

[EMB-115] Document SemVer generation script and workflow

e99b251

[EMB-115] Remove versioning strategy from job summary

fd3a250

[EMB-115] Remove GH Action Debug steps

fded57c

Not needed anymore and only clutter the logs

[EMB-115] Use secrets for passing GITHUB_TOKEN to docker build

e2c8cba

[EMB-115] Remove fallback for DOCKER_REGISTRY variable

3172128

[EMB-115] Grant write permissions to prepare environment job

8b8d996

Revert "[EMB-115] Grant write permissions to prepare environment job"

9634926

This reverts commit 8b8d996.

[EMB-115] Make build_for_ultimaker.sh executable

2e7df8b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[EMB-115] Migrate package release from Cloudsmith to GAR #8

[EMB-115] Migrate package release from Cloudsmith to GAR #8

Uh oh!

jellespijker commented Jan 18, 2026 •

edited by atlassian bot

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[EMB-115] Migrate package release from Cloudsmith to GAR #8

Are you sure you want to change the base?

[EMB-115] Migrate package release from Cloudsmith to GAR #8

Uh oh!

Conversation

jellespijker commented Jan 18, 2026 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

EMB-115 Migrate to Google Artifact Registry (GAR) with Workload Identity Federation and SemVer 2.0

📋 Summary

📦 Semantic Versioning 2.0 Strategy

📊 SemVer Workflow Diagram

🏗️ Version Format Breakdown

📦 Repository Distribution Strategy

1. Automatic Patch Bumping

🔐 Security & Authentication

Workload Identity Federation (WIF)

🚀 Platform Team: Setup Steps

🚀 Setup Steps

Step 1: Create Workload Identity Pool (5 min)

Step 2: Create Workload Identity Provider (5 min)

Step 3: Create Service Accounts (5 min)

Step 4: Create GAR Repositories (10 min)

Step 5: Grant IAM Permissions (5 min)

Step 6: Bind Workload Identity - PER REPOSITORY (2 min each)

Step 7: Configure GitHub Organization Variables (5 min)

Step 8: Enable Required APIs (2 min)

✅ Verification

Verify Workload Identity Pool

Verify Provider

Verify Service Accounts

Verify GAR Repositories

Verify Docker Permissions

Verify APT Permissions

Verify Workload Identity Binding (per repository)

💻 How to install packages locally or on a printer

1. Authenticate your machine

2. Configure the APT Helper

3. Install packages

✅ Verification Checklist

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jellespijker commented Jan 18, 2026 •

edited by atlassian bot

Loading