Find a way to use S3 as a cache backend for our Github actions

[marcoacierno]

We build Docker images in our GitHub actions. We should store the docker layer caches on S3 over GH Caches to optimise performance.

[marcoacierno]

@claude research how to do this

[claude]

Claude finished @marcoacierno's task in 1m 1s —— View job

---

Research: S3 Cache Backend for GitHub Actions Docker Builds

• Analyze current GitHub Actions workflow for Docker builds
• Research S3 cache backend options for Docker layer caching
• Document findings and recommendations

---

Current Setup

Found 4 workflows using docker/build-push-action with GitHub Actions cache:

• build-backend.yml
• build-frontend.yml
• build-pretix.yml
• build-base-pretix.yml

All currently use:

cache-from: type=gha
cache-to: type=gha,mode=max

---

Recommendation: Use Docker BuildKit S3 Cache Backend

Docker BuildKit natively supports S3 as a cache backend. This is the recommended approach for your use case.

How to Implement

Replace the current cache-from and cache-to with S3 configuration:

- name: Build and push
  uses: docker/build-push-action@v6
  with:
    context: ./backend
    file: ./backend/Dockerfile
    builder: ${{ steps.buildx.outputs.name }}
    provenance: false
    push: true
    tags: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.eu-central-1.amazonaws.com/pythonit/pycon-backend:arm-${{ inputs.githash }}
    cache-from: type=s3,region=eu-central-1,bucket=<your-cache-bucket>,name=pycon-backend
    cache-to: type=s3,region=eu-central-1,bucket=<your-cache-bucket>,name=pycon-backend,mode=max
    platforms: linux/arm64

S3 Cache Configuration Options

Option	Description
region	AWS region (e.g., eu-central-1)
bucket	S3 bucket name for storing cache
name	Cache key/prefix (use different names for different images)
mode	min (default) or max - max caches all layers
prefix	Optional prefix within the bucket
blobs_prefix	Prefix for blob storage (default: blobs/)
manifests_prefix	Prefix for manifests (default: manifests/)
access_key_id	AWS access key (can also use env vars)
secret_access_key	AWS secret key (can also use env vars)

Authentication

Since you already have AWS credentials configured via aws-actions/configure-aws-credentials@v4, BuildKit will automatically use the environment variables:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_SESSION_TOKEN (if using STS)

No changes needed for authentication!

---

Setup Steps

1. Create an S3 bucket for the cache (e.g., pythonit-docker-cache)

2. Configure bucket policy/IAM permissions:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "s3:GetObject",
           "s3:PutObject",
           "s3:DeleteObject",
           "s3:ListBucket"
         ],
         "Resource": [
           "arn:aws:s3:::pythonit-docker-cache",
           "arn:aws:s3:::pythonit-docker-cache/*"
         ]
       }
     ]
   }

3. Add lifecycle policy to auto-expire old cache (recommended):

   {
     "Rules": [
       {
         "ID": "ExpireOldCache",
         "Status": "Enabled",
         "Expiration": {
           "Days": 30
         }
       }
     ]
   }

4. Update workflows with S3 cache config (see example above)

---

Benefits of S3 over GHA Cache

Feature	GHA Cache	S3 Cache
Size limit	10 GB per repo	Unlimited
Retention	7 days unused	Configurable
Cross-repo sharing	No	Yes
Speed	Good	Often faster for large caches
Cost	Free (within limits)	S3 storage/transfer costs
Eviction policy	LRU	Configurable lifecycle

---

Example: Full Updated Workflow

- name: Build and push
  uses: docker/build-push-action@v6
  with:
    context: ./backend
    file: ./backend/Dockerfile
    builder: ${{ steps.buildx.outputs.name }}
    provenance: false
    push: true
    tags: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.eu-central-1.amazonaws.com/pythonit/pycon-backend:arm-${{ inputs.githash }}
    cache-from: type=s3,region=eu-central-1,bucket=${{ secrets.DOCKER_CACHE_BUCKET }},name=pycon-backend
    cache-to: type=s3,region=eu-central-1,bucket=${{ secrets.DOCKER_CACHE_BUCKET }},name=pycon-backend,mode=max
    platforms: linux/arm64

---

Alternative: Registry Cache (ECR)

If you prefer not to manage a separate S3 bucket, you can also use ECR as a cache backend:

cache-from: type=registry,ref=${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.eu-central-1.amazonaws.com/pythonit/pycon-backend:cache
cache-to: type=registry,ref=${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.eu-central-1.amazonaws.com/pythonit/pycon-backend:cache,mode=max

This stores cache as OCI images in your existing ECR registry.

---

Documentation References

• Docker BuildKit S3 Cache Backend
• docker/build-push-action Cache Documentation

[marcoacierno]

@claude implement the approach using "Docker BuildKit S3 Cache Backend"

[claude]

Claude finished @marcoacierno's task in 1m 16s —— View job

---

Implementing S3 Cache Backend for Docker Builds

• Read current workflow files to understand the structure
• Update build-backend.yml with S3 cache configuration
• Update build-frontend.yml with S3 cache configuration
• Update build-pretix.yml with S3 cache configuration
• Update build-base-pretix.yml with S3 cache configuration
• Commit and push changes Unable to push (permission limitation)

---

Summary

I've implemented the S3 cache backend changes for all 4 Docker build workflows, but I cannot push the changes due to GitHub App permission restrictions. Workflow files (.github/workflows/*) require workflows permission which my GitHub App doesn't have.

Changes Made (locally committed)

The commit 8482f61 includes these changes:

build-backend.yml:

cache-from: type=s3,region=eu-central-1,bucket=${{ secrets.DOCKER_CACHE_BUCKET }},name=pycon-backend
cache-to: type=s3,region=eu-central-1,bucket=${{ secrets.DOCKER_CACHE_BUCKET }},name=pycon-backend,mode=max

build-frontend.yml:

cache-from: type=s3,region=eu-central-1,bucket=${{ secrets.DOCKER_CACHE_BUCKET }},name=pycon-frontend
cache-to: type=s3,region=eu-central-1,bucket=${{ secrets.DOCKER_CACHE_BUCKET }},name=pycon-frontend,mode=max

build-pretix.yml:

cache-from: type=s3,region=eu-central-1,bucket=${{ secrets.DOCKER_CACHE_BUCKET }},name=pretix
cache-to: type=s3,region=eu-central-1,bucket=${{ secrets.DOCKER_CACHE_BUCKET }},name=pretix,mode=max

build-base-pretix.yml:

• Added AWS credentials step (this workflow previously only had GHCR login)
• Updated cache configuration:

cache-from: type=s3,region=eu-central-1,bucket=${{ secrets.DOCKER_CACHE_BUCKET }},name=base-pretix
cache-to: type=s3,region=eu-central-1,bucket=${{ secrets.DOCKER_CACHE_BUCKET }},name=base-pretix,mode=max

Manual Steps Required

1. Apply the changes manually - You'll need to make these edits yourself or ask someone with workflow write access

2. Create the S3 bucket (e.g., pythonit-docker-cache) with:

   • IAM policy granting s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket
   • Optional: Lifecycle rule to expire objects after 30 days

3. Add repository secret: DOCKER_CACHE_BUCKET with the bucket name

[marcoacierno]

@claude can you create a branch where you create the bucket on terraform in the global infrastructure part?

[claude]

Claude finished @marcoacierno's task in 1m 35s —— View job • claude/issue-4536-20260119-1801 • Create PR ➔

---

Creating S3 Cache Bucket in Terraform

• Explore global infrastructure Terraform files
• Create S3 bucket resource for Docker cache
• Add lifecycle policy for cache expiration
• Commit and push changes

---

Summary

Created infrastructure/global/buckets/docker-cache.tf with:

• S3 Bucket: pythonit-docker-cache
• Encryption: AES256 server-side encryption
• Lifecycle Policy: Automatic expiration after 30 days
• Output: docker_cache_bucket_name for reference

Next Steps

After applying this Terraform:

1. Add DOCKER_CACHE_BUCKET secret to the repository with value pythonit-docker-cache
2. Update the GitHub Actions workflows to use S3 cache (as documented in previous comments)

|