You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
title: IaaC Deploy setup to Amazon EC2 with GitHub actions, Deocker, terraform and self-hosted Docker Registry
3
+
title: "IaaC Simplified: Automating EC2 Deployments with GitHub Actions, Terraform, Docker & Distribution Registry"
4
4
authors: ivanb
5
5
tags: [aws, terraform, github-actions]
6
+
description: "The ultimate step-by-step guide to cost-effective, build-time-efficient, and easy managable EC2 deployments using GitHub Actions, Terraform, Docker, and a self-hosted registry."
7
+
image: "/ogs/ga-tf-aws.jpg"
6
8
---
7
9
10
+
11
+
12
+
13
+
8
14
This guide shows how to deploy own Docker apps (with AdminForth as example) to Amazon EC2 instance with Docker and Terraform involving Docker self-hosted registry.
9
15
10
16
Needed resources:
11
17
- GitHub actions Free plan which includes 2000 minutes per month (1000 of 2-minute builds per month - more then enough for many projects, if you are not running tests etc). Extra builds would cost `0.008$` per minute.
12
-
- AWS account where we will auto-spawn EC2 instance. We will use t3a.small instance (2 vCPUs, 2GB RAM) which costs `~14$` per month in `us-east-1` region (cheapest region).
13
-
- $2 per month for EBS gp2 storage (20GB) for EC2 instance
18
+
- AWS account where we will auto-spawn EC2 instance. We will use `t3a.small` instance (2 vCPUs, 2GB RAM) which costs `~14$` per month in `us-east-1` region (cheapest region). Also it will take `$2` per month for EBS gp2 storage (20GB) for EC2 instance
14
19
15
20
This is it, registry will be auto-spawned on EC2 instance, so no extra costs for it. Also GitHub storage is not used, so no extra costs for it.
16
21
17
22
The setup has next features:
18
-
- Build process is done using IaaC approach with HashiCorp Terraform, so almoast no manual actions are needed from you. Every resource including EC2 server instance is described in code which is commited to repo and should not be manually clicked.
23
+
- Build process is done using IaaC approach with HashiCorp Terraform, so almoast no manual actions are needed from you. Every resource including EC2 server instance is described in code which is commited to repo so no manual clicks are needed.
19
24
- Docker build process is done on GitHub actions, so EC2 server is not overloaded
20
-
- Changes in infrastructure including changing server type, adding S3 Bucket, changing size of sever Disk is also done in code and can be done by commiting code to repo.
25
+
- Changes in infrastructure including changing server type, adding S3 Bucket, changing size of sever disk is also can be done by commiting code to repo.
21
26
- Docker images and cache are stored on EC2 server, so no extra costs for Docker registry are needed.
22
-
- Total build time for average commit to AdminForth app (with vite rebuilds) is around 2 minutes.
27
+
- Total build time for average commit to AdminForth app (with Vite rebuilds) is around 2 minutes.
23
28
24
29
<!-- truncate -->
25
30
@@ -35,9 +40,9 @@ Quick difference between approaches from previous post and current post:
35
40
36
41
| Feature | Without Registry | With Registry |
37
42
| --- | --- | --- |
38
-
| How and where docker build happens | Source code is copied to EC2 and docker build is done there | Docker build is done on CI and docker image is pushed to registry (in this post we run registry automatically on EC2) |
39
-
| Docker build layers cache | Cache is stored on EC2 | GitHub actions has no own Docker cache out of the box, so it should be stored in dedicated place (we use self-hosted registry on the CI as it is free) |
40
-
| Advantages |Much simpler setup with less code (we don't need code to run registry and make it trusted, secured, and don't need extra cache setup as is naturally stored on EC2). | Build is done on CI, so EC2 server is not overloaded. For cheap/middle EC2s CI builds are faster than on EC2. Plus time is saved because we don't need to copy source code to EC2 |
43
+
| How and where docker build happens | Source code is rsync-ed from CI to EC2 and docker build is done there | Docker build is done on CI and docker image is pushed to registry (in this post we run registry automatically on EC2) |
44
+
|How Docker build layers are cached | Cache is stored on EC2 | GitHub actions has no own Docker cache out of the box, so it should be stored in dedicated place (we use self-hosted registry on the EC2 as it is free) |
45
+
| Advantages |Simpler setup with less code (we don't need code to run and secure registry, and don't need extra cache setup as is naturally persisted on EC2). | Build is done on CI, so EC2 server is not overloaded. For most cases CI builds are faster than on EC2. Plus time is saved because we don't need to rsync source code to EC2 |
41
46
| Disadvantages | Build on EC2 requires additional server RAM / overloads CPU | More terraform code is needed. registry cache might require small extra space on EC2 |
42
47
43
48
@@ -46,40 +51,56 @@ Quick difference between approaches from previous post and current post:
46
51
A little bit of theory.
47
52
48
53
When you move build process to CI you have to solve next chellenges:
49
-
1) We need to deliver built docker images to EC2 somehow
54
+
1) We need to deliver built docker images to EC2 somehow (and only we)
50
55
2) We need to persist cache between builds
51
56
52
57
### Delivering images
53
58
54
59
#### Exporing images to tar files
55
60
56
-
Simplest option which you can find is save docker images to tar files and deliver them to EC2. We can easily do it in terraform (using `docker save -o ...` command on CI and `docker load ...` command on EC2). However this option has a significant disadvantage - it is slow. Docker images are big (always include all layers, without any options), so it takes infinity to do save/load and another infinity to transfer them to EC2 (via relatively slow rsync/SSH and relatively GitHub actions outbound connection).
61
+
Simplest option which you can find is save docker images to tar files and deliver them to EC2. We can easily do it in terraform (using `docker save -o ...` command on CI and `docker load ...` command on EC2). However this option has a significant disadvantage - it is slow. Docker images are big (always include all layers, without any options), so it takes infinity to do save/load and another infinity to transfer them to EC2 (via relatively slow rsync/SSH and relatively slow GitHub actions outbound connection).
57
62
58
63
#### Docker registry
59
64
60
-
Second and right option which we will use here - involve Docker registry. Docker registry is a repository which stores docker images. It does storing in a smart way - it stores layers, so if you will update last layer and push it from CI to registry, only last layer will be pushed to registry and then pulled to EC2.
61
-
To give you row compare - whole-layers image might took `1GB`, but last layer created by `npm run build` command might take `50MB`. And most builds you will do only last layer changes, so it will be 20 times faster to push/pull last layer than whole image.
62
-
And this is not all, registry uses unencrypted HTTP/2 protocol so it is faster then SSH/rsync encrypted connection. However you have to be careful with this point and provide another way of authentication (so only you and your CI/EC2 can push/pull images).
65
+
Faster, right option which we will use here - involve Docker registry. Registry is a repository which stores docker images. It does it in a smart way - it saves each image as several layers, so if you will update last layer, then only last layer will be pushed to registry and then only last will be pulled to EC2.
66
+
To give you row compare - whole-layers image might take `1GB`, but last layer created by `npm run build` command might take `50MB`. And most builds you will do only last layer changes, so it will be 20 times faster to push/pull last layer than whole image.
67
+
And this is not all, registry uses TLS HTTP protocol so it is faster then SSH/rsync encrypted connection.
68
+
69
+
Of course you have to care about a way of registry authentication (so only you and your CI/EC2 can push/pull images).
63
70
64
-
What docker registry can you use? So you have next options:
65
-
1) Docker Hub - most famous. It is free for public images, so literally every opensource project uses it. However it is not free for private images, and you have to pay for it. In this post we are considering you might do development for commercial project, so we will not use it.
66
-
2) GHCR - Registry from Google. Has free plan but on it allows to store only 500MB and allows to do 1GB of traffic per month. Then you pay for every extra GB. Probably small images without cache will fit in this plan, but it is not enough for us.
67
-
3) Self-hosted registry repository. In our software development company we use Harbor. It is powerfull free open-source registry which can be easily installed. It allows to push and pull without limit. Also it has internal life-cycle rules which remove unnecessary images and layers. Main drawbacks of it - it is not so fast to install and configure, plus you have to get domain and another server to run it. So unless you are not software dev company, it is not worth to use it.
68
-
4) Self-hosted registry on EC2 itself using official `registry` image. So since we already have EC2, we can run registry on it directly. The `registry` container is pretty low-weight and easy to setup and it will not consume a lot of extra CPU/RAM on server. Plus images will be stored close to application so pull will be fast.
71
+
What docker registry can you use? Pretty known options:
72
+
1) Docker Hub - most famous. It is free for public images, so literally every opensource project uses it. However it is not free for private images, and you have to pay for it. In this post we are considering you might do development for commercial project with tight budget, so we will not use it.
73
+
2) GHCR - Registry from Google. Has free plan but allows to store only 500MB and allows to transfer 1GB of traffic per month. Then you pay for every extra GB in storage and traffic. Probably small images will fit in this plan, but generally even alpine-based docker images are bigger than 500MB, so it is not a good option.
74
+
3) Self-hosted registry web system. In our software development company, we use Harbor. It is a powerful free open-source registry that can be installed to own server. It allows pushing and pulling without limit. Also, it has internal life-cycle rules that cleanup unnecessary images and layers. The main drawbacks of it are that it is not so fast to install and configure, plus you have to get a domain and another powerfull server to run it. So unless you are a software development company, it is not worth using it.
75
+
4) Self-hosted minimal CNCF Distribution [registry](https://distribution.github.io/distribution/) on EC2 itself. So since we already have EC2, we can run registry on it directly. The `registry` container is pretty light-weight and easy to setup and it will not consume a lot of extra CPU/RAM on server. Plus images will be stored close to application so pull will be fast.
69
76
70
-
In the post we will use last (4th way). Our terraform will DevOps registry it automatically, so you don't have to do anything special. There are some tricks of course, but our terraform will handle them.
77
+
In the post we will use last (4th way). Our terraform will deploy registry automatically, so you don't have to do anything special.
71
78
72
79
### Persisting cache
73
80
74
-
Docker builds without layers cache persisting are possible but are very slow. Most build only couple of layers changed, and having no ability to cache them will cause docker builder to regenerate all layers from scratch. It depends, but might for example cause time docker build time increase from a minute to a 10 minutes or even more.
81
+
Docker builds without layer cache persistence are possible but very slow. Most builds only change a couple of layers, and having no ability to cache them will cause the Docker builder to regenerate all layers from scratch. This can, for example, increase the Docker build time from a minute to ten minutes or even more.
75
82
76
-
When you build on GitHub actions you have to persist cache between builds. Out of the box GitHub actions can't save anything between builds, so you have to use external storage.
83
+
Out of the box, GitHub Actions can't save Docker layers between builds, so you have to use external storage.
77
84
78
-
> Though some CI systems can persist docker build cache, e.g. open-source self-hosted Woodpecker CI des it out of the box. However GitHub actions reasonably can't allow such free storage to anyone
85
+
> Though some CI systems can persist docker build cache, e.g. open-source self-hosted Woodpecker CI allows it out of the box. However GitHub actions which is pretty popular, reasonably can't allow such free storage to anyone
79
86
80
87
So when build-in Docker cache can't be used, there is one alternative - Docker BuildKit external cache.
81
-
So BuildKit allows you to connect external storage. There are several options, but most sweet for us is using Docker registry as cache storage (not only as images storage). However drowback appears here.
82
-
Previously we used docker compose to run our app, it can be used to both build and deploy images, but has [issues with external cache connection](https://github.com/docker/compose/issues/11072#issuecomment-1848974315). While they are not solved we have to use `docker buildx bake` command to build images. It is not so bad, but is another point of configuration which we will cover in this post.
88
+
So BuildKit allows you to connect external storage. There are several options, but most sweet for us is using Docker registry as cache storage (not only as images storage to deliver them to application server).
89
+
90
+
> *BuildKit cache in Compose issue*
91
+
> Previously we used docker compose to build & run our app, it can be used to both build, push and pull images, but has [issues with external cache connection](https://github.com/docker/compose/issues/11072#issuecomment-1848974315). While they are not solved we have to use `docker buildx bake` command to build images. It is not so bad, but is another point of configuration which we will cover in this post.
92
+
93
+
### Registry authorization and traffic encryption
94
+
95
+
Hosting custom CNCF registry, from other hand is a security responsibility.
96
+
97
+
If you don't protect it right, someone will be able to push any image to your registry and then pull it to your EC2 instance. This is a big security issue, so we have to protect our registry.
98
+
99
+
First of all we need to set some authorization to our registry so everyone who will push/pull images will be authorized. Here we have 2 options: HTTP basic auth and Client certificate auth. We will use first one as it is easier to setup. We will generate basic login and password automatically in terraform so no extra actions are needed from you.
100
+
101
+
But this is not enough. Basic auth is not encrypted, so someone can perform MITM attack and get your credentials. So we need to encrypt traffic between CI and registry. We can do it by using TLS certificates. So we will generate self-signed TLS certificates, and attach them to our registry.
102
+
103
+
83
104
84
105
# Practice - deploy setup
85
106
@@ -718,8 +739,7 @@ Now open GitHub actions file and add it to the `env` section:
0 commit comments