Skip to content

Commit 4c623a9

Browse files
JonasAlfredssonJonasAlfredsson
committed
Update README
1 parent 5efcc19 commit 4c623a9

File tree

1 file changed

+156
-0
lines changed

1 file changed

+156
-0
lines changed

‎README.md‎

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,158 @@
11
# docker-on-tmpfs
22
GitHub Action which mounts a tmpfs volume on `/var/lib/docker`.
3+
4+
This action was created to solve a [very specific problem][1] that occurs when
5+
building the latest pip `cryptography` package inside a QEMU emulated
6+
`linux/arm/v7` environment (e.g. Rasberry Pi) on a `linux/amd64` computer.
7+
8+
The compilation of the package fails because some weirdness inside a low level
9+
library when it tries to read the filesytem while running inside a 32-bit
10+
environment that is emulated by QEMU running on a 64-bit host. Apparently a
11+
workaround for this is to swap the filesystem to a tmpfs one in order to not
12+
trigger this bug, so we make the entire `/var/lib/docker` folder like that.
13+
14+
Here are links to the biggest threads discussing this issue further:
15+
16+
- https://github.com/docker/buildx/issues/395
17+
- https://github.com/rust-lang/cargo/issues/8719
18+
- https://gitlab.com/qemu-project/qemu/-/issues/263
19+
20+
21+
### Acknowledgments and Thanks
22+
23+
This action has been inspired by the workarounds posted in the threads mentioned
24+
above, but there are three solutions I would like to give extra credit to:
25+
26+
- [`@pierotofy`][4]: For showing how to configure the swap space.
27+
- [`@easimon`][5]: For providing details about GitHub runners.
28+
- [`@nijel`][6]: For showing how a `tmpfs` can be used as a workaround.
29+
30+
31+
## Usage
32+
33+
> :warning: Please read the [More Information](#more-information) section to
34+
understand the physical limitations of the GitHub runners before
35+
changing the values.
36+
37+
```yaml
38+
- name: Run Docker on tmpfs
39+
uses: JonasAlfredsson/docker-on-tmpfs@v1.0.0
40+
with:
41+
tmpfs_size: 5
42+
swap_size: 4
43+
swap_location: '/mnt/swapfile'
44+
```
45+
46+
47+
## More Information
48+
49+
The official [specifications][2] of GitHub runner machines are the following:
50+
51+
- 2-core CPU
52+
- 7 GB of RAM memory
53+
- 14 GB of SSD disk space
54+
55+
However, this does not paint the whole picture, and a more detailed
56+
representation of a newly starter runner would be this:
57+
58+
#### CPU
59+
```
60+
vendor_id : GenuineIntel
61+
cpu family : 6
62+
model : 85
63+
model name : Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
64+
cpu MHz : 2095.197
65+
cache size : 36608 KB
66+
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear
67+
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit
68+
```
69+
70+
#### Memory
71+
72+
```
73+
total used free shared buff/cache available
74+
Mem: 6.8Gi 484Mi 5.3Gi 9.0Mi 1.0Gi 6.0Gi
75+
Swap: 4.0Gi 0B 4.0Gi
76+
```
77+
78+
```
79+
NAME TYPE SIZE USED PRIO
80+
/mnt/swapfile file 4G 0B -2
81+
```
82+
83+
#### Storage
84+
85+
```
86+
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
87+
sda 8:0 0 14G 0 disk
88+
└─sda1 8:1 0 14G 0 part /mnt
89+
sdb 8:16 0 86G 0 disk
90+
├─sdb1 8:17 0 85.9G 0 part /
91+
├─sdb14 8:30 0 4M 0 part
92+
└─sdb15 8:31 0 106M 0 part /boot/efi
93+
```
94+
95+
```
96+
Filesystem Size Used Avail Use% Mounted on
97+
/dev/root 84G 53G 31G 63% /
98+
devtmpfs 3.4G 0 3.4G 0% /dev
99+
tmpfs 3.4G 4.0K 3.4G 1% /dev/shm
100+
tmpfs 695M 1.1M 694M 1% /run
101+
tmpfs 5.0M 0 5.0M 0% /run/lock
102+
tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
103+
/dev/sdb15 105M 5.2M 100M 5% /boot/efi
104+
/dev/sda1 14G 4.1G 9.0G 32% /mnt
105+
```
106+
107+
#### Default Workspace
108+
109+
There is one variable that is extra interesting of those listed [here][3], and
110+
that is `GITHUB_WORKSPACE` which is explained like this:
111+
112+
> Actions and shell commands execute in this directory. An action can modify
113+
the contents of this directory, which subsequent actions can access.
114+
115+
This path defaults to
116+
117+
/home/runner/work/<repo-name>/<repo-name>
118+
119+
so in this repository's case it would be
120+
121+
/home/runner/work/docker-on-tmpfs/docker-on-tmpfs
122+
123+
This is also the path you will get if you run just `echo ${PWD}`.
124+
125+
### Limitations
126+
127+
With the above information we can see that, by default, we are not using the
128+
advertised 14G available on `/mnt`, but instead the 31G on `/`. Furthermore,
129+
of those 14G on `/mnt` 4 are used for the swap file, so it is really only 10G
130+
available there. At startup we are also using ~0.5G RAM, which means that we
131+
have about 6G to play with for our `tmpfs` volume.
132+
133+
What is important to understand is that a `tmpfs` volume writes its data to RAM,
134+
so if you consume more than the available the system will start moving data to
135+
the swap file on the disk. Reading data from disk is **brutally** slow compared
136+
to accessing it from RAM, so when this happens the system might become unusable
137+
slow. However, if you remove the swap file and fill up the RAM the system will
138+
probably crash so it is important to keep if we are to do what this action does.
139+
Linux is also pretty clever, so data more frequently accessed is less likely to
140+
be pushed to swap.
141+
142+
I would suggest you try to keep `tmpfs_size` as small as possible for your
143+
task, in order to minimize the risk of starving the other system processes.
144+
A good thing to do is also to keep the `swap_size` the same size as your
145+
`tmpfs`, and this is even more important in case you want to make the volume
146+
larger than the currently available RAM.
147+
148+
149+
150+
151+
152+
153+
[1]: https://github.com/JonasAlfredsson/docker-nginx-certbot/issues/30
154+
[2]: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners
155+
[3]: https://docs.github.com/en/actions/learn-github-actions/environment-variables
156+
[4]: https://github.com/pierotofy/set-swap-space
157+
[5]: https://github.com/easimon/maximize-build-space
158+
[6]: https://github.com/WeblateOrg/docker/pull/1274/commits/c40a9949596cee31d6a56597e5e3480e0b090d25

0 commit comments

Comments
 (0)