Increased memory consumption

[wasd171]

I have been running the Helm chart 0.3.0 with the default memory limits (640Mi), however, after upgrade to 0.4.0 (so 9.0.0 of the app itself) I have started experiencing OOMs on the same tasks. Raising the memory limit to 1280Mi fixes it

Edit: it is still failing, even with higher memory limits (2Gi). Sometimes it does not crash due to OOM but cache cannot be restored with the following error: Warning: Failed to restore: Premature close. One thing sticks out in the server logs:

ENOENT: no such file or directory, open '/app/.data/cache/9734614233/parts/3'

[LouisHaftmann]

What storage driver are you using?

[wasd171]

filesystem, here is my full env:

env:
  - name: CACHE_CLEANUP_OLDER_THAN_DAYS
    value: "7"
  - name: STORAGE_DRIVER
    value: filesystem
  - name: STORAGE_FILESYSTEM_PATH
    value: /app/.data/cache
  - name: DB_DRIVER
    value: sqlite
  - name: DB_SQLITE_PATH
    value: /app/.data/github-actions-cache-server.db

[LouisHaftmann]

could you try v9.0.1?

[wasd171]

I still have issues with v9.0.2, either OOMs, or with 2Gi memory the following:

[cache-server] ℹ 🚀 Starting GitHub Actions Cache Server (v9.0.2)
[cache-server] ℹ Migrating database...
[cache-server] ✔ Database migrated
Listening on http://[::]:3000
[cache-server-node-1] ℹ Migrating database...
[cache-server-node-1] ✔ Database migrated

[cache-server-node-1]  ERROR  Response: GET /download/43ae2a69-8f81-4cde-9cc8-0c9d69560764 > 500
 ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/4'



  [cause]: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/4'



[request error] [unhandled] [GET] http://github-actions-cache-server.arc-runners.svc.cluster.local/download/43ae2a69-8f81-4cde-9cc8-0c9d69560764
 [H3Error: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/4'] {
  cause: [Error: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/4'] {
    errno: -2,
    code: 'ENOENT',
    syscall: 'open',
    path: '.data/storage/filesystem/388167591/parts/4'
  },
  statusCode: 500,
  fatal: false,
  unhandled: true,
  statusMessage: undefined,
  data: undefined
}
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
    at ServerResponse.setHeader (node:_http_outgoing:642:11)
    at setResponseHeaders (file:///app/server/index.mjs:1107:20)
    at defaultNitroErrorHandler (file:///app/server/index.mjs:4206:5)
    at errorHandler (file:///app/server/index.mjs:4269:13)
    at Object.onError (file:///app/server/index.mjs:5113:14)
    at Server.toNodeHandle (file:///app/server/index.mjs:1915:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5) {
  code: 'ERR_HTTP_HEADERS_SENT'
}

[cache-server-node-1]  ERROR  Response: GET /download/43ae2a69-8f81-4cde-9cc8-0c9d69560764 > 500
 ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/6'



  [cause]: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/6'



[request error] [unhandled] [GET] http://github-actions-cache-server.arc-runners.svc.cluster.local/download/43ae2a69-8f81-4cde-9cc8-0c9d69560764
 [H3Error: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/6'] {
  cause: [Error: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/6'] {
    errno: -2,
    code: 'ENOENT',
    syscall: 'open',
    path: '.data/storage/filesystem/388167591/parts/6'
  },
  statusCode: 500,
  fatal: false,
  unhandled: true,
  statusMessage: undefined,
  data: undefined
}
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
    at ServerResponse.setHeader (node:_http_outgoing:642:11)
    at setResponseHeaders (file:///app/server/index.mjs:1107:20)
    at defaultNitroErrorHandler (file:///app/server/index.mjs:4206:5)
    at errorHandler (file:///app/server/index.mjs:4269:13)
    at Object.onError (file:///app/server/index.mjs:5113:14)
    at Server.toNodeHandle (file:///app/server/index.mjs:1915:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5) {
  code: 'ERR_HTTP_HEADERS_SENT'
}

[cache-server-node-1]  ERROR  Response: GET /download/43ae2a69-8f81-4cde-9cc8-0c9d69560764 > 500
 ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/2'



  [cause]: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/2'



[request error] [unhandled] [GET] http://github-actions-cache-server.arc-runners.svc.cluster.local/download/43ae2a69-8f81-4cde-9cc8-0c9d69560764
 [H3Error: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/2'] {
  cause: [Error: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/2'] {
    errno: -2,
    code: 'ENOENT',
    syscall: 'open',
    path: '.data/storage/filesystem/388167591/parts/2'
  },
  statusCode: 500,
  fatal: false,
  unhandled: true,
  statusMessage: undefined,
  data: undefined
}
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
    at ServerResponse.setHeader (node:_http_outgoing:642:11)
    at setResponseHeaders (file:///app/server/index.mjs:1107:20)
    at defaultNitroErrorHandler (file:///app/server/index.mjs:4206:5)
    at errorHandler (file:///app/server/index.mjs:4269:13)
    at Object.onError (file:///app/server/index.mjs:5113:14)
    at Server.toNodeHandle (file:///app/server/index.mjs:1915:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5) {
  code: 'ERR_HTTP_HEADERS_SENT'
}

[cache-server-node-1]  ERROR  Response: GET /download/43ae2a69-8f81-4cde-9cc8-0c9d69560764 > 500
 ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/5'



  [cause]: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/5'



[request error] [unhandled] [GET] http://github-actions-cache-server.arc-runners.svc.cluster.local/download/43ae2a69-8f81-4cde-9cc8-0c9d69560764
 [H3Error: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/5'] {
  cause: [Error: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/5'] {
    errno: -2,
    code: 'ENOENT',
    syscall: 'open',
    path: '.data/storage/filesystem/388167591/parts/5'
  },
  statusCode: 500,
  fatal: false,
  unhandled: true,
  statusMessage: undefined,
  data: undefined
}
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
    at ServerResponse.setHeader (node:_http_outgoing:642:11)
    at setResponseHeaders (file:///app/server/index.mjs:1107:20)
    at defaultNitroErrorHandler (file:///app/server/index.mjs:4206:5)
    at errorHandler (file:///app/server/index.mjs:4269:13)
    at Object.onError (file:///app/server/index.mjs:5113:14)
    at Server.toNodeHandle (file:///app/server/index.mjs:1915:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5) {
  code: 'ERR_HTTP_HEADERS_SENT'
}

[cache-server-node-1]  ERROR  Response: GET /download/43ae2a69-8f81-4cde-9cc8-0c9d69560764 > 500
 ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/2'



  [cause]: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/2'



[request error] [unhandled] [GET] http://github-actions-cache-server.arc-runners.svc.cluster.local/download/43ae2a69-8f81-4cde-9cc8-0c9d69560764
 [H3Error: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/2'] {
  cause: [Error: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/2'] {
    errno: -2,
    code: 'ENOENT',
    syscall: 'open',
    path: '.data/storage/filesystem/388167591/parts/2'
  },
  statusCode: 500,
  fatal: false,
  unhandled: true,
  statusMessage: undefined,
  data: undefined
}
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
    at ServerResponse.setHeader (node:_http_outgoing:642:11)
    at setResponseHeaders (file:///app/server/index.mjs:1107:20)
    at defaultNitroErrorHandler (file:///app/server/index.mjs:4206:5)
    at errorHandler (file:///app/server/index.mjs:4269:13)
    at Object.onError (file:///app/server/index.mjs:5113:14)
    at Server.toNodeHandle (file:///app/server/index.mjs:1915:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5) {
  code: 'ERR_HTTP_HEADERS_SENT'
}

[cache-server-node-1]  ERROR  Response: GET /download/43ae2a69-8f81-4cde-9cc8-0c9d69560764 > 500
 ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/2'



  [cause]: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/2'



[request error] [unhandled] [GET] http://github-actions-cache-server.arc-runners.svc.cluster.local/download/43ae2a69-8f81-4cde-9cc8-0c9d69560764
 [H3Error: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/2'] {
  cause: [Error: ENOENT: no such file or directory, open '.data/storage/filesystem/388167591/parts/2'] {
    errno: -2,
    code: 'ENOENT',
    syscall: 'open',
    path: '.data/storage/filesystem/388167591/parts/2'
  },
  statusCode: 500,
  fatal: false,
  unhandled: true,
  statusMessage: undefined,
  data: undefined
}
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
    at ServerResponse.setHeader (node:_http_outgoing:642:11)
    at setResponseHeaders (file:///app/server/index.mjs:1107:20)
    at defaultNitroErrorHandler (file:///app/server/index.mjs:4206:5)
    at errorHandler (file:///app/server/index.mjs:4269:13)
    at Object.onError (file:///app/server/index.mjs:5113:14)
    at Server.toNodeHandle (file:///app/server/index.mjs:1915:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5) {
  code: 'ERR_HTTP_HEADERS_SENT'
}

[LouisHaftmann]

Does it work when clearing db and storage and setting a really high memory limit?

[wasd171]

No OOMs, but I am still getting the same logs for memory limit of 8Gi and NODE_OPTIONS="--max-old-space-size=6144"

[LouisHaftmann]

How large are the files you are caching? Is the memory usage spike proportional to the cache file size or does it also happen with very small files?

[wasd171]

This pipeline uses several cache entries around 400MB and 1 small one (4MB). I feel that the increased memory has helped a bit since I only see 1 error like that in the logs (and Warning: Failed to restore: Premature close on GitHub). I have 9 tasks that run in parallel and attempt to retrieve this 400MB cache more or less at the same time. So far I have not seen issues with restoring the 4MB cache entry

[LouisHaftmann]

Please try v9.0.4 🙏

[wasd171]

Unfortunately I am still getting the following issues on the v9.1.0:

1. OOMs (with the default memory limit of 640Mi)
2. Missing cache entries on disk (even though all "Cache saved" steps have finished correctly)

[cache-server-node-1]  ERROR  Response: GET /download/f73ba2b3-7e33-4445-a352-1003b1d78ff5 > 500
ENOENT: no such file or directory, open '/app/.data/cache/5896227196/parts/4'

[dawidmalina]

I started testing the new version today (just in a playground environment) and encountered the same issue. Even with these resource settings:

resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    cpu: 2
    memory: 2Gi

I quickly hit the HPA max:

autoscaling:
  minReplicas: 2
  maxReplicas: 4

A quick fix was to increase the maxReplicas:

autoscaling:
  minReplicas: 2
  maxReplicas: 10

Now I'm running at 10 replicas!

For comparison, my old production setup with v8:

resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    cpu: 1
    memory: 1Gi

With that, I was running up to 3 replicas.

[LouisHaftmann]

Does this only happen on first upload (no existing cache entry) and download or also on every subsequent download? The uploaded parts are combined on first download. I want to know if the combining or just the regular download is the problem.

[dawidmalina]

@LouisHaftmann - I create a single cache file with npm modules (about 300 MB) that is uploaded in the first job and then reused by several jobs in the workflow. Since this is just a playground, there’s no situation with multiple uploads - just one upload in the first job, and all following jobs restore and reuse the same cache.

[LouisHaftmann]

I don't know why v9 would use more memory than v8. v8 literally read whole uploaded parts into memory as buffers. v9 only uses streams everywhere. Please try latest :dev tag. I added some manual gc since natural nodejs gc is not fast enough under heavy load.

[wasd171]

I think :dev made things worse, for me it OOMed on the first cache upload, did not even have a chance to progress to the download part