feat(lambdas): add batch SSM parameter fetching to reduce API calls#5017
feat(lambdas): add batch SSM parameter fetching to reduce API calls#5017thomasnemer wants to merge 1 commit intogithub-aws-runners:mainfrom
Conversation
Add getParameters() function to aws-ssm-util that fetches multiple SSM parameters in a single API call with automatic chunking (max 10 per call per AWS API limits). Apply batch fetching to: - auth.ts: fetch App ID and Private Key in one call (2 calls → 1) - ConfigLoader.ts: fetch multiple matcher config paths in one call - ami.ts: batch resolve SSM parameter values for AMI lookups Also remove redundant appId SSM fetch in scale-up.ts that was only used for logging.
There was a problem hiding this comment.
Pull request overview
This PR optimizes AWS SSM API usage by implementing batch parameter fetching to reduce the number of API calls. The changes introduce a new getParameters() function that retrieves multiple SSM parameters in a single API call (with automatic chunking for AWS's 10-parameter limit) and applies this optimization across multiple Lambda functions.
Changes:
- Added
getParameters()function to aws-ssm-util for batch SSM parameter fetching with automatic chunking - Replaced sequential SSM calls with batch fetching in authentication, configuration loading, and AMI housekeeping
- Removed redundant appId SSM fetch in scale-up.ts that was only used for logging
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| lambdas/libs/aws-ssm-util/src/index.ts | Implements new getParameters() function for batch SSM parameter retrieval |
| lambdas/functions/webhook/src/ConfigLoader.ts | Replaces sequential parameter fetching with batch call for matcher configs |
| lambdas/functions/webhook/src/ConfigLoader.test.ts | Updates tests to mock getParameters() instead of individual getParameter() calls |
| lambdas/functions/control-plane/src/scale-runners/scale-up.ts | Removes redundant appId SSM fetch used only for logging |
| lambdas/functions/control-plane/src/github/auth.ts | Batches App ID and Private Key fetching into single SSM call |
| lambdas/functions/control-plane/src/github/auth.test.ts | Updates tests to verify batch parameter fetching behavior |
| lambdas/functions/ami-housekeeper/src/ami.ts | Refactors to use batch parameter fetching for AMI SSM lookups |
| lambdas/functions/ami-housekeeper/src/ami.test.ts | Updates tests to use GetParametersCommand instead of GetParameterCommand |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| } else { | ||
| this.configLoadingErrors.push( | ||
| `Failed to load parameter for matcherConfig from path ${path}: Parameter not found`, | ||
| ); |
There was a problem hiding this comment.
The code continues processing remaining parameters even after encountering a missing parameter, which may lead to incomplete configuration. Consider returning early or throwing an error when a required parameter is not found to avoid partially constructed configurations.
| ); | |
| ); | |
| // Stop further processing to avoid partially constructed matcherConfig | |
| return; |
| const paramNames = [process.env.PARAMETER_GITHUB_APP_ID_NAME, process.env.PARAMETER_GITHUB_APP_KEY_BASE64_NAME]; | ||
| const params = await getParameters(paramNames); | ||
|
|
||
| const appIdValue = params.get(process.env.PARAMETER_GITHUB_APP_ID_NAME); | ||
| const privateKeyBase64 = params.get(process.env.PARAMETER_GITHUB_APP_KEY_BASE64_NAME); | ||
|
|
||
| if (!appIdValue) { | ||
| throw new Error(`Parameter ${process.env.PARAMETER_GITHUB_APP_ID_NAME} not found`); | ||
| } | ||
| if (!privateKeyBase64) { | ||
| throw new Error(`Parameter ${process.env.PARAMETER_GITHUB_APP_KEY_BASE64_NAME} not found`); |
There was a problem hiding this comment.
The array may contain undefined values if the environment variables are not set. This will cause the getParameters call to fail or return unexpected results. Add validation to ensure both environment variables exist before creating the array.
| const paramNames = [process.env.PARAMETER_GITHUB_APP_ID_NAME, process.env.PARAMETER_GITHUB_APP_KEY_BASE64_NAME]; | |
| const params = await getParameters(paramNames); | |
| const appIdValue = params.get(process.env.PARAMETER_GITHUB_APP_ID_NAME); | |
| const privateKeyBase64 = params.get(process.env.PARAMETER_GITHUB_APP_KEY_BASE64_NAME); | |
| if (!appIdValue) { | |
| throw new Error(`Parameter ${process.env.PARAMETER_GITHUB_APP_ID_NAME} not found`); | |
| } | |
| if (!privateKeyBase64) { | |
| throw new Error(`Parameter ${process.env.PARAMETER_GITHUB_APP_KEY_BASE64_NAME} not found`); | |
| const appIdParamName = process.env.PARAMETER_GITHUB_APP_ID_NAME; | |
| const appKeyParamName = process.env.PARAMETER_GITHUB_APP_KEY_BASE64_NAME; | |
| if (!appIdParamName) { | |
| throw new Error('Environment variable PARAMETER_GITHUB_APP_ID_NAME is not set'); | |
| } | |
| if (!appKeyParamName) { | |
| throw new Error('Environment variable PARAMETER_GITHUB_APP_KEY_BASE64_NAME is not set'); | |
| } | |
| const paramNames = [appIdParamName, appKeyParamName]; | |
| const params = await getParameters(paramNames); | |
| const appIdValue = params.get(appIdParamName); | |
| const privateKeyBase64 = params.get(appKeyParamName); | |
| if (!appIdValue) { | |
| throw new Error(`Parameter ${appIdParamName} not found`); | |
| } | |
| if (!privateKeyBase64) { | |
| throw new Error(`Parameter ${appKeyParamName} not found`); |
| // Handle wildcard patterns by first discovering matching parameters, then | ||
| // fetching their values | ||
| let wildcardValues: Promise<(string | undefined)[]> = Promise.resolve([]); | ||
| let wildcardValuesPromise: Promise<(string | undefined)[]> = Promise.resolve([]); |
There was a problem hiding this comment.
The variable name 'wildcardValuesPromise' is inconsistent with the original 'wildcardValues'. While the rename improves clarity, it creates inconsistency with the pattern used for 'explicitValuesPromise'.
This PR intends to reduce SSM AWS API calls by doing the following:
Add
getParameters()function to aws-ssm-util that fetches multiple SSM parameters in a single API call with automatic chunking (max 10 per call per AWS API limits).Apply batch fetching to:
Also remove redundant appId SSM fetch in scale-up.ts that was only used for logging.