Skip to content

Gitlab CI Pipeline Failures

Jump to bottom
vk0125 edited this page Mar 9, 2023 · 1 revision

This document describes a few of the encountered failures and solutions applied.

Pipeline Failures

  1. Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

DESCRIPTION: Suppose developers are working on a feature branch and have created a merge request to the upstream branch i.e. dev. When the feature branch is updated again, it will trigger the pipeline twice in parallel: One for commit on the feature branch and the second is a new auto-triggered merge result pipeline. Because of this, a user may see failure in any stage(build/deploy) job as shown below:

image

There is a possibility that the pipeline build falls into the situation where helm deployment remains under a "pending" state, and the pipeline will fail consistently for further consecutive triggers.

DEBUGGING: Try to check the history of the previous deployment

WHY: This provides with information mostly like the original installation was never completed successfully and is pending state something like STATUS: pending-upgrade state.

helm history <release> --namespace <namespace>

Command execution result: [NOTE: Taking cortex component as an example]

$ helm history cortex -n cortex
REVISION        UPDATED                         STATUS          CHART           APP VERSION     DESCRIPTION      
68              Wed Dec 21 15:09:16 2022        superseded      cortex-1.6.0    v1.13.1         Upgrade complete 
69              Fri Dec 23 11:29:43 2022        superseded      cortex-1.6.0    v1.13.1         Upgrade complete 
70              Fri Dec 23 11:39:35 2022        superseded      cortex-1.6.0    v1.13.1         Upgrade complete 
71              Fri Dec 23 12:05:53 2022        deployed        cortex-1.6.0    v1.13.1         Upgrade complete 
72              Fri Dec 23 12:48:27 2022        pending-upgrade cortex-1.6.0    v1.13.1         Preparing upgrade

SOLUTION: If any of the releases shown pending, to escape from this state, use the rollback command: helm rollback <release> <revision> --namespace <namespace> NOTE: revision is optional, but you should try to provide it.

Command execution result:

$ helm rollback cortex 71 -n cortex
Rollback was a success! Happy Helming!

$ helm history cortex -n cortex     
REVISION        UPDATED                         STATUS          CHART           APP VERSION     DESCRIPTION      
68              Wed Dec 21 15:09:16 2022        superseded      cortex-1.6.0    v1.13.1         Upgrade complete 
69              Fri Dec 23 11:29:43 2022        superseded      cortex-1.6.0    v1.13.1         Upgrade complete 
70              Fri Dec 23 11:39:35 2022        superseded      cortex-1.6.0    v1.13.1         Upgrade complete 
71              Fri Dec 23 12:05:53 2022        superseded      cortex-1.6.0    v1.13.1         Upgrade complete 
72              Fri Dec 23 12:48:27 2022        pending-upgrade cortex-1.6.0    v1.13.1         Preparing upgrade
73              Sat Dec 24 11:34:30 2022        deployed        cortex-1.6.0    v1.13.1         Rollback to 71

Reference: Fix helm pending-upgrade failure

Clone this wiki locally