Staging Hotfix Release

3AM Quick Guide

• Use this when you need one merged hotfix in staging outside the normal cut-off flow.
• Cherry-pick the merged hotfix into release and confirm only the intended hotfix landed there.
• Open the orchestration pipeline and set action=deploy-release-branch-to-staging, project=<target repository>, and confirm=true: https://gitlab.com/evenergi/develop/-/pipelines/new
• Do not choose bring-all-code-to-release-branch. That action rebuilds release from the default branch, which is not the right path for a selective hotfix.
• Watch the staging deploy through completion, run the hotfix-specific checks, and record what was validated.
• For governed delivery work, add the Validated in Staging label after the checks pass.

Purpose And When To Use It

This runbook explains how to promote a merged hotfix into the release branch and trigger the staging release flow for a specific repository.

Use this workflow when an urgent fix must be deployed to staging outside the normal cut-off cadence, or when the current staging candidate needs an extra fix before production validation completes.

Do not use this workflow for the normal weekly cut-off, for unrelated feature promotion, or when the target repository already documents a different service-specific staging process.

Prerequisites And Permissions

Before using this workflow:

• the hotfix merge request must already be merged and approved for staging
• you must know which repository or service needs the hotfix
• you must have permission to cherry-pick into the release branch, or push a corrected release branch if the GitLab UI path is unavailable
• you must have permission to create and run the orchestration pipeline for the target repository
• you must know where the target service exposes staging deployment status, logs, and health checks
• you must have the test or smoke-check steps needed to validate the fix in staging

In the current BetterFleet setup, operators typically create the orchestration pipeline from:

• https://gitlab.com/evenergi/develop/-/pipelines/new

For a selective staging hotfix, use these guided inputs:

• action=deploy-release-branch-to-staging
• project=<target repository>
• confirm=true

Normal Procedure

1. Confirm the hotfix is ready for staging promotion. Check that the merge request is merged, that the change is the right scope for staging, and that any downstream services affected by the fix are known.
2. Cherry-pick the merged hotfix into release. Use the GitLab merge request cherry-pick action to apply the merged change to the release branch.
3. Verify the release branch now contains the intended hotfix. Confirm the cherry-pick succeeded, the target branch is release, and no unexpected extra commits were introduced as part of the promotion.
4. Create a new orchestration pipeline. Open the shared orchestration pipeline page and start a fresh pipeline with action=deploy-release-branch-to-staging, project=<target repository>, and confirm=true.
5. Do not choose the cut-off action. bring-all-code-to-release-branch is the normal cut-off path for rebuilding release from the default branch. It is not the correct action for promoting a single staging hotfix.
6. Watch the target service deployment through to completion. Follow the downstream pipeline, environment page, and service health checks until the staging deployment finishes.
7. Validate the fix in staging. Run the relevant smoke tests, acceptance checks, or focused manual checks for the hotfix scope. For governed delivery work, add the Validated in Staging merge-request label once the validation passes.

Reference Screenshots

These screenshots use one example service and repository. The Run new pipeline image now reflects the current guided action / project / confirm entrypoint.

Steps 2 And 3: Cherry-Pick The Merged Hotfix Into release

What To Check	Screenshot
Start from the merged merge request and use the Cherry-pick action.
In the cherry-pick dialog, target the release branch before confirming the promotion.

Steps 4 And 5: Start The Guided Release Invocation

Use the orchestration pipeline with action=deploy-release-branch-to-staging, project=<target repository>, and confirm=true. The key safety check is to avoid bring-all-code-to-release-branch for a selective hotfix. The screenshot below shows the current Run new pipeline screen with the guided inputs ready for submission.

Steps 6 And 7: Confirm The Staging Deploy And Validate The Result

Use Operate -> Environments and the target service's own health or smoke checks to confirm staging now points at the intended hotfix deployment.

Decision Points And Exceptions

• If the merge request is not yet merged, or the fix is still changing, stop and complete the normal development and review path first.
• If the release branch needs a full resync from the default branch rather than a selective hotfix promotion, use the repository's normal cut-off process instead of this runbook.
• If the cherry-pick conflicts, resolve the conflict on a local release branch, verify the resulting tree, and push only after confirming the final contents are correct.
• If more than one repository needs the hotfix, repeat the guided release invocation and validation steps for each affected repository and record exactly which targets were promoted.
• If the service uses different job names, approvals, or downstream pipeline entry points, follow the target repository's service-specific release docs.
• If staging validation fails, do not proceed toward production. Either revert the hotfix from release or replace it with a corrected hotfix and rerun the staging deploy.

Validation And Evidence

Treat the staging promotion as complete only when you can point to evidence for each of these checks:

• the merged hotfix is present on release
• the orchestration pipeline used action=deploy-release-branch-to-staging, the intended project, and confirm=true
• bring-all-code-to-release-branch was not selected for the hotfix
• the target service's staging deployment completed successfully
• the staging environment reflects the expected commit or build for the hotfix
• the hotfix-specific smoke tests or acceptance checks passed
• the merge request, deployment note, or handover note records what was validated and for which service
• for governed delivery work, the merge request carries the Validated in Staging label after validation passes

Rollback And Recovery

If the hotfix has not yet been deployed, remove or revert the cherry-picked commit from release and stop the promotion.

If the hotfix was deployed to staging and needs to be backed out:

• revert the cherry-picked hotfix from release
• push the corrected release branch
• rerun the orchestration pipeline with action=deploy-release-branch-to-staging, the same target project, and confirm=true
• revalidate staging before using that environment for further checks

If the release branch state is unclear or includes unintended drift, stop and restore it using the target repository's release-branch recovery process before attempting another deployment.

Links To Service-Specific Details

• Shared CI and release context: CI and Release Integration
• Governed staging validation expectations: Product Engineering Delivery
• BetterFleet service lookup: Service Matrix
• BetterFleet Manage service docs: Manage Services
• BetterFleet Plan service docs: Plan Services
• Target repository details: the target repository's .gitlab-ci.yml, release jobs, environment page, and service-specific operational docs