Staging Hotfix Release

3AM Quick Guide

• Use this when you need one merged hotfix in staging outside the normal cut-off flow.
• Cherry-pick the merged hotfix into release and confirm only the intended hotfix landed there.
• Open the orchestration pipeline and run only release <service name>: https://gitlab.com/evenergi/develop/-/pipelines/new
• Do not run cut-off. That job rebuilds release from the default branch, which is not the right path for a selective hotfix.
• Watch the staging deploy through completion, run the hotfix-specific checks, and record what was validated.
• For governed delivery work, add the Validated in Staging label after the checks pass.

Purpose And When To Use It

This runbook explains how to promote a merged hotfix into the release branch and trigger the staging release flow for a specific service.

Use this workflow when an urgent fix must be deployed to staging outside the normal cut-off cadence, or when the current staging candidate needs an extra fix before production validation completes.

Do not use this workflow for the normal weekly cut-off, for unrelated feature promotion, or when the target repository already documents a different service-specific staging process.

Prerequisites And Permissions

Before using this workflow:

• the hotfix merge request must already be merged and approved for staging
• you must know which service or services need the hotfix
• you must have permission to cherry-pick into the release branch, or push a corrected release branch if the GitLab UI path is unavailable
• you must have permission to create and run the orchestration pipeline for the target service
• you must know where the target service exposes staging deployment status, logs, and health checks
• you must have the test or smoke-check steps needed to validate the fix in staging

In the current BetterFleet setup, operators typically create the orchestration pipeline from:

• https://gitlab.com/evenergi/develop/-/pipelines/new

The service-specific manual job usually follows the pattern release <service name>, for example 01 release vemo-core.

Normal Procedure

1. Confirm the hotfix is ready for staging promotion. Check that the merge request is merged, that the change is the right scope for staging, and that any downstream services affected by the fix are known.
2. Cherry-pick the merged hotfix into release. Use the GitLab merge request cherry-pick action to apply the merged change to the release branch.
3. Verify the release branch now contains the intended hotfix. Confirm the cherry-pick succeeded, the target branch is release, and no unexpected extra commits were introduced as part of the promotion.
4. Create a new orchestration pipeline. Open the shared orchestration pipeline page and start a fresh pipeline for the release flow.
5. Run the service release job. Select the manual job for the target service, usually named release <service name>.
6. Do not run the cut-off job. The cut-off job is for rebuilding release from the default branch. It is not the correct action for promoting a single staging hotfix.
7. Watch the target service deployment through to completion. Follow the downstream pipeline, environment page, and service health checks until the staging deployment finishes.
8. Validate the fix in staging. Run the relevant smoke tests, acceptance checks, or focused manual checks for the hotfix scope. For governed delivery work, add the Validated in Staging merge-request label once the validation passes.

Reference Screenshots

These screenshots use one example service and repository. Your target project name and release <service name> job label may differ.

Steps 2 And 3: Cherry-Pick The Merged Hotfix Into release

What To Check	Screenshot
Start from the merged merge request and use the Cherry-pick action.
In the cherry-pick dialog, target the release branch before confirming the promotion.

Steps 4 Through 6: Run The Service Release Job

Use the orchestration pipeline to run the release <service name> job for the affected service. The key safety check is to choose the service release job and not the cut-off job.

Steps 7 And 8: Confirm The Staging Deploy And Validate The Result

Use Operate -> Environments and the target service's own health or smoke checks to confirm staging now points at the intended hotfix deployment.

Decision Points And Exceptions

• If the merge request is not yet merged, or the fix is still changing, stop and complete the normal development and review path first.
• If the release branch needs a full resync from the default branch rather than a selective hotfix promotion, use the repository's normal cut-off process instead of this runbook.
• If the cherry-pick conflicts, resolve the conflict on a local release branch, verify the resulting tree, and push only after confirming the final contents are correct.
• If more than one service needs the hotfix, repeat the release job and validation steps for each affected service and record exactly which services were promoted.
• If the service uses different job names, approvals, or downstream pipeline entry points, follow the target repository's service-specific release docs.
• If staging validation fails, do not proceed toward production. Either revert the hotfix from release or replace it with a corrected hotfix and rerun the staging deploy.

Validation And Evidence

Treat the staging promotion as complete only when you can point to evidence for each of these checks:

• the merged hotfix is present on release
• the orchestration pipeline was created for the intended promotion
• the correct service release job was run, not cut-off
• the target service's staging deployment completed successfully
• the staging environment reflects the expected commit or build for the hotfix
• the hotfix-specific smoke tests or acceptance checks passed
• the merge request, deployment note, or handover note records what was validated and for which service
• for governed delivery work, the merge request carries the Validated in Staging label after validation passes

Rollback And Recovery

If the hotfix has not yet been deployed, remove or revert the cherry-picked commit from release and stop the promotion.

If the hotfix was deployed to staging and needs to be backed out:

• revert the cherry-picked hotfix from release
• push the corrected release branch
• rerun the relevant release <service name> job
• revalidate staging before using that environment for further checks

If the release branch state is unclear or includes unintended drift, stop and restore it using the target repository's release-branch recovery process before attempting another deployment.

Links To Service-Specific Details

• Shared CI and release context: CI and Release Integration
• Governed staging validation expectations: Product Engineering Delivery
• BetterFleet service lookup: Service Matrix
• BetterFleet Manage service docs: Manage Services
• BetterFleet Plan service docs: Plan Services
• Target repository details: the target repository's .gitlab-ci.yml, release jobs, environment page, and service-specific operational docs