SOP: Production Release & Rollback

Purpose & Scope

1  Purpose

This document provides the official step-by-step procedure for deploying new versions of software to the production environment. It also details the mandatory procedure for performing a rollback in the event of a failure.

2 Scope

This SOP applies to all scheduled production releases and emergency hotfixes for applications managed under the Memorres CI/CD process.

3 Key Roles

  • Release Manager: The individual accountable for the go/no-go decision.
  • Executing Engineer: The DevOps engineer responsible for running the deployment pipeline.
  • Validation Team: The developers and QA engineers responsible for verifying the release.

Pre-Release Checklist (T-1 Hour)

This checklist MUST be completed and verified by the Release Manager before the deployment window begins.

  • Approval Confirmed: Final deployment approval has been given by the project stakeholders.
  • Staging Sign-off: The build has been successfully deployed and passed all QA checks in the staging environment.
  • Rollback Plan Verified: The Executing Engineer has confirmed the rollback procedure and identified the last known stable version to revert to.
  • Monitoring Ready: All monitoring dashboards and alerting systems (e.g., CloudWatch) are confirmed to be active and ready.
  • Stakeholder Communication: An announcement has been sent out in the relevant channels (e.g., #releases Slack channel) notifying stakeholders of the upcoming deployment window.

StandRelease Execution Procedure

This procedure is to be followed step-by-step by the Executing Engineer.

  1. Announce Start: Post a message in the #releases channel announcing that the deployment window has begun.
  2. (If Applicable) Enable Maintenance Mode: Place the application into maintenance mode to prevent user disruption.
  3. Trigger Deployment: Manually trigger the production deployment job in the Jenkins UI for the correct service and branch/commit.
  4. Monitor Pipeline: Actively monitor the build logs in Jenkins until completion.
    • IF PIPELINE FAILS: Immediately stop this procedure and proceed to Rollback Procedure section.
  5. Proceed to Validation: Once the pipeline completes successfully, notify the Validation Team to begin their checks.

Post-Release Validation Checklist

This checklist is to be completed by the Validation Team within 15 minutes of a successful pipeline run.

  • System Health Check (DevOps):
    • For backend services, confirm the new ECS tasks are running and healthy.
    • For frontend services, confirm the new assets are present in the S3 bucket.
    • Check for any immediate critical errors in the application logs.
  • Smoke Test (QA/Developers):
    • Perform a predefined set of manual smoke tests on the application’s critical features (e.g., user login, key API endpoints, main UI elements).
  • Performance Metrics Check (All):
    • Review monitoring dashboards for any negative anomalies in CPU, memory usage, error rates, or response times.
  • Go/No-Go Decision:
    • IF ANY CHECK FAILS: The Release Manager MUST make the decision to roll back. Proceed immediately to Section 5: Rollback Procedure.
    • IF ALL CHECKS PASS: The Release Manager declares the deployment a success. Proceed to finalize the release.

Rollback Procedure (In Case of Failure)

This procedure is the immediate and mandatory response to a failed deployment or validation.

  1. Announce Rollback: The Release Manager posts a message in the #releases channel: “ROLLBACK INITIATED for the service.”
  2. Execute Rollback: The Executing Engineer will redeploy the last known stable version.
    • For Backend (ECS): In AWS management console, update the service with the old stable task-definition version.
    • For Frontend (S3): Re-run the deployment job for the previous stable build, which will copy the old assets back to the S3 bucket.
  3. Verify Rollback: The Validation Team MUST perform the same checks from the Post-Release Validation Checklist to confirm the application has returned to its stable state.
  4. Announce Completion: Post a message announcing the rollback is complete and the application is stable.

Finalizing the Release (On Success)

  • (If Applicable) Disable Maintenance Mode: Remove the application from maintenance mode.
  • Announce Success: Post a message in the #releases channel that the deployment of service was successful and the release window is now closed.
  • Close Tickets: Update and close the relevant JIRA tickets for the release.