Purpose & Scope
1 Purpose
This document provides the official step-by-step procedure for deploying new versions of software to the production environment. It also details the mandatory procedure for performing a rollback in the event of a failure.
2 Scope
This SOP applies to all scheduled production releases and emergency hotfixes for applications managed under the Memorres CI/CD process.
3 Key Roles
- Release Manager: The individual accountable for the go/no-go decision.
- Executing Engineer: The DevOps engineer responsible for running the deployment pipeline.
- Validation Team: The developers and QA engineers responsible for verifying the release.
Pre-Release Checklist (T-1 Hour)
This checklist MUST be completed and verified by the Release Manager before the deployment window begins.
- Approval Confirmed: Final deployment approval has been given by the project stakeholders.
- Staging Sign-off: The build has been successfully deployed and passed all QA checks in the staging environment.
- Rollback Plan Verified: The Executing Engineer has confirmed the rollback procedure and identified the last known stable version to revert to.
- Monitoring Ready: All monitoring dashboards and alerting systems (e.g., CloudWatch) are confirmed to be active and ready.
- Stakeholder Communication: An announcement has been sent out in the relevant channels (e.g., #releases Slack channel) notifying stakeholders of the upcoming deployment window.
StandRelease Execution Procedure
This procedure is to be followed step-by-step by the Executing Engineer.
- Announce Start: Post a message in the #releases channel announcing that the deployment window has begun.
- (If Applicable) Enable Maintenance Mode: Place the application into maintenance mode to prevent user disruption.
- Trigger Deployment: Manually trigger the production deployment job in the Jenkins UI for the correct service and branch/commit.
- Monitor Pipeline: Actively monitor the build logs in Jenkins until completion.
- IF PIPELINE FAILS: Immediately stop this procedure and proceed to Rollback Procedure section.
- Proceed to Validation: Once the pipeline completes successfully, notify the Validation Team to begin their checks.
Post-Release Validation Checklist
This checklist is to be completed by the Validation Team within 15 minutes of a successful pipeline run.
- System Health Check (DevOps):
- For backend services, confirm the new ECS tasks are running and healthy.
- For frontend services, confirm the new assets are present in the S3 bucket.
- Check for any immediate critical errors in the application logs.
- Smoke Test (QA/Developers):
- Perform a predefined set of manual smoke tests on the application’s critical features (e.g., user login, key API endpoints, main UI elements).
- Performance Metrics Check (All):
- Review monitoring dashboards for any negative anomalies in CPU, memory usage, error rates, or response times.
- Go/No-Go Decision:
- IF ANY CHECK FAILS: The Release Manager MUST make the decision to roll back. Proceed immediately to Section 5: Rollback Procedure.
- IF ALL CHECKS PASS: The Release Manager declares the deployment a success. Proceed to finalize the release.
Rollback Procedure (In Case of Failure)
This procedure is the immediate and mandatory response to a failed deployment or validation.
- Announce Rollback: The Release Manager posts a message in the #releases channel: “ROLLBACK INITIATED for the service.”
- Execute Rollback: The Executing Engineer will redeploy the last known stable version.
- For Backend (ECS): In AWS management console, update the service with the old stable task-definition version.
- For Frontend (S3): Re-run the deployment job for the previous stable build, which will copy the old assets back to the S3 bucket.
- Verify Rollback: The Validation Team MUST perform the same checks from the Post-Release Validation Checklist to confirm the application has returned to its stable state.
- Announce Completion: Post a message announcing the rollback is complete and the application is stable.
Finalizing the Release (On Success)
- (If Applicable) Disable Maintenance Mode: Remove the application from maintenance mode.
- Announce Success: Post a message in the #releases channel that the deployment of service was successful and the release window is now closed.
- Close Tickets: Update and close the relevant JIRA tickets for the release.
