Release verification scenario

When post-deploy verification ends as "let's just watch for now," regressions surface late and recovery costs balloon. This guide organizes what to check on WhaTap before and after a deploy, and what criteria should drive a rollback decision.

Who this guide is for

• Teams with fast daily/weekly deploy cadences
• Engineers wiring a quality gate into CI/CD
• Teams operating canary or blue/green deployments
• Teams that want to reduce ad hoc "something feels off" judgments after deploy

Prerequisites

• Team dashboard / alerts in production → Quick Wins (3) complete
• Deploy timestamps are recorded (CI/CD logs from Jenkins, GitHub Actions, ArgoCD, or the team wiki)
• A rollback procedure agreed on by the team (at minimum: "who makes the rollback call")

Release verification timeline

  Pre-deploy     First 10 min       Day 1            Week 1
     ↓              ↓                 ↓                 ↓
① Baseline     ② Acute regression  ③ Delayed reg.   ④ Trend check
  snapshot       detection           detection         & retro

① Pre-deploy — Capture the baseline

Goal: capture the "normal" reference values just before deploy.

Logging the following from the 30–60 minutes right before deploy makes post-deploy comparison clear.

1. Snapshot these metrics from Dashboard:
   • TPS (average and peak)
   • Average and max response time
   • Error rate
   • Active transaction count
2. Screenshot the hitmap pattern (for post-regression comparison).
3. Choose one recording method:
   • Auto-capture as a CI/CD pipeline step
   • Numeric fields in the team's release note template
   • Drop the Flexboard URL + deploy time in the team channel

Regression is visible only when a baseline exists

"Feels slow" is less useful than "180ms before deploy → 260ms after, +45%" for decision-making.

② First 10 minutes — Acute regression detection

Goal: detect obvious immediate regressions within 10 minutes → roll back if needed.

Menus: Application dashboard + the product's Flexboard.

Metrics to watch (priority order)

Table | Metrics to watch right after release

Metric	Regression signal	Immediate action candidate
Error rate	Spike above 1%, especially 5xx	Strongly consider rollback
Average response time	Sustained 30%+ above baseline	Investigate in parallel; roll back above threshold
TPS	Abnormal sharp drop (dependency call failures?)	Check logs/traces
Agent connectivity	Inactive appears	Check process startup failure

Procedure

1. From the deploy completion alert, start a 10-minute timer.
2. Refresh the dashboard every 30–60 seconds (manual refresh unnecessary if auto-refresh is fast).
3. When a metric looks off:
   • Check hitmap pattern — point distribution changes vs. pre-deploy
   • Pull one sample in Transaction trace
   • If needed, switch into the Incident response scenario
4. Rollback decisions follow the pre-agreed trigger criteria (next section).

Example rollback triggers

Objective criteria the team can agree on in advance:

• Error rate > 1% sustained 5 min → auto-rollback or immediate rollback call
• Avg response +50% sustained 10 min → rollback under review
• A new 5xx exception class appears → judgment after cause check

Distinguish gut feeling from criteria

"Feels off" is something to flag, not a decision criterion. With trigger criteria, numbers — not people — decide rollbacks, and the emotional burden drops.

③ Day 1 — Delayed regression detection

Goal: catch issues that only appear after some time has passed.

Common delayed regressions

• Memory leak: heap memory grows gradually 1–12 hours after deploy
• Connection leak: DB connection pool exhaustion trend
• GC frequency increase: doesn't return to normal after warm-up
• Anomaly only at specific times: interaction with batch jobs or scheduled tasks

Procedure

1. 24 hours after deploy, open Dashboard with a two-day range.
   • Directly compare the 24 hours before vs. after deploy
2. Check heap memory, connection pool, and GC trends → Heap memory alert, DB connection pool
3. Review the error type list in the daily report.
4. Inspect performance by traffic time-of-day (peak-hour reproducibility).

④ Week 1 — Trend check & retro

Goal: decide "was this deploy a good change long-term?"

Checklist

1. In Report, compare the week before vs. after deploy with the weekly report.
   • Apdex, error rate, TPS, average/p95/p99 response time
2. Review Event history for that week — did new event rules start firing more?
3. Cross-check with user feedback (if available) — do tech metrics match user experience?

Deploy retro record (team wiki)

Release ${version} retro — ${YYYY-MM-DD}

## Overview
- Purpose: (feature / improvement / fix)
- Scope: (services affected)

## Performance change (1 week before vs. after)
- TPS: X → Y (+Z%)
- Avg response: X ms → Y ms
- p99 response: X ms → Y ms
- Error rate: X% → Y%

## Events
- Events flagged as regression: (if any)
- Newly triggered event rules: (if any)

## Verdict
- Success / Watch / Partial regression / Rolled back

## Apply to next deploy
- (Extra checks, new event rules, test cases, etc.)

Tips for gradual deployments (canary, blue/green)

• Name agents by version/deploy type (e.g., svc-v1.2-stable, svc-v1.3-canary) → easier version-level comparison via Flexboard and event tags
• When canary traffic is small, keep a dedicated filtered dashboard so metric changes don't get buried
• Design a Flexboard that auto-compares error rate and response time while traffic shifts to canary

Scale up with automation

Wire a quality gate into the deploy pipeline

• Add a CI/CD hook for auto-rollback if error rate > 1% within 5 minutes
• Use the WhaTap Open API to fetch last-n-minute metrics for the gate decision

Auto-generate deploy retros via MCP

With WhaTap MCP, ask an AI agent to "build a 1-week-before-vs-after comparison report for the last deploy" to auto-draft the retro. Drastically reduces data gathering time.

Verify

• A pre-deploy baseline is recorded (somewhere)
• It's clear who looks at what at the 10-min / 1-day / 1-week marks
• Rollback decisions are made by pre-agreed numeric criteria
• A week after deploy, a retro record exists in the team wiki

Once release verification becomes routine, "vague post-deploy anxiety" turns into a "structured verification phase." Faster regression detection also lets you shorten the deploy cycle safely.

Next steps

• Response routine when regression is detected → Incident response scenario
• Share pre/post-deploy metrics regularly → Performance reporting scenario
• Tidy up team dashboard, permissions, alerts → Team collaboration scenario