Deliverable
Test strategy — subscription billing web app
How I decide what gets tested, what does not, and what evidence a release decision rests on. One page, because a strategy nobody reads is not a strategy.
Sample deliverable. Written for this site against a fictional product with no client data and no confidential material.
Product
A subscription billing application: self-serve signup, card capture through a third-party payment processor, plan upgrades and downgrades with proration, automated retries on failed payment, and webhook delivery to a downstream ledger. Web and responsive mobile web. No native apps in this release.
The product under test is the demo app in the repository that backs these deliverables. Build 4.2.1-rc3 carries three known defects; 4.2.1 has them fixed.
In scope
- Signup, authentication, session handling, and account recovery.
- Card capture, first charge, and stored payment method management.
- Plan change flows including proration calculations and effective dates.
- Failed payment handling: retry schedule, dunning email, and account suspension.
- Webhook contracts consumed by the downstream ledger service.
- Responsive layout on the device matrix defined in the mobile coverage deliverable.
Out of scope
- The payment processor's own card handling — covered by their certification, not ours.
- Load and performance testing. Scoped as a separate engagement with its own tooling (Locust) and its own report — see the note below.
- Penetration testing. Requires a specialist; flagged to the client as a gap.
- Marketing site content beyond a smoke check.
Writing out of scope down is the point of the document. It converts an assumption into a decision somebody signed off on. Load testing is out of scope here because this product's traffic is predictable — on systems where it is not, I model the worst realistic hour in Locust and treat headroom as a release criterion.
Risk register
| ID | Area | Impact | Likelihood | Test response | Where it is tested |
|---|---|---|---|---|---|
| R-01 | Payment capture | High | Medium | Full scripted coverage plus exploratory session per release. No sampling. | TC-PAY-01 to 03 · Postman charge and database checks · BUG-1142 |
| R-02 | Proration on plan change | High | High | Boundary and calculation testing against finance-supplied expected values. | TC-PLAN-01 and 03 · Postman plan change · BUG-1156 |
| R-03 | Dunning and retry logic | High | Low | Time-shifted environment testing. Cannot be validated in a normal manual pass. | TC-PAY-04, manual only |
| R-04 | Webhook delivery to downstream | Medium | Medium | Contract tests plus replay of malformed and duplicated events. | TC-HOOK-01 and 02 · Postman webhooks folder |
| R-05 | Marketing pages | Low | Medium | Smoke check only. Accepted risk, documented here so nobody is surprised. | Smoke check only |
Approach
- Risk-based coverage. Depth follows the register above, not an even spread across features.
- Scripted regression for anything involving money movement, so results are comparable release to release.
- Timeboxed exploratory sessions on new and recently changed areas, charter written before the session and notes captured during it.
- Automated coverage in Python, PyTest and Playwright for the flows that repeat every release. Nothing gets automated until it has run manually twice and stopped changing.
- API-level validation in Postman where the UI would only obscure the failure, including a check of the database after a charge.
Environments
| Environment | Data | Payment processor | Used for |
|---|---|---|---|
| Development | Seeded fixtures | Sandbox | Developer smoke only |
| Staging | Anonymized production subset | Sandbox | Full regression and exploratory |
| Time-shifted | Cloned from staging, clock advanced | Sandbox | Dunning, retries, renewal cycles |
| Production | Live | Live | Post-release smoke, read-only where possible |
In the demo repo these collapse into one local app. The time-shifted environment is that same app with its sandbox clock moved, which is how the proration cases run.
Entry and exit
Entry to test
- Build deployed to staging and reachable.
- Acceptance criteria written for every story in the release.
- Developer smoke passed. QA does not spend a day proving the build is broken.
Exit to release
- Zero open blocker or critical defects.
- Every R-01 to R-04 risk has executed coverage and a recorded result.
- Open major defects individually accepted by the product owner, in writing.
- Rollback verified, not assumed.
Reporting
One summary per release: what was covered, what was not, open defects by severity, and a plain recommendation to ship or hold. Defect counts alone do not answer the only question a stakeholder is actually asking.
Build 4.2.1-rc3: HOLD
Covered R-01 payment capture, R-02 proration, R-04 webhooks.
18 PyTest and Playwright tests, 19-request Postman collection.
Not covered R-03 dunning and retries. TC-PAY-04 is manual and was not run on this build.
Load and penetration testing are out of scope, as above.
Open 1 blocker BUG-1142 duplicate charge on double submit
1 major BUG-1156 proration overcharges in 31-day months
1 minor BUG-1163 card expiry accepts a past date
Reason Exit criteria need zero open blockers and executed coverage for R-03.
Next Re-test on 4.2.1, which passes 20 of 20 tests, and run TC-PAY-04.An example of the summary for the demo build. The three open defects are the ones filed on the defect reports page.