Deliverable

Automation framework

A small PyTest and Playwright framework: page objects, fixtures, and failure evidence. Readable by someone who did not write it, which is the only property that matters six months in.

Sample deliverable. Written for this site against a fictional product with no client data and no confidential material.

Layout

tests/
├── conftest.py            # server, browser, generated accounts, evidence on failure
├── api.py                 # billing API and sandbox client
├── pages/
│   ├── base_page.py       # waits on conditions, no raw sleeps
│   ├── login_page.py
│   ├── dashboard_page.py
│   └── checkout_page.py
├── data/
│   ├── users.py           # generated accounts, no credentials in source
│   └── finance.py         # expected values supplied by finance
└── suites/
    ├── test_auth.py
    ├── test_checkout.py
    ├── test_plans.py
    ├── test_webhooks.py
    ├── test_billing_db.py # direct SQL against the app database
    └── test_layout.py     # Tier 1 and 2 viewports from the device matrix
pytest.ini                 # markers: smoke, regression, slow, tc, known_defect

Page objects expose intent, not elements. A test calls LoginPage(page).open().sign_in(account) and never knows a selector.

Example

import pytest
from pages.login_page import LoginPage


@pytest.mark.smoke
@pytest.mark.tc("TC-AUTH-01")
def test_sign_in_sets_secure_session(page, context, account):
    dashboard = LoginPage(page).open().sign_in(account)

    assert dashboard.is_loaded()

    cookie = next(c for c in context.cookies() if c["name"] == "session")
    assert cookie["secure"] is True
    assert cookie["httpOnly"] is True


@pytest.mark.regression
@pytest.mark.known_defect("BUG-1142")
@pytest.mark.tc("TC-PAY-03")
def test_double_click_on_confirm_charges_once(page, api, account):
    checkout = open_checkout(page, account)
    checkout.fill_card(*VALID_CARD)

    checkout.double_click_confirm()
    checkout.result_text()
    checkout.wait_for_requests_to_settle()

    assert api.charges(checkout.subscription_id())["count"] == 1

The second test is tied to a defect twice over: the tc marker names the manual case it automates, and known_defect names the bug it reproduces. It fails on the build that has the bug and passes on the build that fixed it.

Rules

Explicit waits on conditions. No fixed sleeps in a test or a page object.
Each test creates the state it needs, a fresh account and a clock it puts back, so tests pass in any order or they are broken.
Nothing gets automated until it has run manually twice and stopped changing. Automating a moving target produces maintenance, not coverage. Two cases in the library stay manual for exactly that reason.
A failed UI test leaves a screenshot, the page source and a Playwright trace in artifacts/, so the failure is already evidence.
Markers separate the smoke suite from full regression, so CI can run the fast set on every commit.
Known defects stay in the suite, marked with their bug ID, and the run summary names them. A defect that is hidden from the suite is a defect that comes back.
A flaky test is a defect in the suite. It gets fixed or deleted, never retried into green.

Run it

Build	Command	Result	Why
4.2.1-rc3	make test	16 passed, 4 failed	The 4 failures are the known-defect tests: BUG-1142 (2), BUG-1156, BUG-1163.
4.2.1 (fixed)	make test-fixed	20 passed	The same suite, unchanged.

=========== Known defects reproduced on build 4.2.1-rc3 ===========
BUG-1142  test_processor_charges_reconcile_with_paid_invoices, test_double_click_on_confirm_charges_once
BUG-1156  test_upgrade_mid_cycle_prorates_a_31_day_period
BUG-1163  test_past_expiry_is_rejected_before_anything_is_sent

Source

Full repository on GitHub, including the small demo app the suite runs against, the Postman collection and the test case library. Setup is a virtual environment and one command, and the README covers the fixture design and where to add a new page object.