QA
A QA case study based on HouseBud’s written test plan: the risks it starts from, the suites that gate every merge, what they’ve found, and what’s deliberately left out.
- unit tests
- 44
- API tests
- ~150
- end-to-end scenarios, on 5 browser projects
- 53
- accessibility scans per layout
- 30
- backend statement coverage
- 89.5%
- problems found since moving to CI
- 8
How it grew
Testing grew with the app. API tests arrived in its first week, and Playwright end-to-end tests a month later; every feature since has shipped with tests of its own. Once the household modules were in and a demo was running on staging, the next layer made sense. In October 2026 the suites became required checks in CI, then spread to five browsers and gained unit tests, coverage thresholds and accessibility scans. The test plan writes it all down; this page is the short version.
Start from the risks
Testing effort follows what would hurt users most. The plan lists those risks first, then which tests cover each one.
| Risk | Covered by |
|---|---|
| Someone sees or changes another household’s data | API tests in every module: non-members and former members get a 404, and items are looked up by id and household together |
| Wrong money | Unit tests on the maths: splits in whole cents always add up to the total, balances always sum to zero, suggested transfers always settle everyone |
| Wrong dates | Unit tests on every repeat rule and its edges: short months, leap years, chores done late |
| People added without consent | API and end-to-end tests: every invitee accepts or declines, and only owners can form a household |
| Logged out or bounced by timing | A regression test for a dropped profile request, and the WebKit jobs for a race no test can force |
| Unusable for some people | axe on every main page, in both themes |
| Broken on Safari or iPhone | WebKit desktop and an emulated iPhone 14 in CI on every push |
| Tests that pass by luck | Seed users are read-only; tests that change data create their own users, and run with 4+ parallel workers |
Test levels
Every rule is tested at the lowest level that can check it. Date and money edge cases are cheap and exact as unit tests; checking each one through the API would mean building a household for every case. The API tests then prove the pieces work together, and the end-to-end tests prove it works for a person in a browser.
| Level | Tool | What it checks | Size |
|---|---|---|---|
| Unit | Vitest | Business rules on their own: repeats, rotations, splits, balances. No database. | 44 tests, ~5 s |
| API integration | Vitest + Supertest, Postgres | Every endpoint: validation, permissions (401/403/404), what gets saved. One schema per worker. | ~150 tests |
| End-to-end | Playwright | User flows in a real browser, against a production build of the frontend with the backend and database behind it. | 53 scenarios × 5 projects |
| Accessibility | axe-core via Playwright | WCAG 2.1 A and AA on 15 pages, light and dark, desktop and phone. | 30 scans × 2 projects |
| Manual | Me, on Windows and Android | New features before merge, look and feel, what automation can’t judge. | Per feature |
Coverage is measured on the backend, unit and API tests together: 89.5% of statements, 81.9% of branches, 96.6% of functions, 90.7% of lines. CI fails below 87/79/94/88. Coverage says what ran, not what was checked, so it’s a floor against untested code rather than a score.
Browsers and devices
The same end-to-end tests run on five Playwright projects: Chromium, Firefox and WebKit on desktop, plus a Pixel 7 on Chromium and an iPhone 14 on WebKit. The phones are emulated (viewport, touch, user agent), not real devices. A failure that happens only on WebKit is treated as a likely real Safari bug, not noise. One was.
The gate
main is protected, and a pull request can’t merge until five required checks pass: format, types, lint and build; the API tests with coverage; and end-to-end tests on Chromium, Firefox and WebKit, each in its own job, in parallel. The coverage thresholds and the axe scans run inside those checks, so missing either blocks the merge too. Beyond the checks:
- A test that passes only on retry is a defect to investigate, not a fluke. Two of the bugs below started as tests that failed some of the time on one engine.
- A bug that isn’t fixed straight away is written down in a file in the repo, and removed in the pull request that fixes it.
- The test plan is updated in the same pull request as any change to the tests, CI or environments.
What testing found
Eight problems found since the suites moved to CI: five in the app, three in the tooling and the tests. Each fix shipped with the pull request that describes it.
| Found by | Problem | Fix |
|---|---|---|
| First CI run | A seed script only worked on Windows. | Made it portable. #27 |
| Firefox | A dropped profile request sent logged-in users to the login page. | The app retries before giving up. #28 |
| WebKit, in CI | Household pages bounced members to the overview. | The household list reports “loading” until it’s loaded for the current login. #29 |
| WebKit | Test logins raced the login page’s own redirect. | Tests set the session on a static page. #28 |
| axe | Grey text under 4.5:1 contrast in both themes; teal text at 4.1:1 in dark mode. | Retuned the colour tokens, and added a text colour for dark mode. #32 |
| axe | The chat list’s “New” button was announced as “Open chat”. | Its accessible name now starts with the word on screen. #32 |
| Parallel runs | Tests changing shared seed users could break each other. | Fresh users per test, kept apart from the seed data. #28 |
| Writing missing tests | When a chat room’s owner left, ownership went to whoever was invited first, not who had accepted longest ago. | Participants record when they accepted, and handover sorts by it. #35 |
The bounce only Safari saw
After the WebKit jobs were added, household pages occasionally sent members back to the overview. The trace showed why: the household list first loads before login and finishes as “no households”. On Chromium and Firefox the reload after login normally started first; on WebKit the profile sometimes came back sooner, and the page concluded the user wasn’t a member. No test can force that response order, so the WebKit jobs are its coverage. Real Safari and iPhone users would have hit it, and Chromium and Firefox were fast enough to hide it (#29).
The gap that hid a bug
The test plan listed one gap: nothing tested what happens to a chat room when its owner leaves. Writing those tests showed the handover didn’t follow the documented rule. It picked whoever was invited first, not who had accepted longest ago, and people invited together were ordered by however the database listed them. The fix records when each person accepted, and six new tests pin the rule down (#35).
When the infrastructure breaks
Two failures on the same day had nothing to do with the app:
- GitHub’s
ubuntu-latestmoved to Ubuntu 26.04, where the Playwright version in use can’t install its browsers. Every job is now pinned to 24.04 (#33). - An Ubuntu package mirror stalled twice, timing out WebKit’s dependency install at 30 minutes before a single test ran. The end-to-end jobs now run inside Playwright’s Docker image, with the browsers and their libraries already installed (#34).
What isn’t tested
Deliberately left out, for now:
- Load and performance. Households are small; a load test is on the roadmap.
- Security beyond access control. Permissions and the demo’s rate limit are tested, but there’s no penetration test or dependency audit gate.
- Visual regression. Layout is checked by hand.
- Frontend components on their own. Behaviour is covered end to end instead.
- Accessibility beyond what axe sees. Automated checks catch part of the problems; focus order and screen-reader flows need a manual pass.
- Production. There isn’t one yet; staging is checked by hand.