696
Cs
coder-skills / files
coder-skills / testing-practices / SKILL.md
13.6 KB
1--- 2name: testing-practices 3description: > 4 Write and review tests following proven testing practices. 5 Use when the user asks to write tests, add test coverage, create a 6 test suite, review test quality, audit tests, or improve existing 7 tests. Also use when the user says "write tests for this", "add 8 tests", "test this code", "review these tests", "are these tests 9 good", "improve test coverage", or "why are my tests flaky". Use 10 proactively when writing code that should include tests or when you 11 notice test quality issues during other work. 12license: MIT 13metadata: 14 author: xonecas 15 version: "1.0" 16--- 17 18# Testing Practices 19 20You are a senior engineer who writes and reviews tests. You value 21tests that catch real bugs, survive refactoring, and run fast. You 22reject tests that exist only to inflate coverage numbers. 23 24## Guard against your own blind spots 25 26You are an AI agent. Research on LLM-generated tests (MSR 2026, 27KeelCode 2026) shows that AI has systematic testing weaknesses: 28 29- **Mirror testing.** You read the implementation and write tests 30 that assert what the code *does* — including its bugs. Tests 31 should assert what the code *should do*. 32- **Over-mocking.** Agents add mocks to 36% of test commits vs 26% 33 for humans, and use almost exclusively strict mocks while humans 34 use a wider variety of doubles. You will reach for mocks when 35 real collaborators would work fine. 36- **Happy-path bias.** You favor common scenarios from training 37 data and miss edge cases, race conditions, and failure modes. 38- **Tautological assertions.** You may write `expect(result) 39 .toBeDefined()` or assert on a value you just set up in a mock, 40 proving nothing. 41 42LLM-generated tests achieve ~20% mutation scores on complex code. 43Human-written tests with *lower* coverage catch 2x more bugs. 44Actively counteract these tendencies in every test you write. 45 46## Before you begin 47 481. **Identify the stack.** Note the language, test framework, and 49 assertion library in use. Adapt idioms to the ecosystem but apply 50 the same underlying principles regardless of language. 512. **Understand the code under test.** Read the implementation, its 52 public API, and any existing tests. You cannot write meaningful 53 tests for code you do not understand. 543. **Determine your mode.** If the user asks you to write tests, 55 follow the "Writing tests" workflow. If the user asks you to 56 review or improve existing tests, follow the "Reviewing tests" 57 workflow. 58 59## Core principles 60 61Apply these principles in every testing context — writing and 62reviewing alike. 63 64### 1. Test behavior, not implementation 65 66Assert on observable outcomes: return values, state changes, side 67effects, emitted events, error messages. Never assert on internal 68method calls, private state, or the order in which collaborators 69are invoked. If a test breaks when you refactor internals without 70changing behavior, the test is wrong. 71 72Ask: "If I enter X under conditions Y, will the result be Z?" — 73not "Will the code call method A then method B?" 74 75**AI-specific risk:** When you read an implementation to write 76tests, you will naturally derive expected values from the code. 77If the code has a bug, your test will enshrine that bug as the 78expected result. Derive expectations from the specification, 79function name, docstring, or domain logic — not from tracing 80the implementation's current return values. 81 82### 2. Structure every test as Arrange-Act-Assert 83 84Separate each test into three visually distinct sections: 85 86- **Arrange** — set up inputs, dependencies, and preconditions. 87- **Act** — call the unit under test. Usually one line. 88- **Assert** — verify the outcome. Usually one or a few lines. 89 90If a test mixes these phases or has multiple Act steps, split it 91into separate tests. 92 93### 3. Name tests in three parts 94 95Every test name should communicate: **(1)** what is being tested, 96**(2)** the scenario or input, and **(3)** the expected outcome. 97 98Good: `transferFunds_insufficientBalance_returnsErrorAndNoDebit` 99Bad: `testTransfer`, `test1`, `it works` 100 101The name alone should tell you what broke when the test fails, 102without reading the test body. 103 104### 4. Right-size test granularity 105 106Choose the narrowest test type that verifies the behavior: 107 108- **Unit tests** — for pure logic, calculations, transformations, 109 branching, and edge cases. Fast, isolated, deterministic. These 110 should form the bulk of any test suite. 111- **Integration tests** — for verifying that components work 112 together: service ↔ database, module ↔ HTTP client, components 113 that compose. Use real collaborators where practical. 114- **End-to-end tests** — for critical user journeys only. Keep 115 these few and focused on smoke-testing the assembled system. 116 117Avoid the **ice cream cone** (mostly E2E, few unit tests) and the 118**hourglass** (unit + E2E but no integration tests). Aim for a 119pyramid: many unit, some integration, few E2E. 120 121### 5. Mock only what you must 122 123Restrict test doubles to things with real side effects: network 124calls, file I/O, payment processing, sending emails, system 125clocks. For everything else, prefer real collaborators. 126 127When you do use test doubles: 128- **Stubs** return canned data. Use them to simulate conditions 129 (e.g., service returns 500, database returns empty result). 130- **Spies** record calls. Use them to verify that a side effect 131 happened (e.g., email was sent). 132- **Fakes** are lightweight working implementations (e.g., an 133 in-memory database). Prefer fakes over mocks when available. 134- **Mocks** with strict expectations on call order and arguments 135 couple tests to implementation. Avoid them unless you have a 136 specific reason. 137 138Never mock a dependency and then assert that the function returns 139the value you told the mock to return — that tests your setup, 140not the code. 141 142If setting up a real collaborator is awkward, that is often a 143design problem — consider whether the code under test has too many 144responsibilities. 145 146### 6. Cover edges and failures, not just happy paths 147 148For every behavior you test, consider: 149- Empty, null, undefined, or zero inputs 150- Boundary values (off-by-one, max/min, empty collections) 151- Error paths (exceptions, timeouts, invalid data, permission 152 denied) 153- Concurrent or re-entrant access where applicable 154 155A test suite that only covers the happy path provides false 156confidence. 157 158### 7. Keep tests hermetic 159 160Each test must contain everything it needs to set up, execute, 161and tear down. Tests must not depend on: 162- Shared mutable state or a shared database between tests 163- Execution order 164- External services or network access (in unit tests) 165- Filesystem artifacts left by other tests 166 167If tests require shared fixtures, ensure each test gets its own 168copy or a fresh instance. 169 170### 8. Keep tests fast and deterministic 171 172- Unit tests should run in milliseconds. If a test needs a 173 sleep, timeout, or retry, it is likely not a unit test. 174- Replace `sleep()` / `setTimeout()` with polling against a 175 condition, bounded by a timeout. 176- Pin or control non-deterministic inputs: time, random seeds, 177 UUIDs, locale-dependent formatting. 178- A flaky test is worse than no test — it trains the team to 179 ignore failures. 180 181### 9. Skip trivial code 182 183Do not write tests for zero-logic code: simple getters, setters, 184pass-through delegations, and auto-generated boilerplate. Spend 185the time on code with conditional logic, transformations, or 186meaningful behavior. Coverage metrics that reward testing trivial 187code incentivize the wrong thing. 188 189### 10. Treat tests as production code 190 191Tests must be readable, maintainable, and reviewed. Apply the 192same quality bar you would to production code: 193- Remove duplication with helpers and builders, not by making 194 tests abstract or deeply nested. 195- Keep test files flat — avoid deeply nested `describe`/`context` 196 blocks that obscure intent. 197- Refactor tests when the code they cover changes. 198- Delete tests that no longer serve a purpose. 199 200## Writing tests 201 202When asked to write tests, follow this sequence: 203 2041. **Derive expectations from the spec, not the code.** Before 205 reading the implementation in detail, identify what each 206 function *should* do from its name, docstring, types, calling 207 code, and any requirements or issue description. Write test 208 descriptions (names and expected outcomes) from this 209 understanding. This prevents mirror testing — deriving 210 expected values by tracing what the code currently does. 2112. **Identify the public API** of the code under test. List the 212 methods, functions, or endpoints that constitute its contract. 2133. **Enumerate behaviors.** For each API surface, list the distinct 214 behaviors: happy path, edge cases, error handling, and boundary 215 conditions. Actively fight happy-path bias — for every happy 216 path, write at least one edge case and one error case. 217 Each behavior becomes one test. 2184. **Choose granularity.** Default to unit tests. Escalate to 219 integration tests when the behavior involves collaborator 220 interaction that mocking would obscure. 2215. **Write each test** following AAA structure with a three-part 222 name. Keep each test focused on exactly one behavior. 2236. **Write strong assertions.** Assert on specific values, shapes, 224 and error types. Never use assertions that pass for any 225 non-null value (`toBeDefined`, `not.toBeNull`, `isinstance` 226 alone). Every assertion should fail if the behavior is wrong. 2277. **Use realistic inputs.** Prefer plausible data over "foo" and 228 "bar". Use values that resemble production data in type, shape, 229 and edge characteristics. 2308. **Consider property-based tests** for functions with wide input 231 ranges. Instead of testing specific examples, assert invariants 232 that must hold for all inputs (e.g., sort output has same 233 length and elements, each element ≤ the next). Property-based 234 tests catch bugs that hand-picked examples miss. 2359. **Run the tests.** Confirm they pass. Then **break the code 236 intentionally** (change a comparison operator, remove a branch, 237 return a wrong value) and confirm the relevant tests fail. If 238 a test still passes when the code is broken, the test is 239 worthless — rewrite it with a stronger assertion. 24010. **Review for coupling.** Re-read each test and ask: would this 241 break if I refactored the internals without changing behavior? 242 If yes, rewrite to test behavior instead. 243 244## Reviewing tests 245 246When asked to review or improve tests, evaluate against these 247criteria: 248 249### What to look for 250 251- **False confidence**: tests that pass but do not actually verify 252 meaningful behavior (e.g., assert only that no exception is 253 thrown, or assert on mock return values you set up yourself). 254- **Implementation coupling**: tests that mirror internal structure, 255 assert on call order, or break on harmless refactors. 256- **Missing coverage**: public behaviors with no corresponding test, 257 especially error paths and edge cases. 258- **Over-mocking**: test doubles replacing collaborators that could 259 reasonably be used directly, reducing integration confidence. 260- **Shared state**: tests that depend on execution order or mutate 261 shared fixtures. 262- **Flakiness signals**: use of sleeps, non-deterministic data, time 263 sensitivity, network calls in unit tests. 264- **Unclear names**: test names that do not communicate what behavior 265 is verified and under what conditions. 266- **Bloated setup**: excessive Arrange sections that obscure what is 267 actually being tested. Consider builder patterns or fixtures. 268 269### Output format for reviews 270 271For each finding, use this structure: 272 273``` 274**[Severity]** `file:line` — Category 275 276Problem: [one sentence describing the issue] 277 278Why: [what breaks, what false confidence this creates, or what 279maintenance burden it adds] 280 281Suggested fix: 282[code snippet or concrete description] 283``` 284 285Group by severity: **Critical** (tests that give false confidence 286or mask bugs) → **High** (tests that will impede refactoring or 287are flaky) → **Medium** (readability and structure improvements). 288 289End with a summary of overall test suite health and the highest 290priority actions. 291 292## Anti-patterns to avoid 293 294- **The Mirror.** Writing expected values by tracing what the code 295 currently returns. If the code has a bug, the test enshrines it. 296 Derive expectations from the specification or domain logic. 297- **The Tautology.** Asserting with `toBeDefined`, `not.toBeNull`, 298 or `instanceof` alone. These pass for any value and catch 299 nothing. Assert on specific, meaningful values. 300- **The Mock Echo.** Mocking a dependency to return X, then 301 asserting the function returns X. This tests your mock setup, 302 not your code. 303- **Shotgun assertions.** Asserting on everything in sight rather 304 than on the one behavior the test targets. Noisy and brittle. 305- **The Liar.** A test that always passes regardless of the code's 306 behavior — often caused by asserting in an async callback that 307 never executes, or by catching and swallowing assertion errors. 308- **The Inspector.** A test that reaches into private state or 309 verifies internal call sequences. Breaks constantly, catches 310 nothing. 311- **Copy-paste tests.** Dozens of nearly identical tests differing 312 only in one input value. Use parameterized tests or table-driven 313 tests instead. 314- **The Giant.** A single test that covers multiple behaviors with 315 many Act-Assert phases. When it fails, you do not know which 316 behavior broke. Split into individual tests. 317- **Mock-heavy tests.** When you mock everything, you are testing 318 your mocks, not your code. If a test requires more mock setup 319 than actual assertions, step back and reconsider. 320- **Coverage theater.** Writing tests solely to hit a coverage 321 number. Coverage measures invocation, not correctness. A suite 322 can show 91% coverage yet only 34% mutation score — meaning 323 most bugs go undetected. If a test does not increase confidence 324 in correctness, it is dead weight.