696
Cs
coder-skills / files
coder-skills / programming-practices / references / research-notes.md
7.1 KB
1# Research Notes for `programming-practices` 2 3This file captures the research basis behind the skill. Keep the 4main `SKILL.md` practical; keep detailed evidence here. 5 6## Official guidance 7 8### OpenAI Codex — Best practices 9Source: https://developers.openai.com/codex/learn/best-practices 10 11Key takeaways used in the skill: 12- Provide **goal, context, constraints, and done-when**. 13- **Plan first** for complex work. 14- Use `AGENTS.md` for durable project guidance. 15- Improve reliability with **testing and review**. 16- Start with the right task context and only add tools that 17 unlock real workflows. 18 19### GitHub Docs — Review AI-generated code 20Source: https://docs.github.com/en/copilot/tutorials/review-ai-generated-code 21 22Key takeaways used in the skill: 23- Start with **functional checks**. 24- Verify **context and intent**, not just syntax. 25- Scrutinize **dependencies**, including hallucinated packages 26 and licensing. 27- Watch for AI-specific pitfalls like ignored constraints, 28 hallucinated APIs, and deleted tests. 29- Use CI, static analysis, and collaborative review. 30 31### Google Cloud Blog — Five Best Practices for Using AI Coding Assistants 32Source: https://cloud.google.com/blog/topics/developers-practitioners/five-best-practices-for-using-ai-coding-assistants 33 34Key takeaways used in the skill: 35- Choose the right AI tool for the task. 36- Document the repo early so later generation is grounded. 37- **Make a plan** before executing complex work. 38- Prompt with relevant detail. 39- Preserve cross-session context in durable files. 40 41## Research on recurring failure modes 42 43### LLM Hallucinations in Practical Code Generation (PACMSE / 2025) 44Source: https://dl.acm.org/doi/10.1145/3728894 45Also available as preprint: https://arxiv.org/pdf/2409.20550 46 47Why it matters: 48- Focuses on **repository-level** generation rather than isolated 49 toy functions. 50- Identifies three major hallucination categories: 51 - Task Requirement Conflicts 52 - Factual Knowledge Conflicts 53 - Project Context Conflicts 54- Highlights four contributing factors: 55 - training data quality 56 - intention understanding capacity 57 - knowledge acquisition capacity 58 - repository-level context awareness 59 60How it shaped the skill: 61- Read the surrounding code first. 62- Match repo-local conventions. 63- Verify APIs and project assumptions. 64- Treat repo context as part of correctness. 65 66### Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code (2024) 67Source: https://arxiv.org/html/2404.00971v3 68 69Why it matters: 70- Builds a broader taxonomy of code hallucinations. 71- Shows that failures are not just about failing tests; they also 72 affect maintainability, readability, efficiency, and alignment 73 with user intent. 74 75How it shaped the skill: 76- Do not equate plausible output with correct output. 77- Check intent alignment, not just compilability. 78 79### We Have a Package for You! / Package Hallucinations (USENIX Security 2025) 80USENIX summary: https://www.usenix.org/publications/loginonline/we-have-package-you-comprehensive-analysis-package-hallucinations-code 81Paper repo summary: https://github.com/Spracks/PackageHallucination 82ArXiv preprint: https://arxiv.org/pdf/2406.10279 83 84Key findings referenced in the skill: 85- 576,000 code samples analyzed across 16 LLMs. 86- Average hallucinated package rate was at least: 87 - **5.2%** for commercial models 88 - **21.7%** for open-source models 89- **205,474** unique hallucinated package names were found. 90 91How it shaped the skill: 92- Never introduce a dependency without verification. 93- Treat unfamiliar package suggestions as a supply-chain risk. 94 95### More Code, Less Reuse: Investigating Code Quality and Reviewer Sentiment towards AI-generated Pull Requests (2026 preprint) 96Source: https://arxiv.org/abs/2601.21276 97 98Why it matters: 99- Reports that agent-authored PRs frequently miss reuse 100 opportunities and contain more redundancy than human PRs. 101- Notes a disconnect between objective quality issues and often 102 neutral or positive reviewer sentiment. 103 104How it shaped the skill: 105- Search for existing helpers before adding new logic. 106- Do not trust surface plausibility alone. 107- Favor reuse over near-duplicate implementations. 108 109### Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub (2026 preprint) 110Source: https://arxiv.org/abs/2601.15195v1 111 112Key findings: 113- Large-scale study of **33k** agent-authored PRs. 114- Not-merged PRs tended to be larger, touch more files, and fail 115 CI/CD more often. 116- Rejection reasons included duplicate PRs, unwanted feature 117 implementations, and agent misalignment. 118 119How it shaped the skill: 120- Keep changes small and scoped. 121- Avoid broad edits for narrow requests. 122- Stay aligned with the user's stated task. 123 124### Analyzing Message-Code Inconsistency in AI Coding Agent-Authored Pull Requests (2026 preprint) 125Source: https://arxiv.org/abs/2601.04886 126 127Key findings: 128- Analyzed **23,247** agentic PRs. 129- **1.7%** showed high message-code inconsistency. 130- The most common inconsistency type was descriptions claiming 131 unimplemented changes (**45.4%**). 132- High-inconsistency PRs had lower acceptance rates and took 133 longer to merge. 134 135How it shaped the skill: 136- Summaries must match the diff. 137- Do not overstate what changed or what was verified. 138 139### AI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agents (2026 preprint) 140Source: https://arxiv.org/abs/2512.22387 141 142Key findings: 143- Evaluated 300 generated projects across Python, JavaScript, 144 and Java. 145- Only **68.3%** executed successfully out of the box in a clean 146 environment. 147- Observed a **13.5x** average expansion from declared to actual 148 runtime dependencies. 149 150How it shaped the skill: 151- Hidden dependencies are common. 152- Working in one environment is not proof that setup is correct. 153- Make runtime assumptions explicit. 154 155### "TODO: Fix the Mess Gemini Created": Towards Understanding GenAI-Induced Self-Admitted Technical Debt (2026 preprint) 156Source: https://arxiv.org/abs/2601.07786 157 158Why it matters: 159- Found recurring self-admitted debt patterns in AI-generated 160 code: postponed testing, incomplete adaptation, and limited 161 understanding of what the code actually does. 162 163How it shaped the skill: 164- Do not ship code you cannot explain. 165- Do not leave uncertainty disguised as completion. 166 167## Notes on evidence quality 168 169- The official docs above are vendor guidance and reflect current 170 operational best practices. 171- The package hallucination work and the PACMSE repository-level 172 hallucination paper are strong anchors because they study 173 code-generation failure modes directly. 174- Several 2026 papers cited here are preprints. They are still 175 useful because they converge on the same practical patterns: 176 larger diffs, poorer reuse, hidden dependencies, overclaiming, 177 and context misalignment. 178 179## Practical synthesis 180 181Across the sources, the same engineering advice keeps showing up: 182 1831. Give the agent the right context. 1842. Plan before coding on complex tasks. 1853. Follow repo-local patterns. 1864. Reuse existing code before inventing new abstractions. 1875. Verify dependencies, APIs, and environment assumptions. 1886. Keep diffs small and scoped. 1897. Run checks. 1908. Report exactly what changed and what was verified.