xonecas.io
696
Cs

coder-skills / files

coder-skills / programming-practices / references / research-notes.md

7.1 KB
1# Research Notes for `programming-practices`
2
3This file captures the research basis behind the skill. Keep the
4main `SKILL.md` practical; keep detailed evidence here.
5
6## Official guidance
7
8### OpenAI Codex — Best practices
9Source: https://developers.openai.com/codex/learn/best-practices
10
11Key takeaways used in the skill:
12- Provide **goal, context, constraints, and done-when**.
13- **Plan first** for complex work.
14- Use `AGENTS.md` for durable project guidance.
15- Improve reliability with **testing and review**.
16- Start with the right task context and only add tools that
17  unlock real workflows.
18
19### GitHub Docs — Review AI-generated code
20Source: https://docs.github.com/en/copilot/tutorials/review-ai-generated-code
21
22Key takeaways used in the skill:
23- Start with **functional checks**.
24- Verify **context and intent**, not just syntax.
25- Scrutinize **dependencies**, including hallucinated packages
26  and licensing.
27- Watch for AI-specific pitfalls like ignored constraints,
28  hallucinated APIs, and deleted tests.
29- Use CI, static analysis, and collaborative review.
30
31### Google Cloud Blog — Five Best Practices for Using AI Coding Assistants
32Source: https://cloud.google.com/blog/topics/developers-practitioners/five-best-practices-for-using-ai-coding-assistants
33
34Key takeaways used in the skill:
35- Choose the right AI tool for the task.
36- Document the repo early so later generation is grounded.
37- **Make a plan** before executing complex work.
38- Prompt with relevant detail.
39- Preserve cross-session context in durable files.
40
41## Research on recurring failure modes
42
43### LLM Hallucinations in Practical Code Generation (PACMSE / 2025)
44Source: https://dl.acm.org/doi/10.1145/3728894
45Also available as preprint: https://arxiv.org/pdf/2409.20550
46
47Why it matters:
48- Focuses on **repository-level** generation rather than isolated
49  toy functions.
50- Identifies three major hallucination categories:
51  - Task Requirement Conflicts
52  - Factual Knowledge Conflicts
53  - Project Context Conflicts
54- Highlights four contributing factors:
55  - training data quality
56  - intention understanding capacity
57  - knowledge acquisition capacity
58  - repository-level context awareness
59
60How it shaped the skill:
61- Read the surrounding code first.
62- Match repo-local conventions.
63- Verify APIs and project assumptions.
64- Treat repo context as part of correctness.
65
66### Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code (2024)
67Source: https://arxiv.org/html/2404.00971v3
68
69Why it matters:
70- Builds a broader taxonomy of code hallucinations.
71- Shows that failures are not just about failing tests; they also
72  affect maintainability, readability, efficiency, and alignment
73  with user intent.
74
75How it shaped the skill:
76- Do not equate plausible output with correct output.
77- Check intent alignment, not just compilability.
78
79### We Have a Package for You! / Package Hallucinations (USENIX Security 2025)
80USENIX summary: https://www.usenix.org/publications/loginonline/we-have-package-you-comprehensive-analysis-package-hallucinations-code
81Paper repo summary: https://github.com/Spracks/PackageHallucination
82ArXiv preprint: https://arxiv.org/pdf/2406.10279
83
84Key findings referenced in the skill:
85- 576,000 code samples analyzed across 16 LLMs.
86- Average hallucinated package rate was at least:
87  - **5.2%** for commercial models
88  - **21.7%** for open-source models
89- **205,474** unique hallucinated package names were found.
90
91How it shaped the skill:
92- Never introduce a dependency without verification.
93- Treat unfamiliar package suggestions as a supply-chain risk.
94
95### More Code, Less Reuse: Investigating Code Quality and Reviewer Sentiment towards AI-generated Pull Requests (2026 preprint)
96Source: https://arxiv.org/abs/2601.21276
97
98Why it matters:
99- Reports that agent-authored PRs frequently miss reuse
100  opportunities and contain more redundancy than human PRs.
101- Notes a disconnect between objective quality issues and often
102  neutral or positive reviewer sentiment.
103
104How it shaped the skill:
105- Search for existing helpers before adding new logic.
106- Do not trust surface plausibility alone.
107- Favor reuse over near-duplicate implementations.
108
109### Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub (2026 preprint)
110Source: https://arxiv.org/abs/2601.15195v1
111
112Key findings:
113- Large-scale study of **33k** agent-authored PRs.
114- Not-merged PRs tended to be larger, touch more files, and fail
115  CI/CD more often.
116- Rejection reasons included duplicate PRs, unwanted feature
117  implementations, and agent misalignment.
118
119How it shaped the skill:
120- Keep changes small and scoped.
121- Avoid broad edits for narrow requests.
122- Stay aligned with the user's stated task.
123
124### Analyzing Message-Code Inconsistency in AI Coding Agent-Authored Pull Requests (2026 preprint)
125Source: https://arxiv.org/abs/2601.04886
126
127Key findings:
128- Analyzed **23,247** agentic PRs.
129- **1.7%** showed high message-code inconsistency.
130- The most common inconsistency type was descriptions claiming
131  unimplemented changes (**45.4%**).
132- High-inconsistency PRs had lower acceptance rates and took
133  longer to merge.
134
135How it shaped the skill:
136- Summaries must match the diff.
137- Do not overstate what changed or what was verified.
138
139### AI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agents (2026 preprint)
140Source: https://arxiv.org/abs/2512.22387
141
142Key findings:
143- Evaluated 300 generated projects across Python, JavaScript,
144  and Java.
145- Only **68.3%** executed successfully out of the box in a clean
146  environment.
147- Observed a **13.5x** average expansion from declared to actual
148  runtime dependencies.
149
150How it shaped the skill:
151- Hidden dependencies are common.
152- Working in one environment is not proof that setup is correct.
153- Make runtime assumptions explicit.
154
155### "TODO: Fix the Mess Gemini Created": Towards Understanding GenAI-Induced Self-Admitted Technical Debt (2026 preprint)
156Source: https://arxiv.org/abs/2601.07786
157
158Why it matters:
159- Found recurring self-admitted debt patterns in AI-generated
160  code: postponed testing, incomplete adaptation, and limited
161  understanding of what the code actually does.
162
163How it shaped the skill:
164- Do not ship code you cannot explain.
165- Do not leave uncertainty disguised as completion.
166
167## Notes on evidence quality
168
169- The official docs above are vendor guidance and reflect current
170  operational best practices.
171- The package hallucination work and the PACMSE repository-level
172  hallucination paper are strong anchors because they study
173  code-generation failure modes directly.
174- Several 2026 papers cited here are preprints. They are still
175  useful because they converge on the same practical patterns:
176  larger diffs, poorer reuse, hidden dependencies, overclaiming,
177  and context misalignment.
178
179## Practical synthesis
180
181Across the sources, the same engineering advice keeps showing up:
182
1831. Give the agent the right context.
1842. Plan before coding on complex tasks.
1853. Follow repo-local patterns.
1864. Reuse existing code before inventing new abstractions.
1875. Verify dependencies, APIs, and environment assumptions.
1886. Keep diffs small and scoped.
1897. Run checks.
1908. Report exactly what changed and what was verified.