Skip to content

ShiyanW/ai-revision-guard

revision-guard

Verified

Use this skill whenever the user asks to revise, edit, polish, rewrite, improve, refine, clean up, tighten, smooth out, or review existing prose — especially academic papers, dissertations, thesis chapters, literature reviews, grant proposals, or journal submissions, including prose in LaTeX source. Also activate when the user asks "is this good enough?", "am I over-editing?", says "lock this paragraph", or is on their second or later revision of the same section. Do NOT activate when the user asks to write something from scratch, or to review code, scripts, or data files.

references/research-evidence.md

View on GitHub
# Research Evidence: Why Over-Refinement and Homogenization Matter

This file lists the research and practitioner observations behind
revision-guard's design. Load it when the user asks why the skill exists or
wants citations for a design decision. Each entry says whether it is
peer-reviewed work, a preprint, or a practitioner post.

---

## 1. The Perfectionism Trap

**Source:** Liz Fosslien, LinkedIn post (2026). Practitioner observation.
**Link:** https://www.linkedin.com/posts/liz-fosslien_ai-is-great-at-many-things-including-amplifying-activity-7435386240037896192-Kzb0

**Observation:** AI amplifies perfectionist tendencies. A model always has
another suggestion when asked, so users iterate past the point of usefulness
and end up with what Fosslien calls "Frankenstein-ed slop": text worse than
the original draft.

**Design implication:** The revision cap (Step 4) and the MARGINAL rating
(Step 3) create explicit stopping points.

---

## 2. LLM Revisions Change Meaning, Even Under Grammar-Only Instructions

**Source:** Abdulhai, M., White, I., Wan, Y., Qureshi, I., Leibo, J. Z.,
Kleiman-Weiner, M., & Jaques, N. (2026). "How LLMs Distort Our Written
Language." arXiv:2603.18161. Preprint, not yet peer-reviewed.
**Link:** https://arxiv.org/abs/2603.18161

**Design:** Two parts. (a) A user study in which participants wrote an
argumentative essay without an LLM, with light LLM help, or with heavy LLM
use. (b) Student essays from the ArgRewrite-v2 corpus, collected in 2021
before LLMs were widespread, revised by three models (gpt-5-mini,
gemini-2.5-flash, and a Claude Haiku model whose version the paper reports
inconsistently) from the same expert feedback the human writers received,
under five instructions ranging from "general revision" to "grammar only."

**Key findings:**
- Heavy LLM use raised the share of essays that took a neutral stance by
  68.9% relative to the control group (the abstract rounds this to "nearly
  70%"). This is a relative increase in the proportion of neutral essays, not
  a 70% movement along a scale.
- Heavy users were more likely to say the essay was not in their own voice.
- Human revisions made small shifts in varied directions. LLM revisions made
  larger shifts, and all three models pushed the essays in the same semantic
  direction.
- Even when asked for grammar edits only, the models changed the essays'
  meaning.
- Lexical divergence from the original draft (Jensen-Shannon) was 0.2-0.3 for
  human revisers; the most aggressive model, gpt-5-mini, reached nearly three
  times that.
- LLM text used more nouns and adjectives and fewer pronouns. In the user
  study, essays written with LLMs had 50% fewer pronouns than human-only
  essays.

**Limits:** Student argumentative essays, not academic papers, and small,
low-cost models rather than each lab's flagship.

**Design implication:** The banned-substitution list (Step 2), the confidence
and first-person checks (Step 5), and the domain presets that lock
argumentative sections target these effects.

---

## 3. What Professional Writers Cut from AI Text

**Source:** Chakrabarty, T., Laban, P., & Wu, C.-S. (2025). "Can AI Writing
Be Salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the
Writing Process through Edits." *Proceedings of the 2025 CHI Conference on
Human Factors in Computing Systems*.
**Link:** https://doi.org/10.1145/3706598.3713559 (preprint:
https://arxiv.org/abs/2409.14509)

**Design:** Professional writers edited LLM-generated paragraphs in creative
domains. The direction matters: humans editing AI text, not AI editing human
text.

**Key findings:** The writers agreed on a seven-category taxonomy of problems
in LLM text: clichés, unnecessary or redundant exposition, purple prose, poor
sentence structure, lack of specificity and detail, awkward word choice and
phrasing, and tense inconsistency. The resulting LAMP corpus holds 1,057
edited paragraphs. GPT-4o, Claude 3.5 Sonnet, and Llama 3.1 70B shared the
same limitations; none outperformed the others on writing quality.

**Design implication:** The pattern catalog
(`references/over-refinement-patterns.md`) covers the same ground: inflated
and ornamental wording, exposition that tells the reader what to think, and
lost specificity.

---

## 4. Homogenization Grows with Volume

**Source:** Moon, K., Green, A. E., & Kushlev, K. (2025). "Homogenizing effect
of large language models (LLMs) on creative diversity: An empirical comparison
of human and ChatGPT writing." *Computers in Human Behavior: Artificial
Humans*, 6, 100207.
**Link:** https://doi.org/10.1016/j.chbah.2025.100207

**Key findings:** Across three preregistered studies of 2,200 college
admissions essays, each additional human-written essay added more new ideas
than each additional GPT-4 essay. The gap grew as more essays were included
and persisted after prompt and parameter changes meant to diversify the AI
output.

**Design implication:** Each additional round compounds homogenization, so
the skill caps rounds instead of trying to prompt the model into diversity.

---

## 5. Variance Shrinks: Regression Toward the Mean

**Source:** Xie, Y., & Xie, Y. (2026). "When artificial intelligence makes
everything similar: The risks of content homogenization." *Chinese Journal of
Sociology*.
**Link:** https://doi.org/10.1177/2057150X261419573

**Key finding:** Generative models are prone to "regression toward the mean":
the variance of their output shrinks relative to real-world distributions.
The authors note that prior work has focused on average performance, with
much less attention to this loss of variance.

**Design implication:** The anchoring step (Step 0) keeps the user's confirmed
version, not the model's, as the reference point for every later edit.

---

## 6. Self-Correction Needs External Feedback

**Sources:**
- Huang, J., Chen, X., Mishra, S., et al. (2024). "Large Language Models
  Cannot Self-Correct Reasoning Yet." *ICLR 2024*.
  https://arxiv.org/abs/2310.01798
- Kamoi, R., Zhang, Y., Zhang, N., Han, J., & Zhang, R. (2024). "When Can
  LLMs Actually Correct Their Own Mistakes? A Critical Survey of
  Self-Correction of LLMs." *Transactions of the Association for
  Computational Linguistics*. https://aclanthology.org/2024.tacl-1.78

**Key findings:** Without external feedback, LLMs struggle to correct their
own responses, and performance sometimes degrades after self-correction
(Huang et al.). Self-correction works when reliable external feedback is
available (Kamoi et al.). Both papers study reasoning tasks; the skill applies
the lesson to writing by analogy.

**Design implication:** The skill does not rely on its own judgment alone. It
adds a script-based diff check and a structured checklist (Step 5), the user's
explicit locks (Step 0), an on-disk round count (Step 4), and an independent
second model (Step 5b).

---

## 7. Tells Live in Structure, Not Word Choice

**Source:** Russell, J., Rajendhran, R., Pham, C. M., Iyyer, M., & Wieting, J.
(2026). "StoryScope: Investigating idiosyncrasies in AI fiction."
arXiv:2604.03136. Preprint.
**Link:** https://arxiv.org/abs/2604.03136

**Design:** 10,272 writing prompts, each answered by one human author and five
LLMs (61,608 stories of about 5,000 words), described by 304 discourse-level
narrative features (character agency, chronology, how explicitly the theme is
stated) rather than word choice.

**Key findings:** Narrative features alone separated human from AI stories at
93.2% macro-F1, keeping over 97% of the performance of models that also used
stylistic cues. AI stories over-explain their themes and favor tidy,
single-track plots; they cluster in a shared region while human stories spread
out.

**Limits:** Fiction, not academic prose.

**Design implication:** A banned-word list only catches the surface. The diff
checker's structural checks (summary closers, stock transitions, uniform
sentence length) and the claims-first order in Step 2 target structure. A
paragraph that explains its own point back to the reader is the academic
version of an over-explained theme.

---

## 8. Practitioner Observations

From the discussion under Fosslien's post:

**XinChi Yang (Principal PM):** "Each iteration pulls language toward the
model's statistical center. After a few passes you start losing the structure
or voice that made the original draft work."

**Ana Petras (Strategic AI Transformation):** Stopped asking "make this better"
and started asking "is this good enough to send?" — fundamentally different
answers and mindset.

**Christopher Nguyen (UX Designer):** "The problem is not the AI. It is that
we outsourced the 'done' decision to a tool that has no concept of done."

**Design implication:** The "Is this good enough?" command and the MARGINAL
rating implement these observations: after a few productive rounds, more
iteration stops paying.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall