System Prompts Versus User Prompts: Empirical Lessons from an 18-Model LLM Benchmark on Hard Constraints

Abstract

This article presents a rigorous comparative benchmark of 18 large language models (LLMs), and a sequel to my previous article about creative writing, on a tightly specified prompt-generation task: crafting a three-scene children’s story under ten discrete constraints, including lexical bans, word-count precision, and structural requirements. The study evaluates models across open-weight and proprietary APIs, quantisation variants, and prompt-engineering strategies, focusing on rule adherence, prose quality, and required editing intervention. The evolution of the system prompt is documented in detail, with verbatim artefacts and impact analysis. Results indicate that no model achieved perfect compliance “out of the box,” but prompt strategy and interface choice significantly influenced outcomes. Actionable recommendations are provided for prompt engineers and production teams seeking both creative output and strict specification fidelity.

1 Introduction

Context and Motivation for Multi-Constraint Prompting

Recent advances in large language models have elevated the baseline for naturalistic, stylistically rich text generation. However, for applications requiring strict adherence to editorial or policy constraints—such as educational publishing, branded content, or regulated outputs—models’ ability to consistently follow complex, multi-part instructions remains an open and practical challenge. Prompt engineering has emerged as a critical discipline for shaping model behavior, but the limits of direct compliance, especially under “hard” constraints like banned lexemes, precise name usage quotas, and narrowly defined document structure, are not well-documented in comparative, empirical terms.

Research Questions and Scope

This research addresses the following questions:

1. How do top-performing LLMs (open and proprietary) compare in their ability to deliver outputs that satisfy a demanding, multi-constraint prompt for a children’s story?
2. To what extent does prompt structure—especially the evolution of the system prompt—affect compliance and editing burden?
3. What are the recurring failure modes, and what workflows best mitigate them in production?
4. How do interface and quantisation choices impact both creative quality and mechanical obedience?

The study spans 18 model runs across different architectures, access modalities, and prompt configurations, providing both quantitative and qualitative evaluation. The analysis is intended for senior ML engineers, prompt designers, and practitioners deploying LLMs in editorial pipelines.

2 Methodology

2.1 Model Line-up & Quantisation

The benchmark covers a broad cross-section of contemporary LLMs, including both open-weight and proprietary offerings. Table 1 in §3.1 enumerates the full line-up. Key features:

• Open-weight models: Qwen-8B-Q5_K_S, Qwen-30B (multiple sampling settings), Qwen-235B-fp8, DeepSeek-r1-0528, Mistral-Medium-2505, Gemma-3-27B-it, and Grok-3 (in both “old prompt” and rerun conditions). Quantisation settings (e.g., GGUF, fp8) and sampling parameters (temperature, top-p, top-k) are noted for each run.
• Proprietary APIs: Claude (Sonnet and Opus, via Anthropic), GPT-4 (o1 and 4.1), OpenAI o3 and GPT-4·5 (webchat), Gemini-Pro and Gemini-Flash (Google), and Grok-3 (xAI chat).
• Frontend modalities: Both API and webchat interfaces were used, with explicit documentation of the sampling parameter availability.

Each model was prompted to generate a three-scene children’s story, with the system/user prompt structure and constraints held as constant as interface limitations allowed. Runs varied in their exposure to different prompt versions (S-0, S-1, S-2; see §4).

2.2 Prompt-Engineering Protocol

The prompt-engineering protocol was designed to probe models’ ability to internalize and execute a specification grid of ten hard constraints. These included:

• Document length (2,100–2,400 words)
• Structural markers (exactly three H2 scenes, specific narrative beats)
• Lexical bans (e.g., “shimmer*”, “melodic”, “ancient”, “silver” > 1 instance)
• Quota for protagonist’s name (“Adrian” exactly two times per scene)
• Scene-specific irreversible losses
• Ban on meta/planning language and body-reaction clichés

The protocol evolved through three primary versions of the system prompt, reflecting increasing explicitness and integration of rules (§4.1–§4.3). User prompts remained minimal (“weak”) to isolate the effect of system-level constraint encoding. All outputs were assessed against a 10-point rubric, with errors and editing effort tabulated (§3.2–§3.3). Quantitative adherence and qualitative features (voice, imagery, structure) were recorded for each run.

3 Results

3.1 Full Benchmark Matrix (Table 1)

Table 1. Complete results matrix for all 18 runs. Each entry includes model, access mode, sampling settings, output characteristics, rule breaks, and rubric score.

The matrix shows that no model achieved a perfect 10, and only a minority met the passing threshold (≥5/10) without significant post-editing. Proprietary models clustered near the top in narrative quality and partial compliance, while open-weight models lagged on rule fidelity but showed progress in creativity.

3.2 Error & Compliance Statistics (Table 2)

Table 2. Aggregate rule-adherence statistics across all 18 runs.

Figure 2. Average 10-Point Compliance by Vendor Family

^{Higher bars ⇢ better constraint compliance across 18 story generations.}

The most persistent errors were lexical leaks—models nearly always reused banned words (“shimmer”, “silver”, etc.) and failed to strictly cap protagonist name usage, regardless of explicit instructions. Only three runs achieved full compliance on irreversible loss and structural markers.

3.3 Editing-Effort Metric (Table 3)

Table 3. Manual edit burden from draft to full spec compliance for key runs.

The editing metric underscores a practical reality: even “winning” models needed several targeted edits to meet all criteria. Lower-ranked outputs required substantial rewriting and augmentation, especially in structure and loss logic.

3.4 Podium & Near-Miss Models

The head-to-head podium is as follows:

• Gold: OpenAI GPT-4 (o1) — Highest compliance score (7/10), vivid and read-aloud-ready prose, and minimal interventions required to achieve full specification. Most errors were lexical or quota-based and easily patched.
• Silver: OpenAI o3 — Comparable narrative polish, with an imaginative guardian and a strong climax. Two minor rule breaks (lexeme, early loss) were trivial to fix, slightly edging out Claude.
• Bronze: Claude-Sonnet v4 — Delivered a warm, storybook voice and rhythmic sentences, following the three-scene arc consistently. Missed some bans and overused the protagonist’s name, but the structure and pacing were otherwise tight.

Near-miss models included:

• Claude-Opus v4: Tighter than some larger models, but the draft introduced extra clichés and exceeded the “Adrian” quota.
• Qwen-30B @ T 0.7–0.8: First open-weight model to cross the 5-point line, but still suffered lexical leaks that required regex cleanup.
• GPT-4·5 (webchat): Rich scene construction, yet output was truncated below the minimum word count and included repeated banned adjectives.
• Gemini-Pro / Flash: Consistent 6/10 scores with good sentence variety, but lexeme and name rules were generally ignored.
• Mistral-Medium: Good style, but omitted required headings and lacked a true external showdown.
• Grok-3 and Gemma-27B: Creative premises but failed to meet multiple structural and length specifications.

4 Evolution of the System Prompt

Figure 3. Prompt-engineering ladder.
S-0 introduced a minimal role + task block.
S-1 delivered the full constraint table (two-turn splice).
S-2 tightened banned-lexeme grammar, explicit 10-pt self-check, and tone guards.

4.1 Prompt S-0 (verbatim, download)

The initial system prompt, S-0, was concise and relied on downstream user turns to specify constraints.

You are a professional children’s book author known for natural, engaging storytelling that sounds as if written by a skilled human. This story will be read aloud to children (and their parents), and must NEVER sound mechanical or repetitive. Use explicit reasoning and self-correction after every scene.

You MUST follow the steps below, and pay extra attention to every instruction labeled IMPORTANT or VERY IMPORTANT.

1. Planning (Before Writing) List the main characters, setting, and the basic story structure (beginning, middle, end). IMPORTANT: Briefly outline how you will avoid overusing the main character’s name and ensure varied, natural sentence flow.

2. Writing Process (Per Scene) For each scene: Write the scene in a way that sounds natural and engaging when read aloud. VERY IMPORTANT: Use the main character’s name ONLY at the start of scenes, during major transitions, or for special emphasis. Otherwise, use pronouns or context. IMPORTANT: Vary sentence structure and rhythm. Avoid repetitive openers or mechanical phrasing. IMPORTANT: Show emotions and actions through description, dialogue, and interaction—not by simply stating them.

3. Self-Editing Chain-of-Thought (After Each Scene) After writing each scene, PAUSE and review using the checklist below: VERY IMPORTANT: Did I use the main character’s name only when necessary? IMPORTANT: Did I avoid repeating words, phrases, or sentence patterns? VERY IMPORTANT: Does the passage sound like it was written by a human, not an AI? IMPORTANT: Are emotions and actions shown naturally, not just told?

   If you notice issues (repetition, awkward flow, unnatural dialogue, mechanical language), you MUST revise the scene before continuing. Summarize in 1–2 sentences how you improved the scene, or confirm that it needed no changes.

4. Quality Control and Final Review (After the Story) After finishing the story, perform a final review: VERY IMPORTANT: Check for main character name repetition and unnatural phrasing throughout the story. IMPORTANT: Ensure varied sentence length and openers. VERY IMPORTANT: Confirm that dialogue and description are natural and engaging. VERY IMPORTANT: The story must have a clear beginning, middle, and end, and be satisfying to read aloud.

   If you find any problems, you MUST revise and finalize before completing the story.

General Rules VERY IMPORTANT: Never allow any passage to sound mechanical, repetitive, or AI-generated. VERY IMPORTANT: Do not leave scenes underdeveloped or unresolved. IMPORTANT: Avoid clichés, padding, and bland statements. VERY IMPORTANT: The goal is a story both children and parents will enjoy, with warmth, imagination, and subtlety.

VERY IMPORTANT: You must explicitly follow this reasoning and self-editing process, giving extra care to all instructions marked IMPORTANT or VERY IMPORTANT. Only deliver story output that fully meets these standards.

(This was the only text in the system slot; most hard specs were provided later in the user turn or not at all.)

4.2 Prompt S-1 (verbatim, download)

Prompt S-1 introduced a structured, tabular specification, though it was initially delivered in two separate user turns, resulting in some models generating before seeing all constraints.

Role & Voice You are a professional children’s-book author. Your style is warm, vivid, and absolutely human when read aloud.

---

0. Story Blueprint (model must do this silently)

• Characters – Adrian (9), one magical ally/guardian, optional antagonist.
• Structure – 3 scenes only:
  1. Entrance (portal & hook)
  2. Trials (real danger; Adrian loses something precious)
  3. Climax → Return (external showdown + emotional cost; home again, changed)

---

1. Hard Output Specs

Item	Requirement
Length	2 100 – 2 400 words (≈ 11–13 min read-aloud)
Scenes	Exactly 3; H2 heading for each
Use of “Adrian”	Exactly 2 per scene
Irreversible cost	Each scene must force a tangible loss
Climax	Scene 3 must feature an external obstacle Adrian alone overcomes
Vocabulary	Ban “shimmer*”, “melodic”, “ancient”, “silver” (max 1)
Body clichés	Replace “heart pounding / breath caught / throat tightens” etc
No meta	Don’t reveal planning

2. Read-Aloud Quality Checklist (model must self-check)

1. Vary sentence rhythm …
2. Dialogue sounds real …
3. Loss and climax feel in-the-moment …
4. Final paragraph gives clear closure.

Deliver only the final story.

(Because it arrived in two separate user turns, some models generated after the first half, ignoring the stricter table that came second.)

4.3 Prompt S-2 (verbatim, download)

Prompt S-2 was a further refinement, integrating all rules in a single contiguous block, with explicit negative prompts and self-revision instructions.

Role & Voice You are a professional children’s-book author. Your style is warm, vivid, and absolutely human when read aloud.

---

0. Story Blueprint (model must do this silently)

• Characters – Adrian (9), one magical ally/guardian, optional antagonist.
• Structure – 3 scenes only:
  1. Entrance (portal & hook)
  2. Trials (real danger; Adrian loses something precious)
  3. Climax → Return (external showdown + emotional cost; home again, changed)

---

1. Hard Output Specs

Item	Requirement
Length	2 100 – 2 400 words (≈ 11–13 min read-aloud)
Scenes	Exactly 3; H2 heading for each
Use of “Adrian”	Exactly 2 per scene
Irreversible cost	Each scene must force a tangible loss (No resets)
Climax	Scene 3 must feature an external obstacle Adrian alone overcomes
Vocabulary	Ban all forms of “shimmer*”, “melodic”, “ancient”, “silver” (max 1)
Body clichés	Replace “heart pounding / breath caught / throat tightens” etc
No meta	Don’t reveal planning

---

Third-person past tense only. Avoid filler adjectives.

2. Read-Aloud Quality Checklist (10-pt rubric)

1. Vary sentence rhythm …
2. Dialogue sounds real …
3. Loss and climax feel in-the-moment …
4. Final paragraph gives clear closure.

If any item fails, revise before output. Deliver only the final story.

4.4 Impact Analysis (Table 4–6)

Table 4. Compliance metrics by system prompt version.

Prompt completeness was pivotal. Under S-0, only rudimentary structure and length were followed. S-1, when fully seen, halved gross violations. S-2, delivered as a single block, further reduced structural misses, but did not entirely eliminate lexical leaks, even in top models.

Additional observations:

• Tone and pacing: S-1 and S-2’s explicit tone guards led to shorter, more direct sentences and fewer adjective chains, especially in GPT-4 and Claude. Qwen and Gemini sometimes over-corrected, producing flatter prose.
• Self-revision: Only the GPT-4 family models responded observably to the “revise before output” instruction, occasionally emitting internal notes before presenting the final text.
• Lexeme bans: Banned-word clusters routinely slipped past, especially in open-weight models and those with non-English tokenization quirks. Even GPT-4 missed inflections like “shimmering” in second passes.
• Loss logic: The S-2 prompt’s explicit “No resets” instruction raised compliance on “loss per scene” from 56% to 100% in late runs.

5 Discussion: Implications & Next Steps

Patterns in Compliance and Creative Output

No model, regardless of size or training regime, achieved perfect adherence to the ten-point specification grid. The most common breaking points were lexical bans (especially for “shimmer*” and “silver”), overuse of the protagonist’s name, and failure to include a tangible loss in every scene. Even top models required fast, targeted edits—primarily regex-based lexeme swaps and name pruning—to reach full compliance.

Frontier models (GPT-4, Claude, Gemini) consistently generated the richest, most human narrative voice, but also the most stubborn lexical violations. Open-weight models exhibited rapid improvement in narrative quality and structural awareness, yet lagged in mechanical obedience, especially on fine-grained word-count and counting-based rules.

Interface and sampling strategy mattered. Local runs offered granular sampling control but risked runaway length; web-based frontends sometimes truncated outputs below minimum word count, regardless of prompt. Relying solely on prompt engineering to enforce “hard” constraints proved insufficient—post-processing remains necessary for publisher-grade deliverables.

5.2 Why Human Review Remains Mandatory 🧑‍⚖️

Even the best-scoring model (GPT-4 · o3, 9 / 10) shipped minor lexical leaks and subtle rhythm glitches. Automated post-processors trimmed ≈ 65 % of rule breaks, but three error classes still demanded human judgment:

Consequently, a human-in-the-loop proofreading pass (≈ 8 min/story) proved non-negotiable, ensuring factual fidelity and emotional nuance that no regex-based linter can guarantee.

🛠️ Editor’s 90-Second Lint Run

1. Confirm word-count ➜ wc -w.
2. Grep banned lexemes ➜ grep -Ei "shimmer|melodic|ancient|silver".
3. Count “Adrian” occurrences (perl -pe 's/Adrian/++$c if /e').
4. Spot-check climax: external obstacle + protagonist agency.
5. Re-read final paragraph: closure & tone guard.

Recommendations for Prompt Engineers

1. Deliver the full spec in a single system prompt block. Models sample eagerly; partial or late-arriving constraints are less likely to be followed.
2. Red-team negative prompts and banned-word lists. Include synonyms and root+inflection expansions to minimize embedding cluster leaks.
3. Repeat fragile constraints. Quota rules and narrow word-count targets benefit from explicit repetition or “run an internal counter” phrasing.
4. Encourage self-rewrite. Only some models act on “revise before output,” but it can close the gap for those that do.
5. Pair prompts with deterministic post-processing. Regex sweeps for banned terms, name quotas, and heading checks reduced average errors from 7 to 3.

Immediate Take-Aways for Production Teams

• Always move critical editorial rules to the system prompt. The top instruction wins in system–user collisions and standardizes output across model vendors.
• Use concise user prompts. Reserve the user message for the core task (“write the story”), leaving all “how” instructions to the system layer.
• Deploy a post-linter in production. Prompting alone is not enough; simple scripts for word count, banned words, and structural checks are fast and effective.
• For deliverables, prefer API endpoints over webchat. APIs offer more consistent output and sampling control, minimizing truncation and hidden pre/post-processing.

6 Conclusion

All things considered, this benchmark demonstrates that, as of mid-2024, no major LLM—irrespective of scale or access modality—delivers publisher-ready, multi-constraint outputs without human-in-the-loop intervention. The evolution from minimal to comprehensive system prompts significantly improved rule adherence and reduced editing burden, but could not fully eliminate lexical leaks or counting failures. Prompt engineering remains an art of trade-offs: more rules bring more compliance, but may slightly dampen creative spontaneity. A hybrid approach—strong system prompts, lean user requests, and light post-linting—yields the most practical path to high-quality, spec-compliant narrative generation in production workflows.

Appendices

A Weak User Prompts (verbatim)

The “weak” user prompts used in all runs were minimal, to isolate the effect of system-level constraint encoding. Examples:

Please write the story.

Great, I’m ready—please write the story.

B 12-Point QA Checklist

1. 2,100–2,400 words total.
2. Three scenes, each with H2 heading.
3. Adrian’s name appears exactly two times per scene.
4. Each scene features a tangible, irreversible loss.
5. Scene three includes an external showdown.
6. No use of banned words (“shimmer*”, “melodic”, “ancient”, more than one “silver”).
7. No body-reaction clichés (e.g., “heart pounding”, “breath caught”).
8. No meta or planning language.
9. Third-person past tense only.
10. Minimal filler adjectives.
11. Varied sentence rhythm and real-sounding dialogue.
12. Final paragraph provides closure.

C Regex Bans & Counting Script

The post-linter script used for manual and automated editing included:

• Word count check:

wc -w story.txt

• Banned lexeme GREP:

grep -Ei 'shimmer|melodic|ancient|silver' story.txt

• Protagonist name frequency:

grep -o 'Adrian' story.txt | wc -l

• Scene heading check:

grep -c '^## ' story.txt

• Loss line presence (manual/regex for “lost|gave up|sacrificed”).

These checks enabled rapid identification of the most common error types post-generation, reducing editorial intervention time to under ten minutes for top-tier model outputs.

Appendix A Glossary