The cost of brevity

A shorter final reply is pleasant to read. It is also a tempting shortcut for estimating the cost of an agent session. I wanted to know whether skills that encourage brevity or smaller changes would reduce the whole bill without dropping useful work.

We compared Ponytail, Caveman and Chisle. Ponytail asks for the smallest complete change and appropriate checks. Caveman compresses prose. Chisle combines terse output with code and context discipline. Our team already uses Ponytail, which is a conflict of interest to disclose rather than a reason to favor its results.

This is an exploratory report from an incomplete, single-model experiment. It supports a narrow observation: shorter final replies did not consistently predict lower reported session cost on this task suite. It does not establish a winning skill or a company-wide adoption policy.

Research direction and interpretation: Fernando Ramos. The Claude overnight team prepared and ran the experiment. Codex recovered the evidence, repaired the grader, re-evaluated saved outputs and checked the analysis. The article separates those contributions from manual work by a person.

The design had eight configurations: no skill, each skill alone, each pair, and all three together. We planned five repeats of each configuration on 12 tasks: 480 task/configuration/repeat cells. The earlier 64-run pilot is a separate dataset and is not pooled into this comparison.

Each task used the same user prompt and a fresh repository. Valid subject rows report the model identifier claude-sonnet-5-5, at medium effort, using Claude Code 2.1.286. The identifier is recorded control information, not a guarantee that provider weights are immutable. Run order was shuffled with a fixed seed.

The task suite covered pagination and money bugs, retry behavior, a refactor, a configuration rename, dependency choice, HMAC verification, code review, explanation and three communication tasks. Hidden tests or required-fact rubrics supplied automated checks. We verified each checker against a known bad and a known good fixture.

All three skills used the same text-injection mechanism. Review tasks also included each package's review companion. Hooks, output-compression integrations and other plugin behavior were inactive. The subject of this experiment is therefore the pinned text bundles, rather than every feature of the packages. Exact commits and licenses are in the skill pins and the source trail.

The twelve challenges

These are the original task IDs and intended challenges, taken from their task.json definitions in the research snapshot.

These fixtures isolate small decisions in short Python files or supplied facts. They do not exercise a large repository's dependency graph, migrations, deployment or sustained maintenance. Repeating them improves repeatability; it does not turn twelve examples into repository-scale evidence.

For each task we compared a configuration's mean with that task's baseline mean, then gave all 12 tasks equal weight. A ratio of 1.15 means a 15% higher point estimate. The 95% intervals use 2,000 bootstrap draws that resample both baseline and treatment repeats within each task.

These intervals describe repeat variability on these fixed tasks. They do not correct for missing observations, multiple comparisons or our choice of tasks, and they are not uncertainty estimates for all real coding work.

Ponytail's cost estimate was about 15% higher than baseline; Caveman's was about 9% higher. Chisle's was about 1% lower, with an interval that includes both savings and extra cost. Combining all three produced a point estimate about 22% higher.

Caveman shortened final replies by about 13% and Chisle by about 16%, by the task-normalized point estimate. The final reply is only one part of a session. Prompt text, tool work, intermediate output and cache accounting also affect reported cost. This experiment does not isolate which component caused any difference.

The numeric comparison

The full CSV includes output-token and reply-word intervals. The raw CLI outputTokens field is distinct from final-reply words; we do not infer hidden reasoning volume from reply length. The retained total_tokens measure sums reported input, output, cache-read and cache-creation fields and does not independently add a thinking field. The article's comparisons use the CLI's cost field and visible word counts, not a claim about hidden reasoning.

The 480-cell attempt produced 425 valid observations. Forty-nine cells returned a session-limit response and six timed out. We excluded those 55 from quality and ratio calculations. They are missing observations, not failed quality checks. Shuffling run order does not make that missingness harmless.

High pass counts do not establish superior correctness or safety. Unequal sample sizes, small synthetic tasks and regex-based fact checks constrain that interpretation. The automated judge panel is incomplete too: Codex returned 168 valid scores and Claude returned 120; 48 Claude judgements failed. Styles can reveal configurations despite anonymous labels. We do not use that partial panel to announce a quality winner. The old judge ledger does not record resolved model identity, so current defaults cannot establish which exact judge models produced it.

The source ledger reports $16.13 for subject-model attempts, using Claude Code's reported list-price accounting. That is neither a reconciled invoice nor the cost of the entire research project. Judge costs and any unreported timeout charges are not included.

On the 425 valid observations from this 480-cell Sonnet 5.5 design, Caveman's final replies were about 13% shorter while reported session cost was about 9% higher. Chisle stayed near baseline cost and Ponytail's point estimate was about 15% higher. Brevity and session efficiency are separate outcomes here. The missing cells and incomplete judging do not establish a quality winner or show that extra cost bought better engineering.

Hypotheses the data cannot yet separate

A newer model may already perform useful engineering practices without a reminder. Repeating advice could add input processing, formatting obligations or instruction-reconciliation work with little marginal benefit; older or lower-baseline models might benefit more. These are untested explanations. Latency or token counts cannot reveal an internal reconciliation process.

FollowBench separates content, situation, style, format and example constraints. That motivates measuring adherence separately from useful task outcomes; it does not establish these mechanisms. Training-time internalization of these specific skills remains untestable here without training provenance or a controlled exposure intervention. Behavioral convergence alone cannot show which texts a model encountered.

An independent review found a real isolation flaw. After the model's Bash session ended, our grader imported generated Python using ordinary host Python. The working directory was temporary, but the process still had host access. The model's Bash sandbox did not protect the later grader.

Codex repaired the grader and Git postprocessing to run inside a macOS OS sandbox, rejected generated symlinks and graded fresh disposable copies reconstructed from saved patches. Checks demonstrate denied reads and writes to an outside fixture, denied TCP, and an import-time write attempt that could not change the host sentinel. Explicit system/runtime paths and filesystem metadata remain readable; this is not a VM or a claim that all outside-root access is impossible.

We then regraded all 425 valid saved outputs in isolation. No recorded pass/fail result changed. Eight focused safety and analysis tests pass, and all 12 graders pass their bad/good fixture checks. An independent review verified all 425 current grader/task/artifact fingerprints. This establishes the recovered checks, not retroactive isolation of the original executions or full confinement of every model-session tool.

The original ledgers remain intact. Recovery made no new model calls. We also corrected the analysis: the original bootstrap held observed baseline means fixed; the recovery resamples those observations as well. The table in this article supersedes the earlier intervals and factorial-effect claims.

Download the research snapshot ZIP, archive SHA-256, file manifest, recovery audit, aggregate CSV, sanitized observations in JSONL and figure provenance.

The ZIP contains pinned skill text with its licenses, task fixtures, the repaired grader, numeric analysis code and 480 sanitized latest observations. The 55 excluded observations have null quality results. It excludes raw transcripts, session/auth metadata, mutable overnight logs and individual saved diffs or answers. It supports numeric reanalysis; replaying grades from saved diffs requires the retained internal research archive. Two exported source edits remove machine-specific CLI paths without changing grader logic; the archive documents them.

To reproduce the numeric ratios after extracting the ZIP:

python3 -B harness/analyze_observations.py

Download the figure generator and input data with its separate SHA-256. It uses Matplotlib 3.10.9 as generation-only tooling, without adding a website runtime dependency. The figures come directly from the linked CSV. Their generator checks the counts against the sanitized observations, verifies that displayed intervals contain their point estimates and records source hashes. The research snapshot ZIP SHA-256 is 126ea810d22e809a0c403c521120f32d7ac936d81ca80fbc83a0c07f5c441eee.

Choose a skill for the behavior you want, then measure complete sessions on work representative of your team. An advertised reduction in prose does not establish a reduction in the bill, and this incomplete subset does not justify adopting or rejecting a skill across the company.

The experiment still lacks 55 subject cells and 48 judge calls. Finishing them requires the same subject setup and available quota. A planned extension would use one common HTTP tool loop across models, with fixed skills, tools, turn limits and isolated fixtures. That differs from V1's Claude Code harness, so its observations must remain a separate dataset. This article reports no new-model results.

The next comparison needs a vector of outcomes. Freeze correctness and regression checks outside the generator's control. Report acceptance and failures before comparing style. For code, measure per-function cyclomatic complexity, function counts, median/maximum complexity and total decision increments, sum(CC - 1). Radon's definition adds one per function; extraction alone can therefore change an average. Record module decisions separately. Cognitive complexity can add an understandability proxy, with Sonar's definition as a reference. Name and pin the analyzer: a third-party implementation is not the official Sonar analyzer or a direct measure of quality. Keep production/test size, dependencies and added/deleted lines separate; low churn can also mean omitted behavior.

Use current frontier judges from different families with a frozen neutral rubric. Give them anonymized artifacts, task requirements and immutable check results, with no treatment skills, treatment labels or generator identities. Rate correctness, maintainability, scope and explanation separately. Reverse pair order and inspect disagreements: MT-Bench documents position, verbosity and self-enhancement biases. Two judges agreeing is not ground truth.

Record end-to-end latency alongside input/output, cache and provider-reported reasoning tokens. Preserve unavailable fields as null; never infer hidden reasoning from reply length or elapsed time. Separate generator and judge costs, distinguish invoices from price estimates, and retain failed attempts. Efficiency requires acceptable quality with less time or cost, rather than a shorter answer alone.