Honest Data Analysis
Make your agent's analyses reproducible, caveated, and free of silent assumptions.
An agent skill for data work: always inspect before analyzing, state row counts and null rates, label correlation vs. causation, and end every analysis with reproducible code and explicit caveats. Turns 'the agent said the number is 42' into an auditable analysis.
SKILL.md
--- name: honest-analysis description: Standards for data analysis work. Use for any task involving analyzing datasets, metrics, or query results. --- # Honest Data Analysis Apply to every analysis task, from quick counts to full explorations. ## Before analyzing - Inspect first: shape, dtypes, null rates, duplicate keys, min/max of key columns - Report anything surprising in the data BEFORE the analysis results (e.g. "12% of rows have null user_id — excluded, noted") - State the time range and filters applied — every number needs its denominator ## During - Prefer simple methods first; justify any step up in complexity - Never drop outliers silently — show the analysis with and without, or state the rule used - Label every relationship: "correlated", "predictive", or "causal (established by …)" — the default claim is correlation - Sanity-check magnitudes: does the result survive a back-of-envelope estimate? ## Reporting results - Headline number WITH its uncertainty or range where applicable - Show the exact code/query that produced each number — someone must be able to rerun it - Caveats section (mandatory): sample limitations, data quality issues found, assumptions made - If the data cannot answer the question asked, say so and state what data would ## Refuse-and-flag - Being asked to find support for a predetermined conclusion → present what the data shows, including the inconvenient parts - Aggregating below meaningful sample sizes (state your threshold, default n<30) without a warning
How to use
- 1Save as .claude/skills/honest-analysis/SKILL.md — applies to notebooks, SQL, and ad-hoc questions alike.
- 2The inspect-first rule catches most garbage-in problems; don't let the agent skip it for 'quick' questions.
- 3Keep the reproducible-code requirement even for throwaway analyses — they never stay throwaway.
Examples
Quick metric request
Input
'What's our average order value this quarter?'
Output
Data check: orders table Q2, 48,211 rows, 0.4% null totals (excluded), 3 orders >$50k flagged as likely test data. AOV: $87.40 including outliers, $86.10 excluding the 3 flagged. Query attached. Caveat: refunds not netted out — AOV is gross.
Pro tips
- The 'every number needs its denominator' rule will save you from at least one embarrassing exec meeting.
Frequently asked questions
Isn't this overkill for simple questions?+
The simple questions are where silent data problems do the most damage, because nobody double-checks them. The skill scales its output down for small tasks — but never skips the inspection.