Automating Technical Blog Discovery: From Commits to Stories

Context

Writing about technical work is valuable but tedious. Every day I push code across multiple repositories, but most of it—README updates, dependency bumps, typo fixes—isn't worth documenting. The genuine engineering stories—debugging sessions, architectural decisions, performance optimizations, and lessons from failures—often disappear into commit history, forgotten before they become valuable posts.

The problem: manually reviewing daily commits to find stories worth telling is friction. I wanted to remove that friction entirely by building an automated pipeline.

The Challenge

This problem has two parts:

Efficiently finding the work: With commits spread across multiple repositories, how do you collect today's activity without hitting API limits or slow iteration through every repo?
Automatically filtering the noise: Most commits are trivial. How do you distinguish between "fix typo in README" and "solved a difficult concurrency bug" without pre-judging everything?

The naive solution—iterate through all repositories and fetch all commits—is O(n repos) and wastes tokens analyzing noise.

The Solution: A Three-Stage Pipeline

Architecture Overview

daily.sh
  ├─ collect_commits.py    [GitHub Search API]
  ├─ collect_diffs.py      [GitHub REST API]
  └─ claude (agent)        [Analyzes & writes]

Stage 1: Efficient Commit Collection

Instead of iterating through repositories, I used the GitHub Search API to query directly:

# Pseudocode: What collect_commits.py does
GET /search/commits?q=author:username+committer-date:2026-08-18

This is O(1) and returns only commits from the target date by the target author, across all repositories in a single request. The results are formatted into structured markdown (temp/commits.md) containing:

Repository name
Commit SHA
Commit message
Author and timestamp

Key insight: The search API is logarithmic in practice. It queries indexed data across all repositories in one request. Iterating through repos, checking for commits, and fetching diffs would be O(n repos) per day—unsustainable as the number of repositories grows.

Stage 2: Fetch Detailed Diffs

With a list of commits, collect_diffs.py fetches detailed diffs from the GitHub REST API:

For each commit SHA:
  GET /repos/{owner}/{repo}/commits/{sha}
  Extract patch and file changes
  Write to temp/diffs.md

This gives Claude the raw material to understand what actually changed.

Stage 3: Intelligent Filtering with Claude

Instead of pre-judging what's interesting with keyword matching, I hand the raw diffs to Claude with clear instructions:

Ignore trivial changes (typos, README updates, dependency bumps)
Prefer architectural decisions, debugging, performance improvements, failures, and lessons
Generate a markdown draft only if the story is genuinely worth telling
Use structured output: context, problem, solution, reasoning, lessons

The shell script orchestrates this:

#!/bin/bash
claude --permission-mode acceptEdits -p "
  Read temp/commits.md and temp/diffs.md

  Determine if there's a genuinely valuable technical story...
  If yes, create drafts/{slug}.md with full article
  If no, report that there's nothing worth writing
"

What Didn't Work

Initial attempts to pre-filter commits by keywords (words like "bug", "fix", "performance") before sending to Claude failed because:

False positives: A commit message could say "fix typo" and still be noise
False negatives: Important work gets committed with vague messages like "refactor auth flow"
Wasted tokens: Sending everything to Claude for re-filtering defeated the purpose

The solution: move the decision entirely to Claude. The filtering logic is now in natural language (the prompt), not in code, making it easy to refine what "blog-worthy" means without touching Python.

Design Decisions and Technical Reasoning

Why GitHub Search API instead of repo iteration?

The search API queries indexed data efficiently. Iterating through repos, checking for commits, and fetching diffs would be O(n repos) per day—unsustainable as repositories multiply.

Why filter at the AI layer?

Keyword matching is brittle and creates false positives and negatives. Natural language filtering is flexible and maintainable. The prompt is the specification, making it versionable in git. If the definition of "blog-worthy" changes, you update the prompt, not the code.

Why drafts instead of auto-commit?

Automation should remove friction, not judgment. The system generates candidates, but you review them. This keeps the human in the loop for editorial decisions while automating discovery and drafting.

Why permission modes matter?

Setting --permission-mode acceptEdits in Claude Code eliminates interactive prompts. Claude can read input files and write drafts without manual approval, enabling hands-free scripting.

Key Technical Insights

API efficiency matters: Search API over repo enumeration saves tokens and latency. By providing Claude only the necessary data, the expensive part (analysis) becomes more efficient.
Let the AI decide: Provide raw data with clear criteria rather than pre-filtering with rules. This is more reliable and maintainable than keyword matching.
Automate the routine: Removes friction from the writing process without removing editorial control.
Version control the prompts: The system's filtering logic lives in the shell script as a string, making it part of git history. This shows intent and decisions better than code comments.
Structured input formats: Both commits.md and diffs.md are machine-readable markdown, making it easy for Claude to parse them reliably.

What's Next

The system currently generates draft files for manual review. Future iterations could:

Automatically commit drafts to the portfolio repository
Generate social media summaries or newsletter snippets
Track which stories get published and learn patterns from editorial decisions
Extend to other formats (videos, threads, documentation)

Lessons Learned

Efficient APIs save tokens: By using Search instead of iteration, you reduce the amount of data Claude analyzes, lowering latency and cost.
Natural language > keyword matching: Filtering in prompts is more flexible and maintainable than brittle rules in code.
Permission modes unlock automation: Hands-free scripting enables workflows that would otherwise require manual intervention.
Drafts as intermediaries: Generating candidates without auto-publishing keeps editorial control while removing friction. This pattern extends to social media, newsletters, and other derived content.
Turn passive history into prompt machines: Commit history becomes a writing prompt machine. Each day automatically surfaces the stories worth telling and packages them as drafts ready for polish and publication.

This system transforms how I think about documenting technical work—from manual curation to automated discovery, with editorial control preserved at the final step.

Automating Technical Blog Discovery From Commits to Stories