Rohit Saini

Blindspot, Part 3: $0.0215 a Report, and the Things That Broke

Mon, 03 Aug 2026 12:00:00 +0530

Part 3 of three on Blindspot. Part 1: reading the video · Part 2: extraction and evidence.

The analysis pipeline is the interesting half. The half that decides whether it survives contact with real traffic is cheaper to describe and more expensive to get wrong.

Check the budget between stages, not after

Every model call records usage and estimated cost, and a per-report budget is checked between pipeline stages. If research exhausts it, the system stops gathering evidence and records that coverage may be incomplete. It does not keep going until the bill happens to stop.

Measured on real Reels:

	cost
Lean profile, per checkable Reel	~$0.015
Strong profile	$0.20–0.25
Built-in web search, per call	~$0.01

Evidence retrieval dominates, because retrieved documents arrive as context tokens.

The local database currently holds 32 completed reports across 13 unique posts, plus one failed job. Those runs total $0.6884, an average of $0.0215 per report.

Everything else exists to keep that number down: exact-report caching, media caching so reposts aren’t retranscribed, triage stopping non-factual input early, a rolling daily budget that can pause the worker, per-user quotas, endpoint rate limits. Promptry records model, prompt version, token counts, latency, and cost for every invocation.

A crashed job should not crash you forever

The worker polls a SQLite-backed job queue. On startup it marks orphaned running jobs as failed rather than replaying them.

A job that reliably crashes the process must not be allowed to keep crashing it. Replay-on-boot turns one poison message into an outage.

Delivery has to be idempotent, because the platform isn’t

Website and Instagram DM share one analysis pipeline but differ at admission and delivery. On a Meta message webhook, Blindspot validates and deduplicates the event, resolves the shared Reel or carousel, reserves the sender’s quota, acknowledges, enqueues, then sends a compact result and a public link when it finishes.

Instagram delivers duplicate webhooks. Users share a second post while the first is still running. So message state is persisted and every step is idempotent: acknowledgement, processing, retry, final send. Failed delivery retries up to five times before being marked terminal.

Report links carry opaque attribution tokens, which connect later feedback to a delivery without exposing the Instagram sender on a public page.

Boring infrastructure, two sharp edges

It all runs on a shared Oracle ARM host behind Cloudflare and nginx: Next.js app, worker, FFmpeg, ingestion tools, and SQLite on one machine, because SQLite needs local shared storage.

The worker doesn’t hot-reload. Editing pipeline code while the web app is running leaves the interface on new code and the worker on old code, which produces results that make no sense until you remember why. serve.sh supervises both, but pipeline changes still need a worker restart.

Two failures worth writing down.

Exit code 139, no error message. A Rosetta x86 Node process was loading an arm64 better-sqlite3 binary. Scripts now run through pnpm exec node so the Node architecture matches the installed native module.

A .gitignore that ignored nothing. The file contained:

/data # database

Git treats the comment as part of the pattern, so it was looking for a directory literally named data # database. The real database and secrets directories were never ignored. Comments go on their own lines now.

What I don’t have yet

The evaluation harness scores stored reports without new API calls. Fixtures carry expected material claims, acceptable verdict sets, minimum evidence requirements, and known missing context.

The target is 50 reviewed Reels, 10 each of directly false claims, real media in the wrong context, defensible-but-selective framing, health/scam/finance, and ambiguous developing stories.

The fixture set currently contains four.

Labels can’t come from Blindspot’s own reports; a reviewer has to establish them from primary sources, and anything political, communal, health, finance, or accusation-related needs a second reviewer. The comparison that matters is strong extraction plus lean research against all-lean, scored on material-claim recall, citation precision, source quality, verdict acceptability, confidence calibration, latency, cost, and unsafe overclaim rate. A confident unsafe error counts for more than an ordinary omission.

Blindspot is live, the pipeline runs end to end, the reports are shareable. What I don’t have is a representative error rate.

Four reviewed Reels can find bugs. They can’t establish trust.

Blindspot, Part 2: Extraction Sets the Ceiling

Mon, 03 Aug 2026 11:00:00 +0530

Part 2 of three on Blindspot. Part 1: reading the video · Part 3: cost, queues, and delivery.

I expected evidence retrieval to be the main quality problem. The first measurements pointed one stage earlier.

On a single Reel, a nano extraction model found 1 of 7 material claims. A stronger model found all seven, and flagged the central framing problem on top: the Reel presented a contested claim still under inquiry as conclusively established.

No amount of excellent research recovers a claim that never entered the evidence request.

Rule 1: a Reel is not a spoken article

The naive design treats a Reel as an article read aloud. Download, transcribe, fact-check the transcript.

But meaning is split across speech, on-screen text, the caption, charts visible for one second, the choice of footage, the publication date, and editing that changes which comparison the viewer sees.

A clip can be authentic while its caption supplies a false location. A number can be correct while the denominator is missing. One Reel can hold seven factual claims, two opinions, and a causal allegation no cited source establishes.

So extraction is one multimodal structured call over transcript, caption, post metadata, and keyframes together, returning strict JSON: checkable claims, on-screen OCR, named entities, events worth retrieving on, and observable framing indicators.

Parsed with JSON.parse, never regex. Claims and framing stay in separate fields. “The payment was ₹10 crore” is checkable. “This proves negligence” is a different kind of claim, extracted only if the Reel actually asserts it as fact.

Rule 2: decide what not to spend money on

Most Instagram posts don’t need a factual investigation. Music, aesthetic edits, jokes, vlogs, and pure political opinion should not be forced into a fact-check verdict.

A cheap triage stage runs after transcription and picks one of three routes:

A no checkable claim ──► deterministic "nothing to check" report
B opinion / framing ──► observable framing only, skip retrieval
C factual claims ──► extraction, retrieval, synthesis

Routes A and B are safety decisions before they are cost decisions. A system that always returns a decisive factual verdict will manufacture one for input that doesn’t support it.

Rule 3: retrieval never gets a verdict

Evidence is another provider seam: built-in web search, an external facts corpus of ~14,000 documents, or both. Blindspot batches every extracted signal into one request, and the service returns passages with provenance: source, URL, publication date, and which claim or keyframe they address.

It is explicitly not allowed to decide whether the Reel is true.

Blindspot Facts service
--------- -------------
read media, extract claims ───► retrieve relevant passages
send keyframes ───► match known images
synthesise report ◄─── return facts with provenance
assign verdicts never assigns a verdict

The point is debuggability. When a conclusion is wrong I can tell whether extraction missed the claim, retrieval returned weak evidence, or synthesis overreached. One end-to-end answer hides all three behind fluent prose.

Rule 4: tell the model what actually happened

An early report claimed Instagram could not be accessed. Plausible-sounding limitation. Completely false. The pipeline had downloaded the video, extracted audio, read keyframes, and produced a transcript.

What had actually happened was that web research found no corroboration. The model turned “search returned nothing” into “the media could not be fetched”, and filed non-evidence notes as sources.

Synthesis input now carries mediaAcquired and keyframeCount, and the prompt states plainly what the pipeline completed. Missing external evidence and missing media are different sentences. Evidence entries must contain URLs; a null search result is represented as no external source found.

Give the model operational state. Don’t expect it to infer operational state from incomplete context.

Verdicts are categories, not a score

No truth percentage. A single number implies precision the evidence doesn’t have and erases disagreement between claims inside one Reel. Outcomes are supported, mixed_evidence, misleading_context, unsupported, satire, not_yet_verifiable.

Every factual conclusion points at cited evidence, and confidence is shown separately from the verdict. A supported claim resting on indirect reporting is not the same object as one backed by a primary document.

The system is allowed to say it can’t verify something yet. Thin sourcing and developing events are normal input, not exceptions.

Next: what it costs, and the parts that break in production.

Blindspot, Part 1: Teaching a Pipeline to Watch the Whole Video

Mon, 03 Aug 2026 10:00:00 +0530

Part 1 of three on Blindspot. Part 2: extraction and the evidence boundary · Part 3: cost, queues, and delivery.

I fed a three-minute protest Reel into the first version. It downloaded the video, transcribed the speech, pulled frames, searched the web, and produced a polished report.

It had missed the police crackdown near the end.

Every frame sent to the vision model came from the fast-cut opening. The pipeline reported success. It had never watched the second half, and nothing in the output said so.

Blindspot turns a public Instagram Reel or post into a claim-level report with citations, missing context, and explicit confidence. It is an evidence-retrieval system, not a truth detector. This part covers the media stage, which is where that first failure lived.

Rule 1: never leave a section of the timeline unseen

The original sampler took the first N scene changes. On a fast-cut intro the frame budget was gone before the video reached anything substantive, so the model saw twelve variations of the same three seconds.

The fix was to stop looking for interesting frames and start guaranteeing coverage. ffprobe reports duration, FFmpeg samples evenly across the whole thing:

frame_count = clamp(round(duration_seconds / 14), 6, 16)

Roughly one frame per 14 seconds, floor of 6, ceiling of 16, scaled to 768px wide. Scene detection survives only as a fallback when duration probing fails.

The invariant changed from a heuristic about interestingness to a guarantee about coverage. Those fail very differently.

Rule 2: one piece of content has two identities

Blindspot keys its cache twice, because platform identity and content identity answer different questions.

Platform identity is instagram:<shortcode>. Exact post already analysed, return the report.

Content identity is a SHA-256 of the media plus a 64-bit difference hash. SHA-256 catches byte-identical files. The perceptual hash survives re-encoding, resizing, and watermarks, so it recognises reposts of the same footage.

On a media-cache hit, Blindspot reuses the transcript and the extracted visual claims. It re-analyses the caption, account, publication date, and framing every time.

That split is the whole product. Authentic footage under a new false caption is the main case Blindspot exists to catch, so reusing the previous verdict would defeat it. Reusing the transcript is free; reusing the conclusion is wrong.

There’s also a FORCE_REEXTRACT=1 escape hatch. Without it, changing the extraction model and rerunning the same footage just benchmarks the old cached claims.

Rule 3: every vendor is a seam

Local development runs whisper.cpp with Metal acceleration: free per run, a few seconds per Reel on Apple Silicon. Production prefers Azure Speech for Hindi-English code-mixed audio, with Bhashini dual ASR able to run both language tracks in parallel for comparison. Groq and OpenAI sit behind them.

All of it lives behind one transcription interface, selected by environment variable. Same for ingestion (yt-dlp, then Apify), LLM calls, and evidence retrieval.

This isn’t architectural taste. Instagram changes its access behaviour, speech vendors change models, rate limits move. Baking one vendor into the pipeline turns every provider problem into an application rewrite.

Rule 4: keep the derivatives, delete the source

FFmpeg pulls 16kHz mono PCM audio alongside the frames. Reels with no audio track are valid input and must not fail the run. Photo and carousel posts take a parallel path where the slides are the keyframes.

Once audio, keyframes, transcript, and hashes exist, the source video is deleted. Reports, transcripts, hashes, claims, citations, keyframes, and timings persist. The original media does not.

URL ──► ingest ──► ffprobe ──► even-interval keyframes ──► hashes
│ │
└──► 16kHz mono audio ──► transcribe ───┴──► cache

Cheap checks run before paid work, which is the only reason the cost numbers in Part 3 stay small.

Next: why extraction, not research, sets the ceiling on report quality.

Parakhi: Where Does Your ₹5 Actually Go?

Sat, 01 Aug 2026 00:00:00 +0000

Type “Parle-G” into parakhi.in and you get a ₹5 biscuit packet taken apart: wheat, sugar, palm oil, the packaging film, GST at 18%, the retailer’s cut, the slice that leaves the country. Each row carries a source and a confidence dot.

It works on 1002 products, 44 categories, and fuel across 12 states. Here is what it took to make those numbers defensible.

The rule that shaped everything

The model never touches a number.

That decision came first and everything else follows from it. An LLM can read a messy search box, guess which category a product belongs to, and normalise a brand name. It cannot be allowed to say “12% of this is imported”, because it will say it confidently, differently each time, with nothing behind it.

So the arithmetic lives in computeBaseline(), a pure function with no network calls in it. Same product in, same breakdown out, every time, forever.

 free text ─┐
barcode ──┼─► resolve BrandIndex fuzzy match
url ─┘ (LLM only as fallback)
│
▼
categorize keyword + HSN rules ──► one of 44
│ hand-written category templates
▼
┌──────────────────────┐
│ computeBaseline() │ pure function, no LLM
└──────────┬───────────┘
│
│ category template composition
│ + CBIC HSN→GST table (56 rows)
│ + Wikidata brand→parent→country
│ + Agmarknet daily mandi prices
│ + Open Food Facts label (confirms only)
▼
Postgres cache ──► ISR page

The 56-row GST table is typed by hand from CBIC notifications. The 44 category templates are each a couple of hours of reading trade-body reports. That work doesn’t scale and that is the point: it’s the part a model can’t do for you.

Every number renders with a source tier and a confidence dot, so a figure backed by a government notification looks different on the page from one backed by an industry estimate. You can see which is which before you decide to believe it.

Problem: a label source that could delete an ingredient

Open Food Facts has real ingredient lists for a chunk of the catalog, which is better data than any template. The first version let the label override the template composition.

Frooti broke it. Its mango pulp row didn’t fuzzy-match anything on the label, so it got dropped. The composition collapsed to sugar and water, both fully Indian, and the score shot up to 99%. The pipeline had made a product look better by deleting its main ingredient.

The fix is a one-way door. A label can add a “✓ on label” confirmation to a component. It can never remove one. On top of that, a generic-token blocklist, because “oil”, “powder” and “flavour” happily match across materials that have nothing to do with each other.

Problem: a rate limiter that limited nothing

Per-IP counts lived in an in-memory Map. On Vercel each lambda gets its own memory, so the counter reset more or less at random.

The global daily cap had a worse bug: it counted rows in the LlmCall table, and the deterministic path never makes an LLM call. That counter sat at zero permanently. Anyone could hammer /api/search and fill the database with junk products for free.

Now there’s a RateHit table in Postgres with an upsert-increment keyed on (scope, window). 10 new analyses per IP per hour, 500 site-wide per day, and the limit applies only on a cache miss. Looking at a product someone already analysed costs nothing and is never throttled.

Problem: migrations through a connection pooler

prisma migrate deploy ran during the Vercel build, so schema changes shipped with the code that needed them. Neat, and it does not work. migrate deploy can’t hold its advisory lock over Neon’s pgbouncer pooler, so every production deploy died on a P1002 timeout. One crashed run left lock 72707369 held and blocked every migration after it. Migrating mid-build also kills the build’s own connection: terminating connection due to administrator command.

Migrations now run in a GitHub Actions workflow over a direct, non-pooled connection, and they clear any stale backend sitting on the lock before starting. Deploys got boring again.

Problem: prerendering 1000 pages through a pool of 3

For SEO, generateStaticParams returned every product slug. Prisma’s pool limit is 3 with a 10-second timeout. Builds crawled, then died around page 280, taking half an hour to fail.

Hero products are prerendered now and everything else is on-demand ISR, cached in Postgres after the first hit. Builds finish in seconds, every page is still server-rendered and indexable, and the sitemap still lists all 1006 URLs.

What’s live

1002 products across 44 categories. 56 HSN→GST rows, ~140 brand-to-country mappings, 25 fuel buildup rows covering petrol and diesel in 12 states plus national LPG. Daily mandi prices from Agmarknet feed the agricultural inputs.

The calibration harness runs against published anchors and currently reports GST exactness at 100%, label match at 84%, and every anchor inside its expected range.

Cost per query is about $0.0001, with a hard $20/month circuit breaker that has never come close to tripping. The bill stays near zero because the expensive part, the arithmetic, doesn’t call anyone.

Try it on something in your kitchen: parakhi.in.

ScamDB: The App Took a Weekend, The Data Took Three Months

Wed, 29 Jul 2026 00:00:00 +0000

Someone in my family almost sent money to a “customs officer” who needed a clearance fee. The number was on WhatsApp. I googled it. Nothing. Two forums, one SEO farm, an answer from 2019.

That’s the whole gap. Every scam number in India has already been reported by someone. It just isn’t searchable.

So I built scamdb.in. Type a number or a UPI ID, see whether anyone has reported it.

The app was a weekend. Next.js 16, Supabase, Google OAuth, moderation queue, SEO-friendly URLs like /phone/9876543210 so Google can actually index them. Boring, done, deployed.

Then I opened it and it had 34 entries. A search engine with 34 rows is not a search engine. It’s a spreadsheet with a domain name.

The next three months were entirely about data.

The pipeline I thought I was building

 r/indianscammers
│
▼
fetch posts ──► regex the number out ──► moderation ──► live

Two hours of work. Ship it.

What actually happened

~90% of posts came back no_entities.

Not because the posts had no numbers. Because the number is in the screenshot.

Nobody types “here is the number that scammed me: 98765 43210.” They post a WhatsApp screenshot. The number is 40 pixels tall, sitting in a green bubble, completely invisible to every regex I own.

Text scraping was never going to work here. OCR wasn’t an optimization I could add later. It was the product.

Reddit fought back on every front

Before I could OCR anything I had to actually get the posts. This took an embarrassing number of attempts:

Anonymous .json endpoints: 403.
Browser-ish requests: you get the HTML webapp, not data.
Move it to GitHub Actions so it runs on a schedule: datacenter IPs get rate-limited harder than my laptop did.
RSS: works! Serves only new and hot. There is no historical anything.
X/Twitter API free tier: 402. It’s write-only now. Reading costs $175/month.

The unblock was arctic-shift, the Pushshift successor. Full Reddit archive, no auth, no rate limits, doesn’t care that I’m a datacenter, and it hands back direct full-res i.redd.it URLs.

First test: two screenshots in, nine numbers out. I stopped worrying about whether this would work.

The actual pipeline

 arctic-shift archive
│ paginate by created_utc
▼
image posts only ──► i.redd.it full-res URL
│
▼
┌──────────────┐
│ vision OCR │ Haiku, falls back to gpt-4o-mini
│ │ model output regex-validated, always
└──────┬───────┘
│
▼
raw_signals ──► deterministic extract ──► LLM enrich ──► moderation ──► live

Every OCR result gets re-validated by regex on the way out. Vision models are very happy to hand you a number that is almost the number in the image, and it looks fine until you dial it. So the transcription doesn’t count for anything on its own. Only what survives the pattern goes in.

Bugs that cost me real weekends

Bug one: the crawler ran forever and made no progress.

Each run pulled the same images, OCR’d them, found nothing, threw them away, and did it again six hours later. The dedup check only knew about images that had produced something. Dead images stayed permanently fresh.

Fix: write a “tried this, got nothing” marker. You have to record the failures too, or you just keep paying for them.

Bug two: OpenAI’s 200K TPM limit.

Images are expensive in tokens. Bulk OCR hit the rate limit within a minute and spent the rest of the run 429-backing-off into a crawl. The correct fix was to pay for more throughput. I chose to let it accumulate slowly for free instead, which is a fix in the same way that “walking” is a fix for a broken car.

Bug three: my regex was importing America.

The phone pattern matched the last 10 digits of any number. +1 863..., +977..., all of it, cheerfully normalized into Indian mobile numbers. A US business had quietly become a suspicious Indian phone number in my database.

Fix: a (?<!\d) lookbehind so the match can’t start mid-number, and a hard reject on any + prefix that isn’t +91.

Bug four: the Twitter crawler, where every single entity was a false positive.

Not most. Every one. Flight numbers, order IDs, prices, dates. The signal-to-noise was so bad that the honest fix was git rm. That commit message is still my favourite in the repo.

Where it is now

1,124 entities. 1,115 phones, 9 UPI IDs, 2,798 raw signals collected, 525 reports approved.

The 9 UPI IDs are not a bug. People screenshot the chat, not the payment page.

Once the archive was swept clean, new posts dropped to single digits per week. That’s correct, and also boring, so I added two more sources.

Reddit comments: about 4% carry a number, pure text, zero vision cost. Every so often someone drops a bulk “here’s the list” comment, which is a very good day.

Public aggregator pages: stored at low confidence, because they arrive with no story attached and I don’t know who put them there.

Arctic-shift also 422s at random with “under maintenance.” Treat it exactly like a 429, retry, move on.

The honest part

999 reports are sitting in the moderation queue, unreviewed.

I built the auto-publish safeguard, wrote the whole “every report needs a human” rule, then became the human. There has been no moderation action since June 7. The bottleneck in my carefully designed pipeline is a guy who also has a job.

The other thing I’d say is that none of the interesting engineering was in the search. Search is a Postgres index. The interesting part was that the data had been sitting in public the entire time, in a format no computer could read.

The number that started this is in there now. Third search result.

Brainrot: The Dumbest Video Format Taught Me the Most About Debugging

Tue, 07 Jul 2026 00:00:00 +0000

You’ve seen the format. Reddit story, robotic narrator, Minecraft parkour underneath, captions bouncing word by word. It floods Shorts because it works, and because someone, somewhere, is paying editors per video to make it.

I automated the whole thing. Local TTS, free. Total cost per video: whatever one LLM call costs. Pennies.

The pipeline

 mood ("funny")
│
▼
┌─────────────┐
│ LLM writes │ 8-20s micro-story
│ the script │ two synced versions (spoken + display)
└──────┬──────┘
│
▼
┌─────────────┐
│ Kokoro │ 82M param TTS model
│ narrates │ Apache 2.0, runs on CPU, costs nothing
└──────┬──────┘
│
▼
┌─────────────┐
│ Whisper │ word-level timestamps
│ times it │ from the generated audio
└──────┬──────┘
│
▼
┌─────────────┐
│ libass │ karaoke captions
│ burns it │ word-by-word highlight
└──────┬──────┘
│
▼
┌─────────────┐
│ FFmpeg │ random gameplay slices
│ cuts it │ exact audio length, 1080x1920
└─────────────┘

The one design decision that actually mattered

Every story ships as two scripts.

A spoken script where “$13.60” becomes “thirteen dollars and sixty cents”, because that’s what the narrator says. And a display script where it stays “$13.60”, because that’s what belongs on screen.

Same sentences, same order, different formatting. Skip this and your captions drift the moment a number shows up.

The alignment trick

Trust Whisper’s ears, never its spelling.

Whisper transcribed my narrator saying “al dente” as “al dent.” Doesn’t matter. I already know the script, I wrote it. So the transcript text gets thrown away entirely and only the timestamps survive. Those timestamps get matched back onto the real script with a diff.

Sync stays word-perfect even when Whisper mishears.

That was the fun part. Then the machine fought back.

The bugs that ate my weekend

Bug one: the pipeline froze at 0% CPU.

Not slow. Parked. PyTorch (for the TTS) and CTranslate2 (for Whisper) each ship their own OpenMP runtime. When both load into one process, their thread pools deadlock. The famous KMP_DUPLICATE_LIB_OK=TRUE workaround only silences the warning. macOS’s sample tool showed the main thread suspended in __kmp_suspend_64, waiting for a signal that never comes.

Fix: force both libraries single-threaded before either one imports. Slower per step, but slow beats never.

Bug two: models I already downloaded still phoned home.

Hugging Face checks for updates on every load, and on my network that check just hung. HF_HUB_OFFLINE=1, gone.

Bug three: Telegram delivery hung past its own timeout.

The request had timeout=120. It hung anyway. Turns out my network’s IPv6 route to api.telegram.org is dead. TCP sits in SYN_SENT at the OS level, below where Python’s timeout lives. curl -6: hangs. curl -4: instant. One monkeypatch to force IPv4 and delivery worked forever after.

None of these bugs were in my code. All of them ate hours. That’s the real cost of “local-first”: you inherit every layer under you.

One more: emoji in burned-in captions render as tofu boxes, because libass has no color-emoji fallback. They get stripped from the video and live on in the post caption instead. Nobody misses them.

The honest part

Renders take a couple of minutes each on CPU. The free LLM tier caps you around 20 videos a day. And the gameplay footage licensing is on you, “found it on YouTube” is not a license.

Also, this format is junk food. I know. That’s why the interesting parts are underneath it.

Code’s open, same repo as ReelGen: github.com/rohitsaini1196/ReelGen

ReelGen: Stop Scrolling Stock Footage, Start Shipping Reels

Mon, 06 Jul 2026 00:00:00 +0000

Every AI-generated Reel looks the same. Same overused Pexels clips, same robotic zoom, same text that screams “a bot made this.”

Got tired of it. Built something that doesn’t.

Topic in, finished reel out. Color-graded, captioned, ready to post. About a penny per run.

How it works

 "Hidden cafes in Tokyo"
│
▼
┌──────────────┐
│ LLM Plans │ hook, beats, closer
│ the Story │ mood arc, color palette
└──────┬───────┘ caption + hashtags
│
▼
┌──────────────┐
│ CLIP Finds │ semantic match
│ the Footage │ not keyword luck
└──────┬───────┘
│
▼
┌──────────────┐
│ FFmpeg Does │ color grading
│ the Grading │ Ken Burns + handheld jitter
└──────┬───────┘
│
▼
┌──────────────┐
│ Pillow Adds │ animated text
│ the Text │ mood-matched font + style
└──────┬───────┘
│
▼
reel.mp4
1080×1920, IG-ready
~$0.01

The actual problem with AI video tools

Zero taste.

Keyword search grabs the first “beach sunset” it finds. Warm or cold, sharp or mushy, doesn’t matter. No color consistency across clips. No pacing, either hard cuts everywhere or dissolves everywhere, never a mix that feels human-edited.

Fixing that turned out to be a plumbing problem, not a model-size problem.

What I did differently

┌─────────────────────────────────────────────────┐
│ │
│ TWO HOOKS, ONE KEEPS │
│ LLM writes two opening angles. │
│ Whichever finds better footage wins. │
│ │
├─────────────────────────────────────────────────┤
│ │
│ SCORE EVERYTHING, FOUR WAYS │
│ │
│ Every candidate clip gets ranked on: │
│ │
│ ✓ Semantic match (CLIP) │
│ ✓ Sharpness │
│ ✓ Palette fit │
│ ✓ Motion stability │
│ │
│ Pick the winner BEFORE settling. │
│ │
├─────────────────────────────────────────────────┤
│ │
│ WEAK MATCH? TRY AGAIN, DIFFERENTLY │
│ │
│ Abstract caption like "you hit your limit" │
│ has no literal visual. LLM rewrites it to │
│ something more concrete. If footage still │
│ can't carry it, text gets a stronger visual │
│ anchor automatically. │
│ │
└─────────────────────────────────────────────────┘

None of that needed a bigger model. Just better plumbing.

The honest part

Free-tier LLM quotas cap out fast. I burned a full day’s limit just testing this thing.

And good editing doesn’t guarantee views. Posting cadence and trending audio still matter more than the cut. This makes reels you won’t be embarrassed to post. It doesn’t make them go viral for you.

Try it

python reelgen.py generate \
 --topic "Hidden cafes in Tokyo's backstreets" \
 --mood calm \
 --style warm-analog

Everything runs locally. No cloud rendering, no subscription.

Code’s open: github.com/rohitsaini1196/ReelGen

The Claude Code Tabs You Forget About

Sat, 23 May 2026 00:00:00 +0000

If you run more than a couple of Claude Code sessions at once, you know the feeling. You kick off work in one tab, switch to another while it thinks, then a third, and twenty minutes later you’ve lost track of which one finished and is sitting there, quietly waiting on your reply.

I kept losing those tabs. So I built claude-session-tracker, a small, local dashboard that surfaces the sessions waiting on you or gone silent.

Why not just use Agent View?

Claude Code ships with Agent View (claude agents), and it’s good, but it tracks background sessions, the ones you launch with --bg. It doesn’t see the normal interactive tabs you open in VS Code and forget about. Those are exactly the ones I lose.

So this fills the gap: it watches the regular sessions, not the dispatched ones.

The mechanism: the file is the truth

The whole thing rests on one decision: a local file is the source of truth, and the server only reads it.

Three Claude Code hooks each append a single JSON line to ~/.claude-sessions/events.jsonl:

SessionStart: a session began
Stop: Claude finished its turn, meaning it’s now waiting on you
PostToolUse: a liveness pulse to show the session is actively working

That’s it. The hooks do nothing but append one line, locally, with no network calls. The dashboard server reads that file and derives the state.

The payoff is that the server is completely disposable. If it’s off, the hooks keep writing and it catches up on the next read. Zero data loss, zero added latency to your turns. A network call inside a hook would have meant silent data loss whenever the server was down, which wasn’t a trade-off I was willing to make.

It’s also privacy-minimized at the source: the hooks write only a session id, the working directory, an event type, and a timestamp. Never your prompts, your code, or Claude’s replies. It’s safe to point at a private repo.

Honest about what it can’t know

The status isn’t binary, and that’s deliberate. A session “waiting on you” is reliable, because Stop fires at the true end of a turn. But a session that’s gone quiet? It might be stalled, or it might just be busy inside one slow tool. You genuinely can’t tell those apart from the outside.

So instead of guessing, the dashboard grades its confidence: possibly stuck after a short quiet period, likely stalled after a longer one. It tells you what it actually knows, and admits what it doesn’t.

The nice touches

Once the data was there, a few things made it feel good to use. You can click a row to focus that specific VS Code window (only if it’s already open, so it never spawns a new one), see a 30-minute activity sparkline per session, and check a rough “engaged time” to see where your attention actually went.

It’s open source, MIT, and runs on the Python standard library, meaning there’s absolutely nothing to pip install.

git clone https://github.com/rohitsaini1196/claude-session-tracker
cd claude-session-tracker
./install.sh # wires the hooks
./run.sh # dashboard at localhost:8787

The dashboard is for the tabs you’d otherwise lose. Everything else can wait.

Meta-Dev-CLI: Stop Clicking, Start Typing

Thu, 02 Apr 2026 00:00:00 +0000

If you’ve ever had to work with Meta’s Developer Dashboard to manage WhatsApp Business Accounts (WABA), you know the “click-fest” I’m talking about.

Finding a specific Phone Number ID, grabbing a temporary access token, or just sending a simple test message involves navigating through multiple tabs, models, and nested menus. It’s a context-switch nightmare.

That’s why I built meta-dev-cli.

The “Stay in the Terminal” Philosophy

Developers (at least the ones I like talking to) want to stay in their flow state. That means staying in VS Code or the terminal.

The goal of this CLI is simple: Whatever you can do in the dashboard, you should be able to do with one command.

Want to send a message?

meta wa send +1234567890 "Hey, it works! 🚀"

Need to check your WABA status or phone numbers?

meta wa phone-numbers

It’s fast, it’s low-friction, and it doesn’t require a browser.

Agent-First Design

One of the most important features of meta-dev-cli is that it wasn’t just built for humans. I built it with AI agents in mind—like Cursor, Claude Code, or GitHub Copilot.

Every single command supports a --json flag.

When you run meta apps list --json, you get a structured payload that an LLM can actually parse and act upon. This makes it incredibly easy to “hire” an agent to manage your Meta apps, automate deployments, or even build a custom monitoring dashboard.

Why does this matter?

Tools shouldn’t just provide functionality; they should reduce cognitive load.

By pulling the Meta ecosystem into the terminal, we’re not just saving a few clicks—we’re making the entire development lifecycle more predictable and automatable.

If you’re working with the WhatsApp Cloud API, give it a spin: pip install -e . (from the repo)

Let’s keep the dashboard for settings we change once a year, and use the terminal for everything else.

Live Kitchen Feeds: Transparency Without the Cloud Bill

Tue, 31 Mar 2026 00:00:00 +0000

For the last 3-4 years, most of my meals have come from apps like Zomato and Swiggy. And honestly, I’m pretty sure that food is often prepared in ways I probably wouldn’t want to see.

That got me thinking: What if customers could watch their food being cooked live? Not some polished marketing B-roll. Just a raw, live feed from the kitchen station while your order is actively being prepped.

Transparency builds trust. Simple as that.

The immediate pushback to an idea like this is always the infrastructure cost. If food delivery giants tried to stream thousands of kitchens using traditional cloud video architecture, their AWS bill would look like a phone number.

But with today’s tech, it doesn’t have to be that way.

The Low-Cost Architecture

Instead of routing all that dense video data through centralized servers, we can just use peer-to-peer streaming via WebRTC.

The setup is fairly basic: You mount a cheap Android phone or a Raspberry Pi near the prep counter. It wakes up and automatically starts streaming only when an order drops.

On the backend, you just need a lightweight signaling server (maybe Node.js or Go with WebSockets). This server’s only job is to introduce the customer’s phone to the kitchen’s camera so they can shake hands.

Once they connect, it establishes a direct P2P link. That means the heavy video traffic never actually touches expensive cloud servers. You’re mostly just paying a few bucks a month for the signaling server and a STUN/TURN fallback.

What about millions of orders?

Scaling this is actually surprisingly chill. Since the heavy lifting, the video streaming, happens at the edge (between the two phones), your central servers are barely doing anything.

A standard WebSocket signaling server can juggle tens of thousands of concurrent connections easily because it’s just passing tiny text tokens (SDPs) back and forth. To handle millions of daily orders, you just scale that signaling layer horizontally using something like Redis Pub/Sub. The cloud infrastructure cost barely flinches, even during the Friday night dinner rush.

Handling the Dinner Rush

Okay, but what happens when a single cook is firing five different orders at the exact same time? You can’t force a budget kitchen tablet on a greasy Wi-Fi network to upload five separate video streams. It would instantly choke.

The fix here is station-based multicasting.

Instead of opening a 1:1 stream for every single order, the kitchen device captures one continuous video feed of the entire prep station. If five active orders are sitting there, the device pushes that single stream to a lightweight SFU (Selective Forwarding Unit), which then mirrors it out to all five customers.

But how do you know which burger is yours?

This is where it gets fun. The kitchen app can run basic on-device object tracking or OCR to read the physical ticket numbers sitting on the counter. It then draws a simple AR overlay, like a glowing digital tag hovering over the plate reading “Order #1234”, directly onto the feed. You watch the wider station workflow, but you immediately spot your meal moving down the line.

Why go through the trouble?

Beyond basic transparency, just having a camera on the line unlocks a lot of interesting data. You could automatically generate time-lapse cooking reels for social media, run lightweight AI models for hygiene detection, or even give restaurants a true, verifiable “Quality Score” based on how they actually handle the food.

Food delivery crushed the convenience problem years ago. But it still completely lacks transparency. With WebRTC and dirt-cheap hardware, letting users watch their food being prepped might just be the most effective trust-building feature we could ship right now.

And surprisingly, it wouldn’t even cost a fortune to build.

Designing for Scale

Thu, 12 Mar 2026 00:00:00 +0000

Modern systems rarely fail because of a single large mistake. They fail because of small assumptions that compound.

A field that’s always populated — until it isn’t. A service that responds in 50ms — until traffic doubles. A queue that never backs up — until a downstream dependency has a bad day.

When designing for scale, the first instinct is usually to reach for more infrastructure. More caches. More queues. More services. More replicas.

That instinct is wrong about 80% of the time.

Scale Is About Clarity

Over the last few years, I’ve come to believe that scale is mostly about clarity. Not clever algorithms. Not exotic databases. Just clarity about four things.

1. Clarity of Data Flow

If you can’t draw how data moves through your system on a napkin, you’re going to have a bad time scaling it.

Here’s a real scenario. We had a service that called three downstream APIs sequentially:

Request → Service A → Service B → Service C → Response

Total latency: sum of all three. At low traffic, this was fine — maybe 200ms total. At high traffic, one slow response from Service C would back up everything upstream.

The fix wasn’t adding more instances of our service. It was restructuring the data flow so that we only needed Service A’s response synchronously. Services B and C could be handled asynchronously via events:

Request → Service A → Response (sync, fast)
↓
Kafka Event
↓ ↓
Service B Service C (async, eventually consistent)

Same outcome. Dramatically different scaling characteristics.

2. Clarity of Ownership

One of the fastest ways to create scaling problems is shared ownership. Two teams writing to the same database table. Three services all maintaining their own copy of user preferences. Five consumers all computing the same aggregation differently.

I follow a simple rule: every piece of data has exactly one owner. The owner is the only service that writes to it. Everyone else reads, caches, or subscribes to change events.

This sounds obvious, but in practice, it’s the first thing that gets violated when teams are moving fast. “We’ll just add a column to their table, it’s faster.” Three months later, you have two services with competing opinions about what a user’s status should be.

3. Clarity of Failure Handling

At small scale, you can get away with “it probably won’t fail.” At large scale, everything that can fail will fail, and probably all at the same time on Friday evening.

The question isn’t “will this fail?” but “when this fails, what happens?”

Some patterns that have saved me:

Circuit breakers on every external call. When a dependency goes down, fail fast instead of queueing up thousands of requests.
Timeouts everywhere. Not just HTTP timeouts. Database query timeouts. Kafka consumer processing timeouts. Redis connection timeouts. I’ve seen systems grind to a halt because a single missing timeout let a slow query hold a connection pool hostage.
Dead-letter queues for every consumer. When a message can’t be processed after N retries, park it somewhere you can inspect later instead of blocking the entire queue.

# A timeout that actually works
import signal

class TimeoutError(Exception):
 pass

def timeout_handler(signum, frame):
 raise TimeoutError("Operation timed out")

def with_timeout(fn, seconds=5):
 signal.signal(signal.SIGALRM, timeout_handler)
 signal.alarm(seconds)
 try:
 return fn()
 finally:
 signal.alarm(0)

4. Clarity of Observability

You can’t scale what you can’t see.

I’ve worked on systems where the only “monitoring” was someone checking if the API returned 200. Green meant good, right? Wrong. The API was returning 200 while silently dropping 30% of messages because a Kafka producer was failing and the error was being swallowed.

Good observability at scale means:

Metrics that matter. Not just CPU and memory. Business metrics. Messages processed per second. P99 latency per endpoint. Queue depth over time. Error rate by type.
Structured logging. Key-value pairs, not strings. When you’re searching through millions of log lines, {"thread_id": "abc", "action": "send", "latency_ms": 342} is infinitely more useful than Sent message to thread abc in 342ms.
Traces that connect the dots. When a request touches five services, a correlation ID that flows through all of them is the difference between “something is slow” and “the Elasticsearch query in Service C is slow because the index isn’t optimized for this access pattern.”

The Trap of Premature Scaling

The biggest trap I see: designing for scale you don’t have yet. Adding Kafka when you have 100 requests per minute. Splitting into microservices when your team is four people. Introducing a distributed cache before profiling where the actual bottleneck is.

Every scaling mechanism you add is also a complexity mechanism. A cache means you now need cache invalidation strategy. A message queue means you need idempotency guarantees. Microservices mean you need service discovery, distributed tracing, and a deployment pipeline that doesn’t take 45 minutes.

Start simple. Measure. Scale the parts that are actually slow.

The One Rule

If I had to condense everything into one rule, it would be this:

The simplest architecture that clearly defines boundaries will almost always outperform a complex one that doesn’t.

Scaling is less about adding components and more about removing ambiguity. When every engineer on the team can look at the system and immediately understand where data flows, who owns what, how failures are handled, and where to look when things go wrong — that system is ready to scale.

Not because the infrastructure is impressive. Because the thinking is clear.

Event-Driven Systems Are Not Magic

Fri, 20 Feb 2026 00:00:00 +0000

There’s a moment in every growing system’s life where someone says, “We should make this event-driven.” And most of the time, they’re right. Decoupling producers from consumers is genuinely powerful. It lets teams work independently, services scale independently, and the overall system becomes more resilient.

But here’s what nobody mentions in the conference talks: events don’t eliminate complexity. They redistribute it.

And if you’re not prepared for where that complexity lands, you’re going to have a worse time than you did with synchronous calls.

What Event-Driven Actually Gives You

Let me start with the good parts, because there are genuinely good parts.

Say you have a system where creating a new user account needs to:

Store the user in the database
Send a welcome email
Create a default workspace
Notify the analytics service

The synchronous version looks like this:

def create_user(request):
 user = db.create_user(request.data)
 email_service.send_welcome(user.email)
 workspace_service.create_default(user.id)
 analytics.track("user_created", user.id)
 return user

This is simple to read and simple to debug. But it has problems. If the email service is slow, the API response is slow. If the analytics service is down, user creation fails. Every dependency is a single point of failure.

The event-driven version:

def create_user(request):
 user = db.create_user(request.data)
 event_bus.publish("user.created", {"user_id": user.id, "email": user.email})
 return user

# Separate consumers, running independently
def on_user_created_send_email(event):
 email_service.send_welcome(event["email"])

def on_user_created_setup_workspace(event):
 workspace_service.create_default(event["user_id"])

def on_user_created_track_analytics(event):
 analytics.track("user_created", event["user_id"])

Now the API is fast — it does one thing and publishes an event. Each side effect runs independently. If the analytics service is having a bad day, users still get created and emails still go out.

This is the pitch. And it’s a real improvement.

But now let’s talk about what comes next.

Problem 1: Ordering

Events in a queue aren’t guaranteed to arrive in the order you think.

Say a user creates an account and immediately updates their profile. You now have two events:

Event 1: user.created { user_id: 42, name: "Rohit" }
Event 2: user.updated { user_id: 42, name: "Rohit S." }

If both events land on different partitions, or if Consumer A processes events faster than Consumer B, you might process the updated event before the created event. Now your downstream service is trying to update a user that doesn’t exist yet.

In Kafka, you can solve this by using the user_id as the partition key. All events for the same user land on the same partition and are processed in order. But this only works within a single topic. Across topics? You’re on your own.

# Producing with a partition key
producer.send(
 topic="user-events",
 key=str(user_id).encode(), # same user → same partition → ordered
 value=serialize(event)
)

There’s no universal ordering across an event-driven system. You need to design for partial ordering and be very explicit about which ordering guarantees you actually need.

Problem 2: Idempotency

Events will be delivered more than once. This isn’t a maybe — it’s a guarantee. Network blips, consumer restarts, rebalances — all of these cause redelivery.

If your consumer does this:

def on_payment_received(event):
 db.add_credit(event["user_id"], event["amount"])

And the event gets delivered twice, the user just got double credit. Congratulations, you now have a financial accounting bug.

The fix is making every consumer idempotent. The simplest way is to track which events you’ve already processed:

def on_payment_received(event):
 if db.event_already_processed(event["event_id"]):
 return # skip duplicate

 db.add_credit(event["user_id"], event["amount"])
 db.mark_event_processed(event["event_id"])

This works, but now you have an extra database table, extra queries on every event, and you need to think about cleanup (that table grows forever). For simpler cases, you can use natural idempotency — operations like SET status = 'active' are inherently idempotent, while INCREMENT balance BY 100 is not.

Problem 3: Schema Evolution

Your event schema will change. New fields get added. Old fields get deprecated. The structure of the payload evolves as the product evolves.

But here’s the catch: producers and consumers are deployed independently. When you add a new field to the user.created event, there’s a window where the producer is sending the new schema but the consumer is still running the old code.

# Version 1 of the event
{"user_id": 42, "email": "rohit@example.com"}

# Version 2, six months later
{"user_id": 42, "email": "rohit@example.com", "plan": "premium", "source": "referral"}

Consumers need to handle both versions gracefully. This means:

Always use optional fields for new data. Never require a field that didn’t exist in v1.
Never remove or rename fields. Add new ones, deprecate old ones.
Version your events. Include a schema_version field so consumers can branch on it if needed.

Some teams use schema registries (like Confluent Schema Registry for Kafka) to enforce compatibility. That adds infrastructure but catches breaking changes before they hit production.

Problem 4: Debugging Across Services

This is the one that hurts the most. In a monolith, debugging is straightforward. You add a breakpoint, step through the code, and follow the logic. In an event-driven system, the “logic” spans multiple services, multiple queues, and multiple timelines.

A user reports: “I created an account but never got my welcome email.”

In the monolith: check create_user, find the bug. Ten minutes.

In the event-driven system:

Did the user.created event get published? Check producer logs.
Did Kafka receive it? Check the topic.
Did the email consumer pick it up? Check consumer group lag.
Did the consumer process it successfully? Check consumer logs.
Did the email service actually send the email? Check that service’s logs.

You’re now debugging across five different log streams, and you need a correlation ID to tie them all together.

# Always propagate a correlation ID through your events
event = {
 "event_id": str(uuid4()),
 "correlation_id": request.headers.get("X-Correlation-ID", str(uuid4())),
 "user_id": user.id,
 "email": user.email,
 "timestamp": datetime.utcnow().isoformat()
}

If you don’t set this up from day one, you’ll wish you had by month two.

So When Should You Use Events?

Events are great when:

The producer doesn’t need to know the result of the consumer’s work
You have multiple independent reactions to the same trigger
You need resilience — consumers can retry independently without blocking each other
Workloads are bursty and you need a buffer

Events are overkill when:

You have a simple request-response flow
You need strong consistency and the result immediately
Your system is small enough that the operational overhead of queues outweighs the benefit
You only have one consumer (just call it directly)

The Real Work

Using events isn’t the hard part. Anyone can publish a message to Kafka. The real work is everything that comes after:

Designing for out-of-order delivery
Making every consumer idempotent
Evolving schemas without breaking things
Building observability that spans service boundaries

The first message you publish is satisfying. The hundredth production bug you debug across an event-driven pipeline is humbling.

Events are a tool. A powerful one. But they’re not magic. Treat them with the respect they deserve, and they’ll serve you well. Treat them as a silver bullet, and they’ll redistribute your complexity to places you weren’t looking.

Debugging Distributed Systems

Thu, 05 Feb 2026 00:00:00 +0000

In a monolith, debugging is straightforward. Something breaks, you get a stack trace, you find the line, you fix it. The entire execution happened in one process, on one machine, in one timeline.

Distributed systems blow all of that up. The “stack trace” is scattered across five services, three message queues, and two databases. The bug might not even be a bug — it might be a timing issue that only happens under specific load patterns on the third Tuesday of the month.

I’ve spent enough time debugging distributed systems that I’ve developed a set of habits and patterns that consistently save me time. None of them are groundbreaking. They’re just discipline applied before things break.

Correlation IDs: Your Lifeline

The single most valuable piece of infrastructure in any distributed system is a correlation ID. It’s a unique identifier generated at the entry point of a request that flows through every service, every queue, every log line.

Without it, debugging looks like this:

Service A logs: "Processed request for user 42"
Service B logs: "Received event for user 42"
Service C logs: "Sent notification for user 42"

Seems traceable, right? Now imagine 500 requests per second for user 42. Which log line in Service B corresponds to which log line in Service A?

With a correlation ID:

Service A: {"correlation_id": "abc-123", "action": "process_request", "user_id": 42}
Service B: {"correlation_id": "abc-123", "action": "handle_event", "user_id": 42}
Service C: {"correlation_id": "abc-123", "action": "send_notification", "user_id": 42}

Now you can grep for abc-123 across all your services and reconstruct the exact path of a single request. This is non-negotiable. If you don’t have this, add it today.

# Middleware that generates or propagates correlation IDs
def correlation_middleware(request, next_handler):
 correlation_id = request.headers.get(
 "X-Correlation-ID",
 str(uuid4())
 )
 # Attach to thread-local context for logging
 context.correlation_id = correlation_id

 response = next_handler(request)
 response.headers["X-Correlation-ID"] = correlation_id
 return response

The key rule: every time you publish a message, make an HTTP call, or write a log line, include the correlation ID. It should propagate like a virus through your entire system.

Structured Logging, Not String Logging

When you’re debugging a monolith with 100 log lines, printf debugging works fine. When you’re debugging a distributed system with 10 million log lines, you need structured logging.

The difference:

# Bad: stringly-typed and impossible to query
logger.info(f"Processing message {msg_id} for thread {thread_id}, took {latency}ms")

# Good: structured, filterable, aggregatable
logger.info("message_processed", extra={
 "message_id": msg_id,
 "thread_id": thread_id,
 "latency_ms": latency,
 "consumer_group": "notification-sender",
 "partition": partition,
 "offset": offset,
})

With structured logs, you can do things like:

Find all messages where latency_ms > 5000 across all services
Filter by consumer_group to see only one consumer’s perspective
Aggregate latency_ms by thread_id to find hot threads
Correlate high latency with specific Kafka partitions

This turns debugging from “reading log files” into “querying a database.” Most log aggregation tools (Datadog, Kibana, CloudWatch) work dramatically better with structured data.

Think in Timelines, Not Call Stacks

This is the biggest mental shift. In a monolith, execution is vertical — function A calls function B calls function C. You read a stack trace top to bottom.

In a distributed system, execution is horizontal — things happen across time on different machines. The order matters, the timing matters, and the gaps between events often tell you more than the events themselves.

When debugging, I build a timeline:

10:00:00.000 Service A Received request correlation_id=abc-123
10:00:00.015 Service A Published event to Kafka correlation_id=abc-123
10:00:00.015 Service A Returned 200 to client correlation_id=abc-123
10:00:02.340 Service B Consumed event correlation_id=abc-123
^^^ 2.3 second gap — why? Consumer lag? Slow deserialization?
10:00:02.380 Service B Called Service C correlation_id=abc-123
10:00:07.380 Service B Timeout from Service C correlation_id=abc-123
^^^ 5 second timeout — Service C is the problem
10:00:07.400 Service B Sent to dead-letter queue correlation_id=abc-123

The timeline immediately reveals the problem: Service C took too long to respond. But without seeing the full timeline, you might have spent an hour looking at Service B’s code trying to figure out why it sent the message to the dead-letter queue.

Replay Events to Reproduce Bugs

One of the advantages of event-driven systems is that events are stored (at least for a while). This means you can reproduce bugs by replaying events — something you can’t do with synchronous HTTP calls.

In Kafka, you can reset a consumer group’s offset to a specific timestamp:

# Reset consumer to replay events from a specific time
kafka-consumer-groups.sh \
 --bootstrap-server localhost:9092 \
 --group my-consumer-group \
 --topic my-topic \
 --reset-offsets \
 --to-datetime 2026-02-05T10:00:00.000 \
 --execute

I’ve used this many times to reproduce production bugs in a staging environment. Copy the events from around the time of the bug, replay them against a staging consumer, and watch what happens. It’s like having a time machine for your system.

Of course, be careful with this. Replaying events against a production database can cause duplicate processing. Always replay against an isolated environment.

The Dashboard That Actually Helps

Most monitoring dashboards show you CPU, memory, and request counts. These are useful for capacity planning but nearly useless for debugging distributed systems.

The dashboard I actually look at during an incident has:

Consumer lag per consumer group. If a consumer is falling behind, something is slow or failing.
P99 latency per service, per endpoint. Not average latency (which hides problems) but the 99th percentile.
Error rate by error type. Not just “errors went up” but “TimeoutError to Service C went from 0.1% to 15%.”
Queue depth over time. Is the queue growing? At what rate? This tells you if your consumers can’t keep up.
Dead-letter queue count. If messages are landing here, something is wrong and needs human attention.

┌─────────────────────────────────────────────┐
│ Consumer Lag │
│ notification-sender: 0 (healthy) │
│ analytics-writer: 45,000 (⚠️ behind) │
│ search-indexer: 0 (healthy) │
├─────────────────────────────────────────────┤
│ DLQ Messages (last 24h) │
│ notification-sender: 3 │
│ analytics-writer: 0 │
│ search-indexer: 127 (🔴 investigate) │
└─────────────────────────────────────────────┘

Embrace the Uncertainty

Here’s the hardest part about debugging distributed systems: sometimes you won’t find a clean root cause. Sometimes the answer is “there was a 200ms network partition between zone A and zone B that caused a cascade of timeouts.” There’s no line of code to fix. There’s no bug to close.

In those cases, the fix isn’t fixing the bug — it’s making the system resilient to that class of failure. Add retries. Add circuit breakers. Increase timeouts. Add fallbacks. Accept that partial failures are normal and design for them.

Distributed systems are fundamentally about trade-offs. You trade simplicity for scalability, consistency for availability, debuggability for independence. The trick is knowing which trade-offs you’ve made, and being prepared when they come to collect.

The best distributed systems debuggers I know aren’t the ones who find bugs the fastest. They’re the ones who build systems that explain themselves when things go wrong.

Invest in correlation IDs, structured logging, and good dashboards. You’re building the tools your 3 AM self will thank you for.

Production-Ready AI

Sun, 18 Jan 2026 00:00:00 +0000

Adding AI to a system is easy. You grab an API key, write a prompt, call the model, and show the result. It works in the demo. The stakeholders nod approvingly.

Then you deploy to production, and everything falls apart in ways you didn’t anticipate.

The model hallucinates a response that contradicts your own data. Latency spikes to 8 seconds because the model is thinking really hard about a simple question. Your monthly bill triples because someone left a retry loop running against a large-context model. A customer sees a response that’s technically correct but completely wrong for their context.

The gap between demo and production AI is enormous. And most of it has nothing to do with the model itself.

The Things That Actually Matter

1. Deterministic Fallbacks

Here’s something I learned early: your AI feature needs to work without the AI.

That sounds contradictory, but it’s the most important design decision you’ll make. If the AI model is unavailable — the API is down, latency is too high, the response is garbage — your system should degrade to something useful, not crash or show a blank page.

For a search system, that means falling back to keyword search when semantic search fails. For an auto-reply feature, it means sending a canned response instead of nothing. For a classification pipeline, it means defaulting to a “needs human review” bucket.

def get_ai_response(prompt, context):
 try:
 response = llm.complete(
 prompt=prompt,
 timeout=3.0 # hard timeout, non-negotiable
 )
 if response.confidence < 0.7:
 return fallback_response(context)
 return response.text
 except (TimeoutError, APIError):
 return fallback_response(context)

The fallback isn’t a failure — it’s a feature. Users care far less about “AI-powered” than they care about “it works.”

2. Observability

AI systems are black boxes by default. The model takes an input, does mysterious computation, and produces an output. If the output is wrong, you have almost no traditional debugging tools.

This is why observability for AI isn’t optional — it’s critical. For every AI call, I log:

The full prompt (or a hash of it if it contains sensitive data)
The model’s response
Latency
Token count (input + output)
Confidence score (if available)
Whether the fallback was triggered
The user’s reaction (did they accept the suggestion? dismiss it? edit it?)

def log_ai_interaction(prompt, response, metadata):
 logger.info("ai_interaction", extra={
 "prompt_hash": hash(prompt),
 "response_length": len(response),
 "model": metadata.model,
 "latency_ms": metadata.latency_ms,
 "input_tokens": metadata.input_tokens,
 "output_tokens": metadata.output_tokens,
 "fallback_used": metadata.fallback_used,
 "correlation_id": metadata.correlation_id,
 })

This lets you answer questions like: “Why did our AI accuracy drop last Tuesday?” Maybe a prompt template was deployed with a typo. Maybe the context window was being exceeded for longer inputs. Maybe the model provider quietly changed their default model version. Without these logs, you’re guessing.

3. Cost Boundaries

AI API calls cost real money, and the cost model is different from anything else in your infrastructure. It’s not per-request or per-second — it’s per-token. A complex prompt with a large context can cost 100x more than a simple one.

I set cost boundaries at multiple levels:

Per-request budget. If a single prompt would cost more than $0.05, truncate the context or use a smaller model.
Per-user daily budget. Prevents a single heavy user from blowing up your bill.
Per-service monthly budget. Hard alerts when spend exceeds projections.

def estimate_cost(prompt, model="gpt-4"):
 token_count = tokenizer.count(prompt)
 cost_per_token = MODEL_COSTS[model]
 estimated_cost = token_count * cost_per_token

 if estimated_cost > MAX_COST_PER_REQUEST:
 # Switch to cheaper model or truncate context
 return use_lighter_model(prompt)
 return call_model(prompt, model)

The most expensive bug I’ve seen in an AI system wasn’t a logic error — it was a retry loop that kept sending the same large prompt to GPT-4. The team didn’t notice for two days. The bill was not pleasant.

4. Latency Budgets

AI model calls are slow compared to a database query. An LLM might take 2-5 seconds for a reasonable response. If your user expects a page load in under a second, you can’t synchronously call an LLM in the request path.

Strategies that work:

Async processing. Queue the AI call and show results when ready (works for background jobs like classification).
Streaming. Stream tokens as they arrive so the user sees something happening (works for chat-like interfaces).
Pre-computation. Run AI during off-peak hours and cache results (works for recommendations, summaries).
Model selection based on latency. Use a fast, smaller model for real-time features and a large model for batch jobs.

The key is deciding your latency budget upfront and designing around it, rather than adding a loading spinner and hoping nobody complains.

AI Should Never Be a Single Point of Failure

This is my one non-negotiable rule. AI features should enhance the experience, not gate it. If the AI is down, the core product should still work. If the AI gives bad output, there should be a way for the system (or the user) to recover gracefully.

Think of AI like a suggestion engine:

Suggestions can be wrong. That’s fine, as long as the user can override them.
Suggestions can be slow. That’s fine, as long as the rest of the system doesn’t wait.
Suggestions can be expensive. That’s fine, as long as you have guardrails.

The best AI features I’ve shipped are the ones where users don’t even realize AI is involved. It just feels like the product is smart. That’s the bar — not flashy “Powered by AI” badges, but quietly useful features that degrade gracefully and cost predictably.

Making AI work in a demo takes an afternoon. Making it production-ready takes weeks. The difference is everything that surrounds the model — fallbacks, logging, cost controls, latency management, and the humble acceptance that sometimes the best AI response is no AI response at all.

Why Queues Matter More Than You Think

Mon, 05 Jan 2026 00:00:00 +0000

Early in my career, I thought queues were just a way to “do stuff later.” Background jobs. Email sending. That kind of thing. They felt like an optimization — nice to have, not essential.

I was wrong. These days, I think queues are the single most underappreciated piece of infrastructure in backend systems. Not because they’re complex (they’re not), but because of how many problems they quietly solve.

The Problem with Synchronous Calls

Let’s start with a common pattern. Service A needs to tell Service B that something happened.

# Service A
def process_order(order):
 db.save_order(order)
 response = requests.post("http://service-b/notify", json=order.to_dict())
 if response.status_code != 200:
 raise Exception("Service B failed")
 return order

This works fine when both services are healthy. But now consider what happens when they’re not:

Service B is down. Your order processing fails even though it has nothing to do with notifications.
Service B is slow. Your request hangs, your thread pool fills up, and Service A starts rejecting all requests.
Traffic spikes. Both services get hammered. Service A can scale, but Service B can’t keep up. Requests start timing out.

The fundamental problem: synchronous calls create tight coupling. Service A’s availability depends on Service B’s availability, and there’s no buffer for traffic differences.

Add a Queue Between Them

# Service A — just publish and move on
def process_order(order):
 db.save_order(order)
 queue.publish("order.processed", order.to_dict())
 return order

# Service B — consume at its own pace
def handle_order_event(event):
 send_notification(event)

That’s it. One small change, and you get:

Decoupling. Service A doesn’t know or care about Service B. It publishes an event and moves on.
Resilience. If Service B goes down, messages pile up in the queue. When it comes back, it processes the backlog. No data loss.
Backpressure. Service B consumes at whatever rate it can handle. If traffic spikes, the queue absorbs the burst.

Backpressure Is the Key

Backpressure is the concept I wish someone had explained to me earlier. It’s simple: when a downstream system can’t keep up, the pressure needs to go somewhere. Without a queue, it goes to the caller — in the form of timeouts, errors, and cascading failures.

With a queue, the pressure goes to the queue itself. Messages stack up, but nobody crashes. The consumer chews through the backlog at a sustainable rate.

Traffic spike:
┌──────────┐
Requests → │ Queue │ → Consumer (steady rate)
100/s │ (buffer) │ 10/s
│ depth: │
│ growing │
└──────────┘
After spike:
┌──────────┐
Requests → │ Queue │ → Consumer (catches up)
5/s │ (buffer) │ 10/s
│ depth: │
│ draining │
└──────────┘

I’ve seen this pattern save production systems multiple times. A marketing email goes out, traffic spikes 10x, and the system survives because there’s a queue between the API and the heavy processing. Without the queue, the whole thing would fall over.

Retry Semantics Come Free

One of the best things about queues: retries are built into the model.

With synchronous calls, you have to build retry logic yourself — exponential backoff, jitter, max attempts, circuit breakers. It’s a lot of code, and it’s easy to get wrong.

With a queue (especially Kafka), if a consumer fails to process a message, it simply doesn’t commit the offset. The message stays in the queue and gets reprocessed. You don’t need to build retry infrastructure — it’s inherent to how consumers work.

# Kafka consumer — automatic retry via offset management
def consume():
 for message in consumer:
 try:
 process(message)
 consumer.commit() # success, move on
 except TransientError:
 # don't commit — message will be redelivered
 log.warning(f"Will retry: {message.key}")
 except PermanentError:
 # send to dead-letter queue, commit, move on
 dead_letter.publish(message)
 consumer.commit()

Of course, this means your consumers need to be idempotent. Processing the same message twice should have the same result as processing it once. That’s the trade-off — but it’s a much easier problem to solve than building a full retry framework.

When Queues Aren’t the Answer

Queues aren’t always right. Here’s when I don’t use them:

When you need an immediate response. If the user is waiting for the result of an operation, a queue adds latency that a direct call doesn’t. You can’t tell a user “your payment is being processed, results in 30 seconds” for an e-commerce checkout.
When the system is simple. Two services, low traffic, both maintained by the same team? A direct HTTP call is fine. Don’t add Kafka for a system that handles 50 requests per minute.
When ordering is critical and complex. Queues can preserve ordering within a partition, but if you need global ordering across multiple data types, you might be adding more complexity than you’re solving.

The Mental Model

I think of queues as shock absorbers. In a car, the road is uneven — bumps, potholes, sudden drops. Without shock absorbers, every bump transfers directly to the passengers. With them, the ride is smooth even when the road isn’t.

Your production traffic is the road. It’s uneven — spikes, bursts, sudden surges from a marketing campaign or a bot crawling your API. Without a queue, every spike transfers directly to your services. With a queue, the spike is absorbed, and your consumers keep processing at a steady, healthy rate.

The best part? This is boring infrastructure. Kafka, RabbitMQ, SQS — all battle-tested, well-documented, and understood by thousands of engineers. You’re not doing anything clever. You’re just putting a buffer in the right place.

And sometimes, that’s all it takes to turn a fragile system into a reliable one.

Boring Systems Win

Wed, 10 Dec 2025 00:00:00 +0000

Every few months, something shiny shows up. A new database that “scales infinitely.” A framework that promises to solve distributed consensus in 10 lines. A storage engine written in Rust that’s “blazingly fast.”

And every time, I have the same thought: will this thing still be maintained when I need to debug it at 2 AM on a Saturday?

The Case for Boring

The most reliable systems I’ve built — and more importantly, the ones I’ve operated — are boring. Not boring as in “nobody thought about them.” Boring as in the choices were so obvious they didn’t generate any debate.

Postgres over the latest NoSQL flavor. A simple REST API over a GraphQL layer nobody asked for. Cron jobs over a custom scheduler. A monolith before microservices.

These aren’t sexy choices. But here’s what they buy you:

Debugging is straightforward. Google your error and there are 400 Stack Overflow answers from 2014.
Onboarding is fast. New engineers don’t need a PhD in your custom abstraction layer.
Failure modes are well-understood. Postgres connection pooling issues? There’s a playbook. Your bespoke event mesh melting down? Good luck.

A Real Example

I once worked on a system that processed millions of messages a day. The original design had four message queues chained together, each with its own retry logic, dead-letter handling, and serialization format.

It was impressive on a whiteboard.

In production, debugging a single failed message meant jumping across four dashboards, correlating timestamps, and praying the logs hadn’t rotated.

We rewrote it with a single Kafka topic and a consumer that wrote to Postgres. One queue, one database, one log stream. The whole pipeline became something a new hire could trace end-to-end in 30 minutes.

Was it less “architecturally elegant”? Sure. But incidents that used to take hours to debug now took minutes.

When Boring Gets Hard

The tricky part is that boring takes discipline. It’s easy to say “just use Postgres” when you’re starting fresh. It’s harder when someone brings a compelling argument for a new technology in a design review.

Here’s my mental filter:

Will someone besides me operate this at 3 AM? If yes, pick the thing they already know.
Does this solve a problem we actually have, or one we might have? Premature optimization isn’t just about code — it applies to architecture too.
How many people in the world have debugged this in production? The size of that community is directly proportional to how fast you’ll recover from outages.

Boring Scales

There’s a misconception that boring tech doesn’t scale. That’s backwards. Boring tech has already scaled — that’s why it’s boring. Postgres handles more production traffic globally than most hyped databases ever will. Kafka has been battle-tested at LinkedIn, Netflix, and thousands of companies that never wrote blog posts about it.

The systems that survive aren’t the ones that trended on Hacker News. They’re the ones that kept running quietly while everything around them caught fire.

Boring systems don’t get applause.

But they don’t page you at 3 AM either.

Data Flow Is Everything

Sat, 22 Nov 2025 00:00:00 +0000

I’ve noticed a pattern over the years. Whenever a system starts behaving weirdly — data shows up late, duplicates appear, something silently goes stale — the root cause is almost never a bug in the traditional sense.

It’s a data flow problem.

Someone didn’t think about where the data comes from, who transforms it, who owns the authoritative copy, and what happens when something in that chain fails.

Start With a Map

Before writing any code for a new system, I draw the data flow. Not a UML diagram — nobody has time for that. Just boxes and arrows on a whiteboard (or, honestly, a notes app). Something like:

User Request
↓
API Gateway → Validation → Service A
↓
Kafka Topic
↓ ↓
Consumer B Consumer C
↓ ↓
Postgres Elasticsearch

This simple sketch already surfaces questions you’d otherwise discover in production:

Consumer B and Consumer C both process the same event. Are they idempotent? What if one succeeds and the other fails?
Postgres and Elasticsearch will eventually diverge. Which one is the source of truth? How do you reconcile?
If the API Gateway retries a failed request, does Service A handle duplicates?

I can’t count the number of times a 10-minute whiteboard session like this prevented weeks of debugging later.

The Four Questions

Whenever I look at any data flowing through a system, I ask four questions:

1. Where does it originate?

Every piece of data has a birth place. A user action, an external API, a scheduled job. Knowing the origin tells you about reliability — user inputs are messy, external APIs are flaky, scheduled jobs can overlap.

2. Who transforms it?

Data rarely stays in its original form. It gets validated, enriched, aggregated, filtered. Each transformation is a place where bugs hide. The more transformations, the harder it is to trace a value back to its origin.

Here’s a pattern that bites people: Service A publishes an event with a raw user input. Service B enriches it by calling an external API. Service C receives the enriched version. Now, when something’s wrong with the enriched data, you need to check three services to find the problem. Sometimes the external API returned bad data. Sometimes Service B had a caching bug. Sometimes Service A sent the wrong field.

3. Who owns it?

This is the question teams skip most often, and it causes the most pain. Ownership means: who is responsible when this data is wrong? If two services both write to the same row, you don’t have shared ownership — you have no ownership.

A rule I follow: one writer per data entity. If orders service writes order status, nobody else does. Other services read it, react to it, cache it — but they don’t write to it. This single constraint prevents an enormous class of data inconsistency bugs.

4. How does it fail?

This is where it gets interesting. Data flow failures are rarely boolean (works/doesn’t work). They’re usually partial, delayed, or silent:

A Kafka consumer falls behind. Data is technically flowing, but 4 hours late.
An API times out and a retry publishes a duplicate event.
A database migration adds a new column, but the producer hasn’t been updated. Now half the events have null in a field downstream services expect.

Drawing the failure paths — not just the happy paths — is what separates design docs that look good from systems that actually survive production.

An Example: The Accidental Dual Write

A while back, I worked on a messaging system where we needed to update both a thread’s status and send a notification whenever a new message arrived.

The first implementation was simple:

def handle_new_message(message):
 db.update_thread_status(message.thread_id, "active")
 notification_service.send(message.thread_id, "new_message")

What could go wrong?

Everything.

If the database update succeeds but the notification call fails, the thread is marked active but nobody gets notified. If I wrapped both in a try-except and retried, I’d potentially send duplicate notifications. If I used a database transaction, it wouldn’t matter because notification_service is a separate system — you can’t transact across two different services.

The fix was to stop trying to do both at once. Instead, we leaned on the data flow:

def handle_new_message(message):
 db.update_thread_status(message.thread_id, "active")
 # That's it. The DB change triggers a CDC event.

# Separate consumer picks up the change event
def on_thread_status_changed(event):
 if event.new_status == "active":
 notification_service.send(event.thread_id, "new_message")

Now each step does exactly one thing. The data flow is clear: message arrives → DB update → change event → notification. If the notification fails, we retry the event. No dual writes, no partial states.

Why This Matters

Most architecture debates — monolith vs. microservices, SQL vs. NoSQL, sync vs. async — are actually data flow debates in disguise. Once you draw the flow clearly, the right architecture usually becomes obvious.

The system’s behavior isn’t defined by its components. It’s defined by how data moves between them.

Start with the flow. The code follows.

Elasticsearch Sharding, Done Right

Sun, 10 Nov 2024 00:00:00 +0000

Elasticsearch is deceptively easy to get started with. Throw some JSON at it, and it indexes. Query it, and you get results. It feels like magic — until your cluster has 500 million documents and everything is on fire.

I’ve spent a lot of time operating Elasticsearch (well, OpenSearch these days, but the concepts are the same). And the lesson I keep learning is that the problems you hit at scale are almost never about Elasticsearch itself. They’re about decisions you made early — sharding strategy and index mappings — that seemed harmless at the time.

What Sharding Actually Is

Let’s start from the basics, because I’ve seen too many engineers use shards without really understanding what they are.

An Elasticsearch index is not a single blob of data sitting on a single machine. It’s split into shards, and each shard is an independent Lucene index. When you create an index with 5 primary shards, you’re essentially creating 5 smaller search engines that work together.

Index: "messages"
├── Shard 0 → Node A
├── Shard 1 → Node B
├── Shard 2 → Node A
├── Shard 3 → Node C
└── Shard 4 → Node B
Each shard is a complete, independent Lucene index.
A search query hits ALL shards in parallel, results are merged.

Why does this matter? Because:

Writes are distributed. Each document gets routed to one shard (by default, based on a hash of the document ID). More shards = writes spread across more nodes.
Reads are parallelized. A search query runs on every shard simultaneously. The results from each shard are merged and returned.
Each shard has overhead. Memory for segment metadata, file handles, thread pools. Shards aren’t free.

That last point is the one most people learn too late.

The Oversharding Problem

The most common mistake I see: too many shards. Someone creates an index with 20 primary shards “just in case” on a 3-node cluster. Each shard has a replica. That’s 40 shards for a single index across 3 nodes — roughly 13 shards per node.

Now multiply that by 30 indices (one per day, for a month of logs). That’s 1,200 shards on a 3-node cluster. Each shard consumes heap memory, file descriptors, and thread pool resources. The cluster spends more time managing shard metadata than actually searching.

Symptoms of oversharding:

Cluster state updates become slow
JVM heap pressure is high even with small indices
Search latency increases despite low query volume
Node startup takes forever because it’s recovering hundreds of shards

How Many Shards Do You Actually Need?

My rule of thumb:

Target 20-40 GB per shard. This is the sweet spot Elastic themselves recommend. Small enough to recover quickly, large enough to be efficient.
Keep total shards per node under 600-800. Beyond this, cluster state management becomes a bottleneck.
Start with 1 shard for small indices. Seriously. If your index is under 10 GB, one primary shard is fine.

Example calculation:
Expected index size: 100 GB
Target shard size: 30 GB
Primary shards needed: ceil(100 / 30) = 4
With 1 replica: 4 × 2 = 8 total shards
On a 3-node cluster: ~2-3 shards per node ✓

You can always add more shards later via reindexing or the split API. You can’t easily reduce them. Start small.

Index Mappings: The Thing You Can’t Fix Later

If sharding is the physical layout, mappings are the logical layout. They define how Elasticsearch interprets and stores each field in your documents.

And here’s the thing people don’t realize: once a field is mapped, you can’t change its type without reindexing. If you accidentally let Elasticsearch auto-detect that a field is text when it should be keyword, you’re stuck with it until you create a new index and migrate your data.

Text vs Keyword

This is the single most important mapping decision:

text: The field is analyzed — broken into tokens, lowercased, stemmed. Good for full-text search. “Quick brown fox” becomes ["quick", "brown", "fox"].
keyword: The field is stored as-is. Good for exact matches, aggregations, sorting. “Quick brown fox” stays "Quick brown fox".

{
 "mappings": {
 "properties": {
 "message_body": {
 "type": "text",
 "analyzer": "standard"
 },
 "status": {
 "type": "keyword"
 },
 "user_email": {
 "type": "keyword"
 },
 "created_at": {
 "type": "date",
 "format": "strict_date_optional_time||epoch_millis"
 }
 }
 }
}

Common mistakes I’ve seen:

Email addresses mapped as text. Elasticsearch tokenizes rohit@example.com into ["rohit", "example.com"]. Now searching for the exact email doesn’t work as expected, and aggregations on email are meaningless.
Status fields mapped as text. You want to filter by status: "active", but because it’s analyzed, you also match documents containing the word “active” anywhere in a text field.
Numeric IDs mapped as text. Thread IDs, user IDs — these should be keyword. You’re never going to full-text search an ID.

Turn Off Dynamic Mapping

By default, Elasticsearch will auto-detect field types when it sees a new field. This sounds helpful but causes problems:

A field that looks like a number gets mapped as long. Later you send a string value for the same field. Indexing fails.
String fields default to both text and keyword (a multi-field mapping), doubling the storage.
You end up with hundreds of fields you didn’t plan for, all consuming resources.

{
 "mappings": {
 "dynamic": "strict",
 "properties": {
 "thread_id": { "type": "keyword" },
 "message": { "type": "text" },
 "timestamp": { "type": "date" }
 }
 }
}

With "dynamic": "strict", Elasticsearch rejects any document with a field not in your mapping. It feels restrictive, but it prevents mapping surprises in production.

Managing Shards Over Time

For time-series data (logs, events, metrics), the standard approach is time-based indices — one index per day or per week. This gives you natural data lifecycle management: old indices can be closed, frozen, or deleted without affecting current data.

But time-based indices can lead to oversharding if you’re not careful. 365 days × 5 shards × 2 (with replicas) = 3,650 shards per year for a single data type.

Index Lifecycle Management (ILM)

ILM automates the lifecycle of your indices through phases:

Hot → Warm → Cold → Delete

Hot: Actively writing. Fast storage. Full replicas.
Warm: No longer writing. Can reduce replicas. Move to cheaper storage.
Cold: Rarely queried. Frozen indices use minimal resources.
Delete: Data retention period expired.

{
 "policy": {
 "phases": {
 "hot": {
 "actions": {
 "rollover": {
 "max_size": "30gb",
 "max_age": "1d"
 }
 }
 },
 "warm": {
 "min_age": "7d",
 "actions": {
 "shrink": { "number_of_shards": 1 },
 "forcemerge": { "max_num_segments": 1 }
 }
 },
 "delete": {
 "min_age": "90d",
 "actions": { "delete": {} }
 }
 }
 }
}

The shrink action in the warm phase is particularly useful — it reduces the number of shards in an index. That 5-shard index that made sense during active writes can become a 1-shard index once it’s read-only. This keeps your total shard count manageable.

Rollover Instead of Date-Based Names

Instead of creating logs-2024-11-10, logs-2024-11-11, etc., use rollover aliases. Elasticsearch creates a new index when the current one hits a size or age threshold.

This means your shards stay at a consistent, healthy size regardless of traffic patterns. On a busy day, you might roll over twice. On a quiet day, not at all. The data drives the sharding, not the calendar.

The Cheat Sheet

Here’s what I wish someone had told me early on:

Decision	Recommendation
Shard size	Target 20-40 GB per shard
Starting shards	1 for small indices, calculate for large ones
Replicas	At least 1 for production, 0 for dev
Dynamic mapping	Set to `strict` — define your fields explicitly
IDs and status fields	`keyword`, never `text`
Free-text content	`text` with an appropriate analyzer
Time-series data	Use ILM with rollover
Old indices	Shrink, then freeze or delete

The boring truth about Elasticsearch is that the clusters I’ve seen run smoothly aren’t the ones with the most nodes or the cleverest queries. They’re the ones where someone thought carefully about shards and mappings before the first document was indexed.

Get the foundation right, and everything else follows. Get it wrong, and no amount of hardware will save you.

You Probably Don't Need Microservices

Thu, 15 Jun 2023 00:00:00 +0000

In 2023, Amazon’s Prime Video team published a blog post that broke the internet — at least the engineering corner of it. They moved a monitoring service from a distributed microservices architecture back to a monolith. The result? 90% cost reduction.

The reaction was predictable. Half the internet said “see, microservices are a scam.” The other half said “you’re missing the context.” Both were partially right.

But the real takeaway wasn’t about Amazon at all. It was about the rest of us — the teams that adopted microservices not because they needed to, but because it felt like the right thing to do.

How We Got Here

Around 2015-2018, microservices became the default answer to every architecture question. Netflix talked about them. Uber talked about them. Conference talks were wall-to-wall microservices. If you were building a monolith, you were doing it wrong.

The pitch was compelling:

Independent deployment
Teams can move fast without stepping on each other
Scale individual services based on demand
Use the best language/framework for each service

All true. All real benefits. For organizations with hundreds of engineers deploying dozens of times a day.

The problem? Most of us aren’t Netflix. Most teams have 5-15 engineers, deploy a few times a week, and are building products where the biggest bottleneck is figuring out what to build — not how to scale it.

The Hidden Costs

What nobody talked about in those conference talks was the operational overhead. When you split a monolith into 10 services, you don’t just get 10 independent deployables. You get:

10 CI/CD pipelines to maintain
10 sets of logs to search through when something breaks
Network calls where you used to have function calls — each one a potential failure point
Distributed transactions where you used to have database transactions
Service discovery, load balancing, circuit breakers — infrastructure that didn’t exist before because it didn’t need to
Data consistency problems because you now have 10 databases instead of one

I’ve seen teams spend more time debugging inter-service communication than building features. The service mesh becomes the product, and the actual product becomes an afterthought.

Monolith:
function_a() → function_b() → function_c()
Total failure points: 0 network calls
Microservices:
Service A --HTTP-→ Service B --gRPC-→ Service C
Total failure points: 2 network calls, 2 serializations,
2 deserializations, load balancer hops,
DNS resolution, TLS handshakes...

The Monolith Isn’t the Problem

When teams hit scaling issues with a monolith, the instinct is to break it apart. But most of the time, the monolith isn’t the problem — the structure of the monolith is.

A well-structured monolith has clear module boundaries, separated concerns, and defined interfaces between components. It’s basically microservices in a single process, without the network overhead.

# A well-structured monolith — clear boundaries, single process

# orders/service.py
class OrderService:
 def create_order(self, user_id, items):
 order = self.order_repo.save(user_id, items)
 self.event_bus.emit("order.created", order) # in-process event
 return order

# notifications/handlers.py
class NotificationHandler:
 def on_order_created(self, event):
 self.email_service.send_confirmation(event.order)

# billing/handlers.py 
class BillingHandler:
 def on_order_created(self, event):
 self.invoice_service.generate(event.order)

Same separation of concerns. Same independent logic. But function calls instead of HTTP calls. Shared transactions instead of distributed ones. One deployment instead of three.

If the day comes when the notifications team needs to deploy 10 times a day independently, or the billing module needs to scale to 100x the traffic of orders — then you extract it. Not before. Not because a conference talk said so.

When Microservices Actually Make Sense

I’m not saying microservices are always wrong. They make sense when:

You have multiple teams that need to deploy independently and you’re stepping on each other’s toes
One part of the system has fundamentally different scaling needs — a real-time chat feature vs. a batch report generator
You’re integrating systems across different technology stacks — an ML pipeline in Python talking to a web app in Go
Organizational boundaries align with service boundaries — Conway’s Law is real

The key word is need. Not “might need someday” or “Netflix needs this.” Actual, present-tense need backed by real pain.

My Rule of Thumb

Start with a monolith. Structure it well. When a specific module has a clear, demonstrable reason to be independent — different scaling needs, different team ownership, different deployment cadence — extract it.

This isn’t revolutionary advice. It’s boring advice. And by now you know how I feel about boring.

The best architecture isn’t the one with the most boxes on the diagram. It’s the one where every box earns its operational overhead. For most teams, most of the time, that means fewer boxes than you think.