The hardest part of a consistent social content program isn't writing. It's knowing what to write about next week before this week is done. Most content teams solve this badly: a Slack channel where someone drops a link when they notice something interesting, a monthly "trends meeting" that recycles the same sources, a content calendar with suspicious amounts of "TBD" in the third and fourth weeks. The underlying problem is that trend discovery is manual, asynchronous, and dependent on whoever happens to be paying attention.
We built the scout system in Habitu Studio to automate that layer. Here's how it works, why the architecture looks the way it does, and what we learned building it, including the Reddit driver pivot that added two weeks of unplanned work to the slice.
Stage 1: Source Discovery
Arbitrary websites via Apify cheerio-scraper. For any URL a user pins as a source, a competitor's blog, an industry publication, a food-trend newsletter, a brand's Instagram link-in-bio page, we run an Apify cheerio-scraper job. Cheerio is the right tool here: it's fast, it doesn't need a full browser context for most editorial sites, and it returns structured page content we can pass downstream without a heavyweight preprocessing step.
Reddit via dedicated posts-search scraper. Reddit was the tricky one. The initial plan used a browser-playwright driver. We shipped it, ran it in staging, hit Reddit's anti-automation layer immediately, reverted, and spent a week evaluating alternatives. We landed on the Apify easyapi/reddit-posts-search-scraper, which uses Reddit's search surface rather than browser-based scraping. It's more stable, requires no session management, and returns clean post metadata, title, score, comment count, subreddit, timestamp, without the noise of a full DOM scrape.
The scout system also supports an operator-curated subreddit feed mode. For niche restaurant brands with a specific community fit, a specialty coffee chain monitoring r/Coffee, a Korean BBQ brand tracking r/KoreanFood, the operator pins specific subreddits rather than running open search. The feed mode returns the top posts from each pinned subreddit on a configurable schedule, pre-filtered by score threshold so low-engagement posts don't make it into the classifier queue.
Stage 2: The Claude Classifier
Raw scraper output is not a content calendar. A cheerio scrape returns article snippets, metadata, and sometimes navigational noise. A Reddit search returns a list of posts that may include memes, complaints, gear talk, and a handful of things that are actually relevant to a restaurant brand trying to produce content.
The classifier's job is transformation, not filtering. We pass the raw output from Stage 1 to a Claude call with a structured prompt that asks for: (1) a relevance score against the brand's category and content pillar map, (2) a transformation from raw signal to an actionable idea brief, not "there's a trend about birria," but "5 reasons birria tacos are outperforming burritos on TikTok this quarter, angles: texture video, chef POV, dip ritual," and (3) a confidence flag that marks the idea as high/medium/low based on how much corroboration exists across sources in the same run.
The classifier output is a JSONL stream of idea objects. Each object carries the source URL, the relevance score, the idea brief, confidence, and a suggested content format (short-form video, carousel, static post, long-form caption). The format suggestion is soft, the content team can override it, but it gives the operator a starting point rather than a blank page.
Scrapability Rank: Killing Dead Sources Early
One of the less glamorous parts of the system is the scrapability scoring that runs at pin time. When a user pins a new source URL, before it enters the regular scout rotation, the system runs a lightweight probe: can we retrieve content from this URL, does it return HTML we can extract signal from, how long does it take, does it 403 or bot-check on the first request?
Sources that fail the probe get a low scrapability rank and are demoted immediately. This matters more than it sounds. A dead or paywalled source that sits in the rotation will silently produce empty classifier runs. The operator sees no ideas from that source, assumes it's a slow trend week, and the bad source stays pinned indefinitely. We saw this pattern in early testing: 3 of 12 pinned sources in our own scout setup were returning no signal because they'd quietly gone behind a login wall or changed their DOM structure. The scrapability rank surfaces that problem immediately rather than burying it in empty results.
Low-rank sources are flagged in the UI with a warning and a "re-probe" button. They don't get removed automatically, the operator may know the source is seasonally quiet and want to keep it, but they're visible as a maintenance item rather than a silent data gap.
The Reddit Driver Pivot in Detail
The browser-playwright approach for Reddit felt reasonable in the design phase. Reddit's public search is straightforward HTML, playwright handles it cleanly in dev, and we already had playwright in the dependency tree for other scraping tasks.
In practice: Reddit detects playwright-driven sessions at scale via a combination of navigator fingerprint checks and rate patterns. The session would complete cleanly on the first run, return 403s on the second, and require a full session reset to recover. Rotating headers didn't fix it. Adding delays made it marginally better but not production-stable.
The switch to easyapi/reddit-posts-search-scraper cost about a week of integration work and a notable change to the data shape, the scraper returns structured JSON rather than raw HTML, which simplified the Stage 2 classifier prompt substantially. Post content, score, and metadata came pre-extracted. The pivot was unplanned but the outcome was cleaner than the original design. We now treat it as the right call in hindsight, not a detour.
How We Use Scout for Habitu's Own Content
We use the scout system to find what to write about for Habitu's own marketing content, including the posts on this blog. The pinned source list for Habitu's scout setup includes the major restaurant-tech publications, several mid-market operator communities on Reddit, and a handful of food-trend newsletters that have proven to generate high-signal classifier output.
The practical result: our content calendar is rarely empty past the current week. The scout run surfaces 8-12 ranked idea briefs per cycle. We pick 2-3, validate them against our content pillar map, and move them into draft. The ideas we don't use sit in the log and occasionally resurface weeks later when a related trend confirms what the scout spotted early.
The meta-lesson from building this: you cannot spec a content radar in the abstract. Every data source has a different failure mode, a different signal density, and a different relationship with scraping infrastructure. The only way to know which sources are worth maintaining is to run them and measure. The scrapability rank and the cost log are the two instrumentation surfaces that make that feedback loop tight enough to act on. Ship the instrumentation first; the content quality follows from there.
Studio is part of the tooling layer behind Habitu's branded app and content publishing surface. The scout system ships as part of Studio v2 (v0.37.0.0, 2026-05-03). The next slice adds scheduled scout runs and a trend-clustering pass that groups related ideas across sources before they hit the classifier queue.