Skip to content

Commit dba4b95

Browse files
ralyodioclaude
ralyodio
and
claude
authored
Stop the autoblog spam: every subject, none researched alone (#212)
* Write about every subject, and never one on its own coinpayportal.com published nineteen consecutive articles about peptide vendors — "skye peptides", "pure peptide labs", "wolverine stack peptides". Those are competitor storefronts in an industry the site sells payment processing to. Two independent defects produced them. Truncation. The site listed ten subjects. buildSeeds() sliced them to five, the DataForSEO loop took three of those, and subject #1 was handed to the buyer-journey model as its entire query. marijuana, dispensary, weed, iptv and torrents had never produced a single keyword. Platform-wide that is thirty-two subjects across nine sites that have never once been written about. The top-up sweep re-ran the same truncated set every time the queue drained, so the skew compounded instead of averaging out. An unanchored gate. A candidate survived if it contained the seed token. Expanding the bare word "peptide" returns the peptide industry's own vocabulary, and all of it passed a test that only ever asked "is this about peptides?" — never "is this about what we sell to them?". Both answers are the same: a subject is only ever researched crossed with a modifier. "peptide" is not a topic; "peptide merchant account" is. The cross is also the floor — it is built from two operator-controlled columns, so a blog that cannot reach any upstream still publishes, and still publishes about itself. An anchorless site now errors rather than falling back to the expansion that caused this. Allocation is fair-share weighted toward whatever is behind, and emission is interleaved: fairness measured over a quarter still reads as spam if it arrives as six peptide posts in a row. Duplicate detection moved from exact string match to a stemmed, order-independent fingerprint, which is what lets it see that "peptide payments" and "peptide payment" are the same article. Both shipped, nine days apart, in May. Covering ten subjects costs less than the three it replaced: the crosses go into one keywordIdeas call, which takes 200 seeds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013cbcXdpLFUgJZuVfCB4Gzv * One tick joins the network, and it starts ticked Published articles now carry an ad unit and a short list of related posts — partner blogs from the exchange, plus real posts from the RSS Amplifier directory so the block is not visibly the same three domains every time. Splitting "show ads" from "join the link network" would produce four states, two of them incoherent (take partner backlinks, refuse to give them) and all four needing a human to reason about. So it is one column, ads_enabled, defaulting true. Nobody is asked. A network everybody has to opt into is one that stays empty. The slot is provisioned rather than requested. ad_slots.status defaults to 'inactive' and serveAd() returns null for a non-active slot before the house-ad fallback, so a slot created at the default renders an empty div for ever and looks exactly like a broken embed. These are created active, once per project, reused on redelivery. The block is built at delivery, not generation, because the hosting site is only known once a guest post's target is resolved — and it is the host's readers who see the ad and the host's owner who opted in, not the author's. Titles and links come from RSS feeds crawled off the open web and land in HTML on a customer's domain. Everything interpolated is escaped, hrefs are allowlisted to http(s), and directory links are nofollow ugc: those publishers agreed to nothing, and passing them ranking signal would be spending someone else's reputation. Partner links stay followed — that reciprocity is the point of the exchange and is recorded on both sides. A failure anywhere in here costs the block, never the article. Publishing without an ad costs an impression; failing to publish costs the customer the thing they pay for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013cbcXdpLFUgJZuVfCB4Gzv * Daemonize the feed crawl, and put a status page on it The autoblog cites real posts from RSS Amplifier topic feeds. Those were fetched live, inside article delivery, on a three-second timeout — a third-party HTTP call on the critical path of the one operation a customer pays for. And the source was invisible: a topic feed going missing showed up as an empty block and nothing else, indistinguishable from a topic nobody had configured. Both have the same fix. The worker now reads feeds on its own 30-minute tick (each source at most every 6h, 25 per sweep, least-recently-fetched first, so work per tick is fixed however many subjects the platform grows to), caches what came back, and keeps the record of every attempt. Delivery reads rows. A cache miss costs one article its citation block rather than blocking a publish on somebody else's server. The source list is derived from every active site's master keywords on each sweep, never curated. A blog that adds a subject has that feed crawled on the next tick with nobody filing a request — which is what "no human intervention" actually requires, since a curated list is a queue of requests waiting for somebody. That is also why the status page has no "add a feed" button: it would imply a step that does not exist. Two judgements worth keeping. A 200 carrying no items is not a success — it is what the directory returns for a topic nobody publishes under, and counting it as one parks an empty topic at the front of the least-recently-fetched queue for ever, crowding out real feeds. And a source that gives up is parked, not deleted: a deleted row is re-derived from the same master keyword on the very next sweep and starts failing again, which is an infinite retry wearing the costume of a clean table. The page says "stalled" rather than naming a cause, because it cannot tell a stopped worker from an unreachable directory and those have different fixes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013cbcXdpLFUgJZuVfCB4Gzv * The anchor has to be a word the subject did not already supply Pulling the real lx_keyword rows for all seventeen active sites showed the peptide articles were not a coinpayportal problem. Every blog had it. vu1nz.com, CI/CD supply chain security, was queued to write about "adt home security", "brinks home security" and "security public storage". threatcrush, a SOC blog, had "evap line first response" and "emergency response liberty county" — pregnancy tests and a Roblox game. pairux had "garage door opener remote" and "remote control lawn mower". bittorrented had "micron technology" and "palantir technologies". crawlproof itself had "bayesian optimization", "scipy optimization minimize" and, genuinely, "xoloitzcuintli price". Two fixes, both found by running the gate against those rows rather than against fixtures. Anchors now exclude subject words. vu1nz's subject "devops security" and its niche both contain "security", so "adt home security" matched the subject on `security` and then matched the anchor on the same word — satisfying a two-part test with one token. An anchor has to be evidence the subject match did not already provide, or it is not a second test. And a site whose niche says nothing its subjects do not gets a commercial vocabulary instead of an empty anchor set. Mining vu1nz's niche yields only words its own subjects contain, and the alternative to a fallback there was either erroring out thirteen live sites or admitting everything. The vocabulary is narrower than the obvious one. "teams" and "business" were in the first draft and both had to come out: "community emergency response team" is a real queued keyword and it passed on the token `team`. A generic English noun cannot carry half of a two-part test however natural it reads. All 65 of those real strings are now regression cases, with the on-niche keywords each site should still accept alongside them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013cbcXdpLFUgJZuVfCB4Gzv * Rescue the good keywords the gate was also rejecting, and purge the rest Running the gate against all 744 live queued rows rather than fixtures found three things wrong with it, all of them over-rejection. The stemmer was broken. "-es" always lost two characters, so "codes" became "cod" while "code" stayed "code" and the two never matched — which meant a subject of "promo codes" could not match the keyword "promo code", and a coupon blog's entire queue was condemned. It also made several peptide rejections pass for the wrong reason: "skye peptides" was failing the subject match, not the anchor. Now strips one character by default and two only after a sibilant, so "boxes" still gives "box". A complete match on a multi-word subject is now its own evidence and needs no anchor. Every bad keyword found on live sites matched exactly one generic word out of a multi-word subject — "security" from "devops security", "remote" from "remote control". None matched a subject in full. Meanwhile "abercrombie promo code" matches "promo codes" completely and is exactly right for a coupon blog, which has no narrowing term to offer because its subject IS its topic. Single-word subjects are excluded, since that exemption is the original bug. And the commercial vocabulary is unioned into every anchor set rather than only used as a fallback. bl0ggers' niche yields {human, loop} once its own subjects are removed — non-empty, so the fallback never fired, and a thin anchor set rejected "ai writing tools". Together these took the verdict from 92 keywords kept to 279, with all 65 pinned junk strings still rejected. Then applied it: 465 off-niche queued rows deleted across sixteen sites. Published rows are never touched — those are URLs that exist on somebody's blog, and deleting the keyword would not unpublish the article, only lose the record of it. khipu-agency has no master keywords, so it was left alone rather than emptied on the strength of an opinion the gate cannot form. Most sites are now under the top-up threshold and will refill through the anchored pipeline on the next cron tick. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013cbcXdpLFUgJZuVfCB4Gzv --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent ae6b61d commit dba4b95

21 files changed

Lines changed: 2949 additions & 192 deletions

‎app/(app)/dashboard/autoblog/crawler/page.tsx‎

Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
// Live view of the feed crawler, modelled on rssamplifier.com/crawlstats.
2+
//
3+
// The autoblog cites real posts from RSS Amplifier topic feeds, and until the
4+
// crawl was daemonized that source was invisible: a topic going missing looked
5+
// identical to a topic nobody had configured. This is where that difference
6+
// becomes visible.
7+
//
8+
// Deliberately read-only and unconfigurable. The feed list is derived from
9+
// every active site's master keywords on each sweep, so there is nothing here
10+
// to add or remove — a subject added to a blog appears within a tick. An
11+
// "add a feed" button would imply a curation step that does not exist, and
12+
// would create a queue of requests waiting for a human, which is the thing
13+
// the automation is meant to remove.
14+
15+
import { serviceClient } from "@/lib/supabase/service";
16+
import { ago, loadCrawlerStats } from "@/lib/lx/feedCrawlStats";
17+
18+
// Always fresh: a status page served from a cache reports the cache's health,
19+
// not the crawler's, and would go on saying "healthy" through an outage.
20+
export const dynamic = "force-dynamic";
21+
22+
function Stat({
23+
label,
24+
value,
25+
note,
26+
}: {
27+
label: string;
28+
value: string;
29+
note?: string;
30+
}) {
31+
return (
32+
<div className="card p-4">
33+
<div className="text-2xl font-bold tabular-nums">{value}</div>
34+
<div className="text-xs uppercase tracking-wider text-[var(--color-muted)]">
35+
{label}
36+
</div>
37+
{note ? (
38+
<div className="mt-1 text-xs text-[var(--color-muted)]">{note}</div>
39+
) : null}
40+
</div>
41+
);
42+
}
43+
44+
export default async function CrawlerStatusPage() {
45+
const { stats, sources } = await loadCrawlerStats(serviceClient());
46+
const now = Date.now();
47+
48+
return (
49+
<div className="mx-auto max-w-5xl">
50+
<h1 className="text-3xl font-bold">Crawler status</h1>
51+
<p className="mt-2 text-[var(--color-muted)]">
52+
Live view of the daemon that reads the RSS Amplifier topic feeds the
53+
autoblog cites from. The feed list is derived from every active
54+
site&apos;s master keywords — nothing here is curated by hand. The same
55+
numbers are available as{" "}
56+
<a className="underline" href="/api/lx/feed-crawl/status">
57+
JSON
58+
</a>
59+
.
60+
</p>
61+
62+
{stats.stalled ? (
63+
<div className="mt-4 rounded border border-[var(--color-danger,#b91c1c)] p-3 text-sm">
64+
<strong>Stalled.</strong> {stats.dueNow.toLocaleString()} feed(s) are
65+
due and nothing has been read successfully since{" "}
66+
{ago(stats.lastSuccessAt, now)}. Either the worker is not running or
67+
the directory is unreachable — this page cannot tell which, and the
68+
two have different fixes.
69+
</div>
70+
) : null}
71+
72+
<div className="mt-6 grid grid-cols-2 gap-3 sm:grid-cols-4">
73+
<Stat
74+
label="Feeds"
75+
value={stats.sources.toLocaleString()}
76+
note={`${stats.active.toLocaleString()} active`}
77+
/>
78+
<Stat
79+
label="Due now"
80+
value={stats.dueNow.toLocaleString()}
81+
note="waiting to be crawled"
82+
/>
83+
<Stat
84+
label="Cached posts"
85+
value={stats.items.toLocaleString()}
86+
note={`${stats.newItems24h.toLocaleString()} new in 24h`}
87+
/>
88+
<Stat
89+
label="Erroring"
90+
value={stats.erroring.toLocaleString()}
91+
note={`${stats.gaveUp.toLocaleString()} given up`}
92+
/>
93+
</div>
94+
95+
<p className="mt-3 text-xs text-[var(--color-muted)]">
96+
Last successful fetch {ago(stats.lastSuccessAt, now)} ·{" "}
97+
{stats.neverFetched.toLocaleString()} never fetched · generated{" "}
98+
{stats.generatedAt}
99+
</p>
100+
101+
<h2 className="mt-8 text-xl font-semibold">Feeds</h2>
102+
<p className="mt-1 text-sm text-[var(--color-muted)]">
103+
One row per subject any active blog covers. A feed is re-read at most
104+
every 6 hours, least-recently-fetched first, 25 per sweep — so the work
105+
per tick is fixed however many subjects the platform grows to.
106+
</p>
107+
108+
{sources.length === 0 ? (
109+
<p className="mt-4 text-sm text-[var(--color-muted)]">
110+
No feeds yet. The first sweep derives them from the active sites&apos;
111+
master keywords; if this stays empty, no site has master keywords set.
112+
</p>
113+
) : (
114+
<div className="mt-4 overflow-x-auto">
115+
<table className="w-full text-sm">
116+
<thead>
117+
<tr className="text-left text-xs uppercase tracking-wider text-[var(--color-muted)]">
118+
<th className="py-2 pr-4">Topic</th>
119+
<th className="py-2 pr-4">Status</th>
120+
<th className="py-2 pr-4">Last read</th>
121+
<th className="py-2 pr-4">Last success</th>
122+
<th className="py-2 pr-4 text-right">Items</th>
123+
<th className="py-2">Last error</th>
124+
</tr>
125+
</thead>
126+
<tbody>
127+
{sources.map((source) => (
128+
<tr
129+
key={source.id}
130+
className="border-t border-[var(--color-border,#2a2a2a)]"
131+
>
132+
<td className="py-2 pr-4">
133+
<a className="underline" href={source.url} rel="noopener">
134+
{source.topic}
135+
</a>
136+
</td>
137+
<td className="py-2 pr-4">
138+
{source.status === "given_up" ? (
139+
<span
140+
title={`${source.consecutive_failures} consecutive failures`}
141+
>
142+
given up
143+
</span>
144+
) : source.consecutive_failures > 0 ? (
145+
<span>failing ({source.consecutive_failures})</span>
146+
) : (
147+
<span>active</span>
148+
)}
149+
</td>
150+
<td className="py-2 pr-4">{ago(source.last_fetch_at, now)}</td>
151+
<td className="py-2 pr-4">
152+
{ago(source.last_success_at, now)}
153+
</td>
154+
<td className="py-2 pr-4 text-right tabular-nums">
155+
{source.item_count.toLocaleString()}
156+
</td>
157+
<td className="py-2 text-xs text-[var(--color-muted)]">
158+
{source.last_error ?? ""}
159+
</td>
160+
</tr>
161+
))}
162+
</tbody>
163+
</table>
164+
</div>
165+
)}
166+
</div>
167+
);
168+
}

‎app/(app)/dashboard/projects/[id]/autoblog/setup/form.tsx‎

Lines changed: 71 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,8 +23,10 @@ type Existing = {
2323
target_audiences: string[];
2424
description: string;
2525
seed_keywords: string[];
26+
master_keywords: string[];
2627
modifiers: string[];
2728
preserve_keywords: boolean;
29+
ads_enabled: boolean;
2830
keywords: string[];
2931
seo_title: string | null;
3032
seo_description: string | null;
@@ -109,6 +111,14 @@ export function SetupForm({
109111
(initial?.target_audiences ?? []).join(", "),
110112
);
111113
const [description, setDescription] = useState(initial?.description ?? "");
114+
const [masterKeywords, setMasterKeywords] = useState(
115+
// Falls back to the seed list for any site the backfill has not reached,
116+
// so the field is never blank on a project that already has subjects.
117+
(initial?.master_keywords?.length
118+
? initial.master_keywords
119+
: (initial?.seed_keywords ?? [])
120+
).join(", "),
121+
);
112122
const [seedKeywords, setSeedKeywords] = useState(
113123
(initial?.seed_keywords ?? []).join(", "),
114124
);
@@ -118,6 +128,11 @@ export function SetupForm({
118128
const [preserveKeywords, setPreserveKeywords] = useState<boolean>(
119129
initial?.preserve_keywords ?? false,
120130
);
131+
// Default ON, matching the column default. A new project is in the network
132+
// unless its owner takes the tick out.
133+
const [adsEnabled, setAdsEnabled] = useState<boolean>(
134+
initial?.ads_enabled ?? true,
135+
);
121136
const [keywords, setKeywords] = useState(
122137
(initial?.keywords ?? []).join("\n"),
123138
);
@@ -425,6 +440,13 @@ export function SetupForm({
425440
// Each line is a CSV row "<keyword>,<volume>". For seed-building and
426441
// dedupe we only care about the keyword half (everything before the
427442
// first comma).
443+
function masterKeywordsAsArray(): string[] {
444+
return masterKeywords
445+
.split(",")
446+
.map((s) => s.trim())
447+
.filter(Boolean);
448+
}
449+
428450
function keywordsAsArray(): string[] {
429451
return keywords
430452
.split("\n")
@@ -569,9 +591,11 @@ export function SetupForm({
569591
niche,
570592
targetAudiences: audiences,
571593
description,
594+
masterKeywords,
572595
seedKeywords,
573596
modifiers,
574597
preserveKeywords,
598+
adsEnabled,
575599
keywords,
576600
seoTitle,
577601
seoDescription,
@@ -843,6 +867,34 @@ export function SetupForm({
843867
onChange={(e) => setDescription(e.target.value)}
844868
/>
845869
</div>
870+
<div>
871+
<label className="text-xs uppercase tracking-wider text-[var(--color-muted)]">
872+
Master keywords (comma-separated, 3-12 subjects)
873+
</label>
874+
<input
875+
className="input mt-1 w-full font-mono text-sm"
876+
type="text"
877+
placeholder="peptide, crypto, casino, marijuana, iptv, torrents"
878+
value={masterKeywords}
879+
onChange={(e) => setMasterKeywords(e.target.value)}
880+
/>
881+
<p className="mt-1 text-xs text-[var(--color-muted)]">
882+
The subjects this blog covers, and the list it always rebuilds from.
883+
The planner splits every research run evenly across{" "}
884+
<strong>all</strong> of them and fills whichever is furthest behind
885+
first, so a subject cannot run away with the schedule. Each one is
886+
researched crossed with a modifier below — never on its own.{" "}
887+
{masterKeywordsAsArray().length > 12 ? (
888+
<strong className="text-[var(--color-danger,#b91c1c)]">
889+
{masterKeywordsAsArray().length} entered — only the first 12 are
890+
kept.
891+
</strong>
892+
) : (
893+
<>{masterKeywordsAsArray().length} subject(s).</>
894+
)}
895+
</p>
896+
</div>
897+
846898
<div>
847899
<label className="text-xs uppercase tracking-wider text-[var(--color-muted)]">
848900
Seed keywords (comma-separated, 1-3 word head terms)
@@ -913,7 +965,25 @@ export function SetupForm({
913965
<span>
914966
<span className="font-medium">Preserve keywords</span> — Refetch
915967
flows skip the seed list and the long-tail list (still refresh
916-
niche, audiences, description).
968+
niche, audiences, description). Master keywords are always
969+
preserved and are unaffected by this.
970+
</span>
971+
</label>
972+
</div>
973+
974+
<div>
975+
<label className="flex items-center gap-2 text-sm">
976+
<input
977+
type="checkbox"
978+
checked={adsEnabled}
979+
onChange={(e) => setAdsEnabled(e.target.checked)}
980+
/>
981+
<span>
982+
<span className="font-medium">Join the CrawlProof network</span> —
983+
published articles carry an ad unit and a short list of related
984+
posts from partner blogs and the RSS Amplifier directory. Partner
985+
links are followed; directory links are <code>nofollow ugc</code>.
986+
An ad slot is created for this project automatically.
917987
</span>
918988
</label>
919989
</div>

‎app/(app)/dashboard/projects/[id]/autoblog/setup/page.tsx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ import { SetupForm } from "./form";
77
export const metadata = { title: "Autoblog · Setup" };
88

99
const SITE_COLUMNS =
10-
"id, domain, blog_root_url, sitemap_url, niche, target_audiences, description, seed_keywords, modifiers, preserve_keywords, keywords, seo_title, seo_description, tone, competitors, webhook_url, webhook_secret, daily_article_count, publish_days, publish_hour, internal_links_per_article, backlinks_enabled, external_links_per_article, banner_style, status";
10+
"id, domain, blog_root_url, sitemap_url, niche, target_audiences, description, seed_keywords, master_keywords, modifiers, preserve_keywords, ads_enabled, keywords, seo_title, seo_description, tone, competitors, webhook_url, webhook_secret, daily_article_count, publish_days, publish_hour, internal_links_per_article, backlinks_enabled, external_links_per_article, banner_style, status";
1111

1212
export default async function AutoblogSetupPage({
1313
params,

‎app/actions/linkExchange.ts‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -289,6 +289,9 @@ export type SiteInput = {
289289
niche: string;
290290
targetAudiences: string;
291291
description: string;
292+
// Comma-separated 3-12 subjects this blog covers. The durable list the
293+
// keyword planner allocates evenly across; see lib/lx/topicPlan.
294+
masterKeywords: string;
292295
// Comma-separated 1-3 word head terms for DataForSEO expansion.
293296
seedKeywords: string;
294297
// Comma-separated tail terms ("payments", "merchant account") that the
@@ -297,6 +300,9 @@ export type SiteInput = {
297300
// When true, Refetch flows skip overwriting the keywords text[]. Used
298301
// after a hand-curated build via seeds × modifiers.
299302
preserveKeywords: boolean;
303+
// Network opt-in: ad units and partner/directory links in published
304+
// articles. Default true — see the migration note on lx_site.ads_enabled.
305+
adsEnabled: boolean;
300306
// Comma-separated. parseList() into a text[].
301307
keywords: string;
302308
seoTitle: string;
@@ -360,6 +366,12 @@ export async function createOrUpdateSite(
360366
const seedKeywords = parseList(input.seedKeywords, 50, 40);
361367
const modifiers = parseList(input.modifiers, 20, 40);
362368
const preserveKeywords = !!input.preserveKeywords;
369+
// The DB constraint caps this at 12; parse to the same number so an
370+
// oversized paste is trimmed here rather than rejected as a write error the
371+
// form cannot explain.
372+
const masterKeywords = parseList(input.masterKeywords, 12, MAX.keyword);
373+
// Absent reads as opted in, matching the column default.
374+
const adsEnabled = input.adsEnabled !== false;
363375
// Keywords use parseKeywordRows so each row keeps its `,<volume>` hint.
364376
// Allow up to ~80 chars per row to fit "keyword phrase here,12345".
365377
const keywords = parseKeywordRows(input.keywords, MAX.keywords, MAX.keyword + 20);
@@ -433,6 +445,8 @@ export async function createOrUpdateSite(
433445
target_audiences: audiences,
434446
description,
435447
seed_keywords: seedKeywords,
448+
master_keywords: masterKeywords,
449+
ads_enabled: adsEnabled,
436450
modifiers,
437451
preserve_keywords: preserveKeywords,
438452
keywords,
@@ -489,6 +503,8 @@ export async function createOrUpdateSite(
489503
target_audiences: audiences,
490504
description,
491505
seed_keywords: seedKeywords,
506+
master_keywords: masterKeywords,
507+
ads_enabled: adsEnabled,
492508
modifiers,
493509
preserve_keywords: preserveKeywords,
494510
keywords,
@@ -534,6 +550,8 @@ export async function createOrUpdateSite(
534550
target_audiences: audiences,
535551
description,
536552
seed_keywords: seedKeywords,
553+
master_keywords: masterKeywords,
554+
ads_enabled: adsEnabled,
537555
modifiers,
538556
preserve_keywords: preserveKeywords,
539557
keywords,

‎app/api/lx/feed-crawl/status/route.ts‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
import { NextResponse } from "next/server";
2+
import { serviceClient } from "@/lib/supabase/service";
3+
import { loadCrawlerStats } from "@/lib/lx/feedCrawlStats";
4+
5+
export const runtime = "nodejs";
6+
// A cached status response reports the cache's health rather than the
7+
// crawler's, and would keep answering "fine" straight through an outage.
8+
export const dynamic = "force-dynamic";
9+
10+
/**
11+
* Counts only.
12+
*
13+
* The per-feed rows stay on the dashboard page behind a session: this is the
14+
* shape a monitor polls, and it has no reason to carry the error strings some
15+
* upstream server wrote.
16+
*/
17+
export async function GET() {
18+
const { stats } = await loadCrawlerStats(serviceClient());
19+
return NextResponse.json(stats, { headers: { "cache-control": "no-store" } });
20+
}

0 commit comments

Comments
 (0)