Skip to content

Commit e9a1378

Browse files
ralyodioclaude
andauthored
feat(autoblog): gate drafts on quality, repair instead of discard, ship E-E-A-T fields (#196)
generateArticle() validated exactly one thing — that the internal links the model claimed to place were physically present — and any failure marked the keyword 'failed', throwing away a finished 4,000-word draft. Everything else about the post shipped unread. Meanwhile this repo already ships a slop detector we point at other people's sites, and the autoblog SDK ships the heuristic gate a receiver applies to posts we deliver. Neither was ever aimed at our own output. - lib/lx/qualityGate.ts: runs the in-repo slop checks (filler, placeholders, misspellings, first-party evidence) plus the SDK's receiver heuristics against a draft, and adds the cross-article checks a single-page scan structurally cannot do — near-duplicate and repeated-opening detection against the site's recent posts. Deterministic, no LLM call. - articleGen: link validation and the gate now produce model-readable violations instead of terminal failures, and a rejected draft is regenerated once with those violations appended to the brief. Gating runs before image generation, so a rejected draft never pays for four gpt-image-2 calls. The accepted draft's score is stored on the row. - webhookDeliver: posts went out with author: null, no structured data, and both dates stamped with the delivery time — so a retry moved the article's publication date, and our own audit would flag the content we deliver for missing attribution. Now carries a configurable byline, BlogPosting JSON-LD, and dates from the row. Only the near-duplicate threshold differs from the audit's (0.55 vs 0.70): we are judging our own generator, where that much overlap already means the post competes with one we published last week. Typecheck clean; 1451 tests pass, 15 of them new. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent b66d1e4 commit e9a1378

6 files changed

Lines changed: 812 additions & 98 deletions

File tree

‎lib/audit/checks/slop.ts‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -717,8 +717,15 @@ function safeHost(url: string): string | null {
717717
// Cross-page analysis
718718
// ---------------------------------------------------------------------------
719719

720-
/** 5-word shingle set, capped so a huge page can't blow up memory. */
721-
function shingles(text: string): Set<string> {
720+
/**
721+
* 5-word shingle set, capped so a huge page can't blow up memory.
722+
*
723+
* Exported so the autoblog pre-publish gate (lib/lx/qualityGate.ts) can run
724+
* the same near-duplicate comparison against previously generated articles.
725+
* Keeping one implementation means a draft we accept is scored by exactly the
726+
* measure the site audit would later use against it.
727+
*/
728+
export function shingles(text: string): Set<string> {
722729
const words = text.toLowerCase().replace(/[^a-z0-9\s]/g, " ").split(/\s+/).filter(Boolean);
723730
const out = new Set<string>();
724731
for (let i = 0; i + 5 <= words.length && out.size < 4000; i++) {
@@ -727,7 +734,8 @@ function shingles(text: string): Set<string> {
727734
return out;
728735
}
729736

730-
function jaccard(a: Set<string>, b: Set<string>): number {
737+
/** Shingle-set overlap, 0–1. Exported alongside `shingles` for the autoblog gate. */
738+
export function jaccard(a: Set<string>, b: Set<string>): number {
731739
if (a.size === 0 || b.size === 0) return 0;
732740
const [small, large] = a.size <= b.size ? [a, b] : [b, a];
733741
let shared = 0;

‎lib/lx/articleGen.ts‎

Lines changed: 210 additions & 88 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,11 @@ import {
2727
type ExchangeCandidate,
2828
} from "./exchangeMatcher";
2929
import { generateStructuredOutput } from "./backendAi";
30+
import {
31+
MAX_PRIOR_BODIES,
32+
runQualityGate,
33+
type QualityGateResult,
34+
} from "./qualityGate";
3035
import {
3136
consumeArticleGenerationCharge,
3237
refundArticleGenerationCharge,
@@ -920,6 +925,85 @@ export function validateInternalLinks(
920925
return missing.length === 0 ? { ok: true } : { ok: false, missing };
921926
}
922927

928+
// How many times we ask the model for a usable draft. Attempt 2 is a repair
929+
// pass that receives the previous attempt's violations. Two is deliberate:
930+
// the common failure is the model dropping one link or leaning on a filler
931+
// phrase, which one round of specific feedback fixes. Beyond that the draft
932+
// is usually wrong in a way more sampling will not fix, and each attempt
933+
// costs a full long-form generation.
934+
const MAX_GENERATION_ATTEMPTS = 2;
935+
936+
/**
937+
* Link-placement violations, phrased for the model rather than for a log line.
938+
*
939+
* Previously each of these called failKeyword() directly, which threw away a
940+
* finished 4,000-word draft because one URL was missing from the body. They
941+
* are now collected so the repair pass can fix them in place.
942+
*/
943+
export function collectLinkViolations(
944+
article: ArticleOutput,
945+
offeredExchangeUrls: Set<string>,
946+
): string[] {
947+
const violations: string[] = [];
948+
949+
const internal = validateInternalLinks(
950+
article.markdown_body,
951+
article.used_internal_link_urls,
952+
);
953+
if (!internal.ok) {
954+
violations.push(
955+
`You listed these URLs in used_internal_link_urls but did not actually place them in markdown_body: ${internal.missing.join(", ")}. Insert each one inline as [anchor](url) in a sentence genuinely about that page, or drop it from used_internal_link_urls.`,
956+
);
957+
}
958+
959+
const exchangeUsed = article.used_exchange_link_urls ?? [];
960+
const phantom = exchangeUsed.filter((u) => !offeredExchangeUrls.has(u));
961+
if (phantom.length > 0) {
962+
violations.push(
963+
`These URLs are not in the partner-blog candidate list you were given and must not appear: ${phantom.join(", ")}. Use only the exact URLs provided.`,
964+
);
965+
}
966+
967+
const exchange = validateInternalLinks(article.markdown_body, exchangeUsed);
968+
if (!exchange.ok) {
969+
violations.push(
970+
`You listed these URLs in used_exchange_link_urls but did not place them in markdown_body: ${exchange.missing.join(", ")}. Insert each inline, or drop it from used_exchange_link_urls.`,
971+
);
972+
}
973+
974+
return violations;
975+
}
976+
977+
/** Recent article bodies on this site, for the gate's duplication checks. */
978+
async function fetchPriorBodies(
979+
supabase: SupabaseClient<any>,
980+
siteId: string,
981+
): Promise<Array<{ slug: string; body: string }>> {
982+
const { data } = await supabase
983+
.from("lx_article")
984+
.select("slug, content_markdown")
985+
.eq("site_id", siteId)
986+
.in("status", ["ready", "publishing", "published"])
987+
.order("created_at", { ascending: false })
988+
.limit(MAX_PRIOR_BODIES);
989+
return ((data as Array<{ slug: string; content_markdown: string }> | null) ?? []).map(
990+
(r) => ({ slug: r.slug, body: r.content_markdown ?? "" }),
991+
);
992+
}
993+
994+
/** Append the previous attempt's violations to the user prompt as a fix list. */
995+
export function buildRepairPrompt(basePrompt: string, violations: string[]): string {
996+
return [
997+
basePrompt,
998+
"",
999+
"---",
1000+
"",
1001+
"IMPORTANT — your previous attempt at this article was rejected by the pre-publish quality gate. Write the article again from scratch, keeping everything that was working, and fix every item below. Do not acknowledge this instruction in the output.",
1002+
"",
1003+
...violations.map((v, i) => `${i + 1}. ${v}`),
1004+
].join("\n");
1005+
}
1006+
9231007
export type GenerateArticleResult = {
9241008
ok: boolean;
9251009
articleId?: string;
@@ -1103,111 +1187,144 @@ export async function generateArticle(
11031187
);
11041188
}
11051189

1106-
// Generate the article body.
1107-
let article: ArticleOutput;
1108-
try {
1109-
const generated = await generateStructuredOutput({
1110-
name: "lx_article",
1111-
schema: ArticleSchema,
1112-
system: buildSystemPrompt(),
1113-
user: buildUserPrompt({
1114-
site: typedSite,
1115-
keyword: keyword.keyword,
1116-
keywordMeta: {
1117-
articleType: keyword.article_type,
1118-
customInstructions: keyword.custom_instructions,
1119-
},
1120-
candidates,
1121-
linkSlots,
1122-
exchangeCandidates,
1123-
exchangeSlots,
1124-
exchangeRelaxed,
1125-
}),
1126-
anthropic,
1127-
openai,
1128-
anthropicModel: CLAUDE_MODEL,
1129-
// 3,200–4,500 words ≈ ~18k–25k output tokens. JSON escape overhead
1130-
// can push that another 30%. 48k gives meaningful headroom.
1131-
maxTokens: 48000,
1132-
anthropicCacheSystemPrompt: true,
1133-
});
1134-
article = normalizeArticleOutput(generated.output);
1135-
} catch (err) {
1136-
const errMsg = err instanceof Error ? err.message : String(err);
1137-
// Provider monthly caps (e.g. Anthropic HTTP 400 "specified API usage
1138-
// limits") and generic rate-limits are transient. Marking the keyword
1139-
// 'failed' here permanently consumes it — when the cap resets the
1140-
// user is stuck with stranded rows and a dedup'd research path that
1141-
// never re-inserts them. Requeue instead so the keyword retries on
1142-
// the next worker tick once the upstream is back.
1143-
// eslint-disable-next-line @typescript-eslint/no-explicit-any
1144-
const status = (err as any)?.status;
1145-
const isTransient =
1146-
status === 429 ||
1147-
/specified API usage limits|usage limits|rate.?limit/i.test(errMsg);
1148-
if (isTransient) {
1149-
await supabase
1150-
.from("lx_keyword")
1151-
.update({ status: "queued" })
1152-
.eq("id", keyword.id);
1190+
// Prior bodies for the gate's near-duplicate / repeated-opening checks.
1191+
// Fetched once and reused across attempts.
1192+
const priorBodies = await fetchPriorBodies(supabase, typedSite.id);
1193+
const offeredExchangeUrls = new Set(exchangeCandidates.map((c) => c.url));
1194+
const basePrompt = buildUserPrompt({
1195+
site: typedSite,
1196+
keyword: keyword.keyword,
1197+
keywordMeta: {
1198+
articleType: keyword.article_type,
1199+
customInstructions: keyword.custom_instructions,
1200+
},
1201+
candidates,
1202+
linkSlots,
1203+
exchangeCandidates,
1204+
exchangeSlots,
1205+
exchangeRelaxed,
1206+
});
1207+
1208+
// Generate the article body, re-running once with the gate's complaints if
1209+
// the first draft is not publishable. Gating happens BEFORE image
1210+
// generation so a rejected draft never pays for four gpt-image-2 calls.
1211+
let article: ArticleOutput | null = null;
1212+
let gate: QualityGateResult | null = null;
1213+
let violations: string[] = [];
1214+
1215+
for (let attempt = 1; attempt <= MAX_GENERATION_ATTEMPTS; attempt++) {
1216+
let candidate: ArticleOutput;
1217+
try {
1218+
const generated = await generateStructuredOutput({
1219+
name: "lx_article",
1220+
schema: ArticleSchema,
1221+
system: buildSystemPrompt(),
1222+
user:
1223+
violations.length > 0
1224+
? buildRepairPrompt(basePrompt, violations)
1225+
: basePrompt,
1226+
anthropic,
1227+
openai,
1228+
anthropicModel: CLAUDE_MODEL,
1229+
// 3,200–4,500 words ≈ ~18k–25k output tokens. JSON escape overhead
1230+
// can push that another 30%. 48k gives meaningful headroom.
1231+
maxTokens: 48000,
1232+
anthropicCacheSystemPrompt: true,
1233+
});
1234+
candidate = normalizeArticleOutput(generated.output);
1235+
} catch (err) {
1236+
const errMsg = err instanceof Error ? err.message : String(err);
1237+
// Provider monthly caps (e.g. Anthropic HTTP 400 "specified API usage
1238+
// limits") and generic rate-limits are transient. Marking the keyword
1239+
// 'failed' here permanently consumes it — when the cap resets the
1240+
// user is stuck with stranded rows and a dedup'd research path that
1241+
// never re-inserts them. Requeue instead so the keyword retries on
1242+
// the next worker tick once the upstream is back.
1243+
// eslint-disable-next-line @typescript-eslint/no-explicit-any
1244+
const status = (err as any)?.status;
1245+
const isTransient =
1246+
status === 429 ||
1247+
/specified API usage limits|usage limits|rate.?limit/i.test(errMsg);
1248+
if (isTransient) {
1249+
await supabase
1250+
.from("lx_keyword")
1251+
.update({ status: "queued" })
1252+
.eq("id", keyword.id);
1253+
await refundArticleGenerationCharge(supabase, charge);
1254+
console.warn(
1255+
`[lx] keyword ${keyword.id} requeued (transient backend AI error): ${errMsg}`,
1256+
);
1257+
return { ok: false, error: "backend AI transient error" };
1258+
}
1259+
await failKeyword(supabase, keyword.id, `backend AI error: ${errMsg}`);
11531260
await refundArticleGenerationCharge(supabase, charge);
1261+
return { ok: false, error: "backend AI error" };
1262+
}
1263+
1264+
// Link placement, then the content gate. Both produce model-readable
1265+
// violations rather than terminal failures, so attempt 2 can fix them.
1266+
//
1267+
// Exchange links are framed as UP TO (not EXACTLY) in the prompt, so an
1268+
// empty used_exchange_link_urls is a judgement call, not a violation —
1269+
// only a claimed-but-absent or never-offered URL counts.
1270+
const linkViolations = collectLinkViolations(candidate, offeredExchangeUrls);
1271+
1272+
// The gate needs rendered HTML. Render the body without images: the
1273+
// inline-image markers are comments and the hero is not in the body, so
1274+
// the word count, link density, and slop signals are unaffected — and
1275+
// rendering here means a rejected draft never reaches image generation.
1276+
let gateHtml = "";
1277+
try {
1278+
gateHtml = await markdownToHtml(candidate.markdown_body);
1279+
} catch (err) {
1280+
// A body that will not render is not repairable by re-prompting on
1281+
// content grounds, but the next attempt may simply not hit it.
11541282
console.warn(
1155-
`[lx] keyword ${keyword.id} requeued (transient backend AI error): ${errMsg}`,
1283+
`[lx] gate render failed on attempt ${attempt}`,
1284+
err instanceof Error ? err.message : err,
11561285
);
1157-
return { ok: false, error: "backend AI transient error" };
11581286
}
1159-
await failKeyword(supabase, keyword.id, `backend AI error: ${errMsg}`);
1160-
await refundArticleGenerationCharge(supabase, charge);
1161-
return { ok: false, error: "backend AI error" };
1162-
}
11631287

1164-
// Validate that all "used" internal links are actually present in the body.
1165-
const linkCheck = validateInternalLinks(
1166-
article.markdown_body,
1167-
article.used_internal_link_urls,
1168-
);
1169-
if (!linkCheck.ok) {
1170-
await failKeyword(
1171-
supabase,
1172-
keyword.id,
1173-
`model claimed links not present: ${linkCheck.missing.join(", ")}`,
1288+
const result = gateHtml
1289+
? runQualityGate({
1290+
html: gateHtml,
1291+
title: candidate.title,
1292+
metaDescription: candidate.meta_description,
1293+
priorBodies,
1294+
})
1295+
: null;
1296+
1297+
article = candidate;
1298+
gate = result;
1299+
violations = [...linkViolations, ...(result?.violations ?? [])];
1300+
1301+
if (violations.length === 0) break;
1302+
1303+
console.warn(
1304+
`[lx] keyword ${keyword.id} attempt ${attempt}/${MAX_GENERATION_ATTEMPTS} rejected (slop=${result?.score ?? "n/a"}): ${violations.join(" | ")}`,
11741305
);
1175-
await refundArticleGenerationCharge(supabase, charge);
1176-
return { ok: false, error: "internal-link validation failed" };
11771306
}
11781307

1179-
// Validate exchange links similarly. The prompt frames these as UP TO
1180-
// (not EXACTLY) — a 0 here just means the model judged none fit, not a
1181-
// failure. Only flag when the model claims a URL it didn't actually
1182-
// place, or claims a URL we never offered as a candidate.
1183-
const exchangeUrlsUsed = article.used_exchange_link_urls ?? [];
1184-
const offeredExchangeUrls = new Set(exchangeCandidates.map((c) => c.url));
1185-
const phantomExchange = exchangeUrlsUsed.filter(
1186-
(u) => !offeredExchangeUrls.has(u),
1187-
);
1188-
if (phantomExchange.length > 0) {
1189-
await failKeyword(
1190-
supabase,
1191-
keyword.id,
1192-
`model used exchange URLs not in candidate list: ${phantomExchange.join(", ")}`,
1193-
);
1308+
if (!article) {
1309+
// Unreachable — the loop either assigns or returns — but it keeps the
1310+
// narrowing honest for everything below.
1311+
await failKeyword(supabase, keyword.id, "no draft produced");
11941312
await refundArticleGenerationCharge(supabase, charge);
1195-
return { ok: false, error: "exchange-link validation failed" };
1313+
return { ok: false, error: "no draft produced" };
11961314
}
1197-
const exchangeCheck = validateInternalLinks(
1198-
article.markdown_body,
1199-
exchangeUrlsUsed,
1200-
);
1201-
if (!exchangeCheck.ok) {
1315+
1316+
if (violations.length > 0) {
12021317
await failKeyword(
12031318
supabase,
12041319
keyword.id,
1205-
`model claimed exchange links not present: ${exchangeCheck.missing.join(", ")}`,
1320+
`quality gate failed after ${MAX_GENERATION_ATTEMPTS} attempts (slop=${gate?.score ?? "n/a"}): ${violations.join(" | ")}`,
12061321
);
12071322
await refundArticleGenerationCharge(supabase, charge);
1208-
return { ok: false, error: "exchange-link validation failed" };
1323+
return { ok: false, error: "quality gate failed" };
12091324
}
12101325

1326+
const exchangeUrlsUsed = article.used_exchange_link_urls ?? [];
1327+
12111328
// Slugify + dedupe slug.
12121329
const baseSlug = article.slug || slugify(article.title);
12131330
const finalSlug = await uniqueSlug(supabase, typedSite.id, baseSlug);
@@ -1332,6 +1449,11 @@ export async function generateArticle(
13321449
tags: article.tags,
13331450
internal_links: internalLinksPayload,
13341451
outbound_links: outboundLinksPayload,
1452+
// Gate verdict for the draft we accepted. Stored so a site's quality
1453+
// can be tracked over time rather than only enforced at a threshold —
1454+
// a blog whose scores are creeping up is drifting before it fails.
1455+
slop_score: gate?.score ?? null,
1456+
slop_issues: gate ? gate.issues : [],
13351457
status: "ready",
13361458
})
13371459
.select("id")

0 commit comments

Comments
 (0)