Skip to content

Commit 99532f8

Browse files
ralyodioclaude
andauthored
tracker: count people, not beacons — visitor rollup + scripted cap (#263)
"Human visits" was every beacon from a non-crawler user agent: the page view plus four scroll depths, every click and every form submit, three to four per page view. On four properties it read 51,531 in a week while the raw event table held 420 distinct visitor ids in a day, which is also what datafa.st reported for the same sites. Nothing outside the 24h raw table kept a visitor id, so weekly uniques could not be stated at all. - tracker_visitor_daily_stats: one row per (project, UTC day, visitor id) with event and pageview counts and which side of the human/bot line the visitor ended the day on. tracker_touch_visitor upserts it per beacon in one round trip and applies the scripted cap: more than 500 events or 200 page views from one visitor in a day flips it to bot for the day, and the ingest route counts every later beacon from it under bot:scripted. A stock-UA headless browser was the largest "human" source and nothing in the classifier could see it. - tracker_visitor_totals / tracker_visitor_daily_series: exact distinct visitors over a window (and the window before) and per day. - Stats page, /dashboard cards and /dashboard/analytics lead with "Human visitors" and "Page views" from the rollup; the old figure stays as "Human events" with a definition that says what it is. A failed rollup read leaves the tile out rather than showing 0. - /api/tracker/v1/stats totals: visitors and pageviews come from the rollup (null when unreadable, never 0), events is the old number. The CLI line prints all three. totalsFromSeries no longer returns beacons as visitors, so `crawlproof dashboard` cost-per-visitor is finally per visitor. Migration applied to production 2026-09-21 via the Supabase MCP; rows begin that day and the UI captions any window that reaches further back. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
1 parent e28f93b commit 99532f8

13 files changed

Lines changed: 951 additions & 70 deletions

File tree

‎app/(app)/dashboard/analytics/page.tsx‎

Lines changed: 67 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -30,8 +30,17 @@ import {
3030
BOTS_LABEL,
3131
HUMANS_DEFINITION,
3232
HUMANS_LABEL,
33+
VISITORS_DEFINITION,
34+
VISITORS_LABEL,
3335
humansFrom,
3436
} from "@/lib/tracker/humans";
37+
import {
38+
VISITORS_CAPTION,
39+
fetchVisitorTotals,
40+
sumVisitorTotals,
41+
visitorsPartial,
42+
type VisitorTotals,
43+
} from "@/lib/tracker/visitors";
3544
import {
3645
TrackerAnalytics,
3746
type TrackerListItem,
@@ -71,14 +80,18 @@ type PortfolioProject = {
7180
organization_id?: string | null;
7281
};
7382

74-
// Every figure on this page leads with humans (bucket not `bot:`, AI referrals
75-
// included) and shows bot crawls apart. See lib/tracker/humans.ts.
83+
// Every figure on this page leads with people — distinct visitors from the
84+
// visitor rollup — then human EVENTS (bucket not `bot:`, AI referrals
85+
// included; several beacons per page view) and shows bot crawls apart. See
86+
// lib/tracker/humans.ts and lib/tracker/visitors.ts.
7687
type ProjectRow = {
7788
project: PortfolioProject;
7889
totals: ProjectTotals;
79-
/** Human visits, this window vs the one before. */
90+
/** Distinct human visitors this window vs the one before; null when the rollup is unavailable. */
91+
visitors: VisitorTotals | null;
92+
/** Human events, this window vs the one before. */
8093
trend: Trend;
81-
/** Daily human visits for the sparkline; null when over the row budget. */
94+
/** Daily human events for the sparkline; null when over the row budget. */
8295
samples: number[] | null;
8396
};
8497

@@ -217,6 +230,22 @@ export default async function PortfolioAnalyticsPage({
217230
const portfolio = sumTotals(totalsByProject.values());
218231
const trends = totalsTrends(portfolio);
219232

233+
// People. One more RPC, kept apart from the eleven above because its
234+
// failure means "no visitor figure", not "no traffic": the tile and the
235+
// column are left out rather than zeroed.
236+
const visitorsByProject = await fetchVisitorTotals(supabase, projectIds, days, "human");
237+
const portfolioVisitors = visitorsByProject
238+
? sumVisitorTotals(visitorsByProject.values())
239+
: null;
240+
const visitorsTrend = portfolioVisitors
241+
? computeTrend(portfolioVisitors.visitors, portfolioVisitors.prevVisitors)
242+
: null;
243+
const visitorsCaption = !visitorsByProject
244+
? "Visitor counts are unavailable right now; the figures below count events."
245+
: visitorsPartial(days)
246+
? VISITORS_CAPTION
247+
: null;
248+
220249
// Ranked by current-window HUMAN volume: the biggest properties get the
221250
// chart bands, and the daily-detail budget is spent on them first. Ranking
222251
// by events would hand the top band to whichever site a crawler is hitting.
@@ -264,6 +293,9 @@ export default async function PortfolioAnalyticsPage({
264293
return {
265294
project,
266295
totals,
296+
visitors: visitorsByProject
297+
? (visitorsByProject.get(project.id) ?? { visitors: 0, prevVisitors: 0, pageviews: 0, prevPageviews: 0 })
298+
: null,
267299
trend: computeTrend(totals.humans, totals.prevHumans),
268300
samples: byDay ? axis.map((day) => byDay.get(day) ?? 0) : null,
269301
};
@@ -444,11 +476,19 @@ export default async function PortfolioAnalyticsPage({
444476
</p>
445477
</section>
446478

447-
<div className="grid gap-3 sm:grid-cols-2 lg:grid-cols-4">
479+
<div className="grid gap-3 sm:grid-cols-2 lg:grid-cols-5">
480+
{visitorsTrend && (
481+
<TrendMetric
482+
label={VISITORS_LABEL}
483+
trend={visitorsTrend}
484+
tone="accent"
485+
hint={VISITORS_DEFINITION}
486+
/>
487+
)}
448488
<TrendMetric
449489
label={HUMANS_LABEL}
450490
trend={trends.humans}
451-
tone="accent"
491+
tone={visitorsTrend ? "muted" : "accent"}
452492
hint={HUMANS_DEFINITION}
453493
/>
454494
<TrendMetric
@@ -470,6 +510,9 @@ export default async function PortfolioAnalyticsPage({
470510
hint={ALL_EVENTS_DEFINITION}
471511
/>
472512
</div>
513+
{visitorsCaption && (
514+
<p className="-mt-1 text-xs text-[var(--color-muted)]">{visitorsCaption}</p>
515+
)}
473516

474517
{portfolio.events === 0 && portfolio.prevEvents === 0 ? (
475518
<section className="card p-4">
@@ -485,13 +528,21 @@ export default async function PortfolioAnalyticsPage({
485528
<div>
486529
<h2 className="text-lg font-semibold">Portfolio trend</h2>
487530
<p className="text-sm text-[var(--color-muted)]">
488-
Daily human visits, stacked by property. Bot crawls across
531+
Daily human events, stacked by property. Bot crawls across
489532
every property are the dashed line, kept out of the stack.
490533
</p>
491534
</div>
492535
<span className="text-xs text-[var(--color-muted)]">
536+
{portfolioVisitors && (
537+
<>
538+
<span title={VISITORS_DEFINITION}>
539+
{portfolioVisitors.visitors.toLocaleString()} human visitors
540+
</span>
541+
{" · "}
542+
</>
543+
)}
493544
<span title={HUMANS_DEFINITION}>
494-
{portfolio.humans.toLocaleString()} human visits
545+
{portfolio.humans.toLocaleString()} human events
495546
</span>
496547
{" · "}
497548
<span title={BOTS_DEFINITION}>
@@ -633,8 +684,11 @@ function ProjectTrendTable({ rows }: { rows: ProjectRow[] }) {
633684
<thead>
634685
<tr className="border-b border-[var(--color-border)] text-left text-xs text-[var(--color-muted)]">
635686
<th className="py-2 pr-3 font-medium">Property</th>
687+
<th className="py-2 pr-3 text-right font-medium" title={VISITORS_DEFINITION}>
688+
Visitors
689+
</th>
636690
<th className="py-2 pr-3 text-right font-medium" title={HUMANS_DEFINITION}>
637-
Humans
691+
Events
638692
</th>
639693
<th className="py-2 pr-3 text-right font-medium" title={HUMANS_DEFINITION}>
640694
Previous
@@ -650,7 +704,7 @@ function ProjectTrendTable({ rows }: { rows: ProjectRow[] }) {
650704
</tr>
651705
</thead>
652706
<tbody>
653-
{rows.map(({ project, totals, trend, samples }) => (
707+
{rows.map(({ project, totals, visitors, trend, samples }) => (
654708
<tr
655709
key={project.id}
656710
className="border-b border-[var(--color-border)] last:border-0"
@@ -676,6 +730,9 @@ function ProjectTrendTable({ rows }: { rows: ProjectRow[] }) {
676730
</span>
677731
</Link>
678732
</td>
733+
<td className="py-2 pr-3 text-right tabular-nums" title={VISITORS_DEFINITION}>
734+
{visitors ? visitors.visitors.toLocaleString() : "—"}
735+
</td>
679736
<td className="py-2 pr-3 text-right tabular-nums" title={HUMANS_DEFINITION}>
680737
{totals.humans.toLocaleString()}
681738
</td>

‎app/(app)/dashboard/page.tsx‎

Lines changed: 39 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@ import { createClient } from "@/lib/supabase/server";
33
import { rpcFailed, type Loaded } from "@/lib/loaded";
44
import { ScoreBadge } from "@/components/score-badge";
55
import { FontSparkline } from "@/components/font-sparkline";
6-
import { BOTS_DEFINITION, HUMANS_DEFINITION } from "@/lib/tracker/humans";
6+
import { BOTS_DEFINITION, HUMANS_DEFINITION, VISITORS_DEFINITION } from "@/lib/tracker/humans";
7+
import { fetchVisitorDailySeries } from "@/lib/tracker/visitors";
78
import { ProjectLogo } from "@/components/project-logo";
89
import { StatsUnavailable } from "@/components/stats-unavailable";
910
import { backfillProjectLogo } from "@/app/actions/createProject";
@@ -121,11 +122,12 @@ export default async function DashboardPage({
121122
// project has an lx_site row in status=active; social is "on" when at
122123
// least one social account is linked at the project level.
123124
const projectIds = (projects ?? []).map((p) => p.id);
124-
const [autoblogIds, socialIds, latestPosts, traffic] = await Promise.all([
125+
const [autoblogIds, socialIds, latestPosts, traffic, visitorsByProject] = await Promise.all([
125126
fetchEnabledProjectIds(supabase, "lx_site", projectIds, { status: "active" }),
126127
fetchEnabledProjectIds(supabase, "sp_site_account", projectIds),
127128
fetchLatestBlogPostByProject(supabase, projectIds),
128129
fetchSevenDayTraffic(supabase, projectIds),
130+
fetchSevenDayVisitors(supabase, projectIds),
129131
]);
130132
const trafficByProject = traffic.data;
131133
const trafficFailed = traffic.failed;
@@ -266,23 +268,27 @@ export default async function DashboardPage({
266268
<div>
267269
<div
268270
className="text-xs font-medium text-[var(--color-fg)]"
269-
title={trafficFailed ? undefined : HUMANS_DEFINITION}
271+
title={trafficFailed ? undefined : visitorsByProject ? VISITORS_DEFINITION : HUMANS_DEFINITION}
270272
>
271273
{trafficFailed
272274
? "Traffic unavailable"
273-
: `${totalHumans(trafficByProject.get(p.id) ?? []).toLocaleString()} human visits`}
275+
: visitorsByProject
276+
? `${sum(visitorsByProject.get(p.id)).toLocaleString()} human visitors`
277+
: `${totalHumans(trafficByProject.get(p.id) ?? []).toLocaleString()} human events`}
274278
</div>
275279
<div
276280
className="text-[11px] text-[var(--color-muted)]"
277-
title={trafficFailed ? undefined : BOTS_DEFINITION}
281+
title={trafficFailed ? undefined : `${HUMANS_DEFINITION} ${BOTS_DEFINITION}`}
278282
>
279283
{trafficFailed
280284
? "Query failed \u2014 not zero"
281-
: `${totalBots(trafficByProject.get(p.id) ?? []).toLocaleString()} bot hits \u00b7 Past 7 days`}
285+
: `${visitorsByProject ? `${totalHumans(trafficByProject.get(p.id) ?? []).toLocaleString()} human events \u00b7 ` : ""}${totalBots(trafficByProject.get(p.id) ?? []).toLocaleString()} bot hits \u00b7 Past 7 days`}
282286
</div>
283287
</div>
284288
{!trafficFailed && (
285-
<FontSparkline samples={humanSamples(trafficByProject.get(p.id))} />
289+
<FontSparkline
290+
samples={visitorsByProject?.get(p.id) ?? humanSamples(trafficByProject.get(p.id))}
291+
/>
286292
)}
287293
</div>
288294
{orgSchemaReady && (
@@ -436,6 +442,32 @@ async function fetchSevenDayTraffic(
436442
return { data: out, failed: false };
437443
}
438444

445+
// Distinct human visitors per day for the past seven UTC days, per project:
446+
// the number the card leads with. The visitor rollup only began on
447+
// lib/tracker/visitors.ts VISITORS_SINCE, and it counts people where
448+
// dashboard_project_traffic counts beacons (several per page view), which
449+
// is why the two figures on a card differ by a hundredfold and both are
450+
// shown. null when the RPC failed or is not deployed: the card then leads
451+
// with events under their real name rather than showing "0 visitors".
452+
async function fetchSevenDayVisitors(
453+
supabase: Awaited<ReturnType<typeof createClient>>,
454+
projectIds: string[],
455+
): Promise<Map<string, number[]> | null> {
456+
const days = lastSevenDays();
457+
const byProject = await fetchVisitorDailySeries(supabase, projectIds, 7, "human");
458+
if (!byProject) return null;
459+
const out = new Map<string, number[]>();
460+
for (const projectId of projectIds) {
461+
const byDay = byProject.get(projectId);
462+
out.set(projectId, days.map((day) => byDay?.get(day) ?? 0));
463+
}
464+
return out;
465+
}
466+
467+
function sum(samples: number[] | undefined) {
468+
return (samples ?? []).reduce((total, n) => total + n, 0);
469+
}
470+
439471
function lastSevenDays() {
440472
return Array.from({ length: 7 }, (_, index) => {
441473
const date = new Date();

‎app/(app)/dashboard/projects/[id]/stats/page.tsx‎

Lines changed: 33 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,14 @@ import {
88
TrackerAnalytics,
99
type TrackerPanels,
1010
} from "@/components/charts/tracker-analytics";
11-
import { fetchPanels, PANEL_KEYS } from "@/lib/tracker/panels";
11+
import { fetchPanels, PANEL_KEYS, resolveDays, rollupDays } from "@/lib/tracker/panels";
1212
import { DEFAULT_TRACKER_RANGE, trackerRange } from "@/lib/tracker/ranges";
1313
import { headlineTiles, whoOrDefault, whoToKind } from "@/lib/tracker/who";
14+
import {
15+
VISITORS_CAPTION,
16+
fetchVisitorTotals,
17+
visitorsPartial,
18+
} from "@/lib/tracker/visitors";
1419
import { InstallSnippet } from "./install-snippet";
1520
import { TrackerToggle } from "./tracker-toggle";
1621
import { CareersToggle } from "./careers-toggle";
@@ -65,31 +70,41 @@ export default async function ProjectStatsPage({
6570
// Every panel is fetched at the toggle's `kind`, so a card and the tiles
6671
// above it never disagree on who they are counting.
6772
const range = trackerRange(DEFAULT_TRACKER_RANGE);
68-
const panels = (await fetchPanels(
69-
supabase,
70-
id,
71-
PANEL_KEYS,
72-
range,
73-
kind,
74-
)) as unknown as TrackerPanels;
73+
// The visitor rollup is day-resolution, so a sub-day range reads today's
74+
// rollup — the same rule the device and exit-page cards apply.
75+
const visitorDays = rollupDays(range, await resolveDays(supabase, id, range));
76+
const [panels, visitorTotals] = await Promise.all([
77+
fetchPanels(supabase, id, PANEL_KEYS, range, kind) as unknown as Promise<TrackerPanels>,
78+
fetchVisitorTotals(supabase, [id], visitorDays, kind),
79+
]);
7580

76-
// Headline metrics come straight from the series so they stay exact even
77-
// though Top sources below is truncated to the top 10 buckets. The page
78-
// leads with humans (every bucket that is not `bot:`, AI referrals
79-
// included) and shows bot crawls apart — see lib/tracker/humans.ts for why
80-
// the old bot-inclusive total is no longer a headline. Under Humans or
81-
// Bots only that side's tiles are shown (lib/tracker/who.ts).
81+
// Headline metrics: the people first. Distinct visitors and their page
82+
// views come from the visitor rollup (lib/tracker/visitors.ts); the event
83+
// figures come straight from the series so they stay exact even though Top
84+
// sources below is truncated to the top 10 buckets. The event count used to
85+
// be labelled "Human visits" and ran ~100x above the number of people — it
86+
// is stats.js firing several beacons per page view — so it now sits after
87+
// the visitor tiles under its real name. Under Humans or Bots only that
88+
// side's tiles are shown (lib/tracker/who.ts).
8289
const points = panels.series.points;
8390
const totalAi = points.reduce((s, p) => s + p.ai, 0);
8491
const totalBot = points.reduce((s, p) => s + p.bots, 0);
8592
const totalHuman = points.reduce((s, p) => s + p.humans, 0);
8693
const grandTotal = points.reduce((s, p) => s + p.events, 0);
8794
const eventTotal = points.reduce((s, p) => s + p.pageviews + p.interactions, 0);
95+
const visitors = visitorTotals?.get(id) ?? (visitorTotals ? { visitors: 0, pageviews: 0 } : null);
8896
const tiles = headlineTiles(who, {
97+
visitors: visitors ? visitors.visitors : null,
98+
pageviews: visitors ? visitors.pageviews : null,
8999
humans: totalHuman,
90100
ai: totalAi,
91101
bots: totalBot,
92102
});
103+
const visitorsCaption = visitors === null
104+
? "Visitor counts are unavailable right now; the figures below count events."
105+
: visitorsPartial(visitorDays)
106+
? VISITORS_CAPTION
107+
: null;
93108

94109
// Older projects have rollup rows in tracker_daily_stats but nothing in
95110
// tracker_event_daily_stats, which would leave Event mix empty on a page
@@ -265,7 +280,7 @@ export default async function ProjectStatsPage({
265280
<WhoToggle value={who} />
266281
</div>
267282

268-
<div className="grid gap-3 sm:grid-cols-3">
283+
<div className="grid gap-3 sm:grid-cols-2 lg:grid-cols-4">
269284
{tiles.map((tile) => (
270285
<Metric
271286
key={tile.key}
@@ -276,6 +291,9 @@ export default async function ProjectStatsPage({
276291
/>
277292
))}
278293
</div>
294+
{visitorsCaption && (
295+
<p className="-mt-1 text-xs text-[var(--color-muted)]">{visitorsCaption}</p>
296+
)}
279297

280298
{grandTotal === 0 && eventTotal === 0 ? (
281299
<section className="card p-4">

0 commit comments

Comments
 (0)