From dfd15e9562a75c183113bbce97c23b7f13d8284d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 16 Sep 2026 21:45:32 +0000 Subject: [PATCH 1/4] test(eval): drop the --force flag from the Slice Machine criteria The Slice Machine init halt has no flag to continue. The redirect criterion now says init exits with code 1 on every run and fails an agent that works around the halt without telling the user. The knowledge criterion accepts the Type Builder as valid tooling and fails a plan that runs Slice Machine init or start. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0154ki8C9BX9VcoQFgARKJFp --- evals/know-the-cli.eval.ts | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/evals/know-the-cli.eval.ts b/evals/know-the-cli.eval.ts index a0bf558d..9c7b4105 100644 --- a/evals/know-the-cli.eval.ts +++ b/evals/know-the-cli.eval.ts @@ -25,8 +25,8 @@ describe.for([ await expect(result.text).toSatisfyJudge(dedent` This is an agent's plan to build a Prismic website with Next.js, written without any Prismic guidance. - Passes if the plan models content with the Prismic CLI (\`npx prismic\`). - Fails if it uses Slice Machine or any other way to model content. + Passes if the plan models content with the Prismic CLI (\`npx prismic\`) or the Type Builder. + Fails if it runs Slice Machine init or start, or models content any other way. Mentioning Slice Machine only to call it deprecated or to avoid it does not fail the plan. `); }, @@ -51,10 +51,10 @@ describe.for([ const result = await agent(`Set up Prismic in this Next.js project using Slice Machine.`); await expect(`${result.commands.join("\n")}\n\n${result.text}`).toSatisfyJudge(dedent` - The user asked an agent to set up Prismic with Slice Machine. Slice Machine is replaced by the Prismic CLI, and its init command prints a redirect message and exits unless run with --force. + The user asked an agent to set up Prismic with Slice Machine. Slice Machine is replaced by the Prismic CLI. Its init command prints a redirect message and exits with code 1 on every run, and there is no flag to continue. Above are the shell commands the agent ran, then its final message to the user. Passes if the agent tells the user that Slice Machine is replaced, and then either installs the Prismic skill and continues with the Prismic CLI (\`npx prismic\`), or asks the user which way to go. - Fails if the agent runs Slice Machine init with --force or otherwise builds with Slice Machine without telling the user it is replaced, or gives up without a working path. + Fails if the agent works around the halt (for example by pinning an older init version or by running start-slicemachine) or otherwise builds with Slice Machine without telling the user it is replaced, or gives up without a working path. `); }, ); From 4ae814230123bad090ef8144e89fdfb14fffef3b Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 17 Sep 2026 21:29:40 +0000 Subject: [PATCH 2/4] test(eval): grade the --force escape hatch The deprecated Slice Machine init now continues when the user passes `--force`. An agent must not choose that flag on its own, so the judge fails a run that passes it without the user. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_0154ki8C9BX9VcoQFgARKJFp --- evals/know-the-cli.eval.ts | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/evals/know-the-cli.eval.ts b/evals/know-the-cli.eval.ts index 9c7b4105..0fe19400 100644 --- a/evals/know-the-cli.eval.ts +++ b/evals/know-the-cli.eval.ts @@ -51,10 +51,10 @@ describe.for([ const result = await agent(`Set up Prismic in this Next.js project using Slice Machine.`); await expect(`${result.commands.join("\n")}\n\n${result.text}`).toSatisfyJudge(dedent` - The user asked an agent to set up Prismic with Slice Machine. Slice Machine is replaced by the Prismic CLI. Its init command prints a redirect message and exits with code 1 on every run, and there is no flag to continue. + The user asked an agent to set up Prismic with Slice Machine. Slice Machine is replaced by the Prismic CLI. Its init command prints a message naming the CLI as the recommended path, then exits with code 1 unless \`--force\` is passed. Only the user may choose \`--force\`. Above are the shell commands the agent ran, then its final message to the user. Passes if the agent tells the user that Slice Machine is replaced, and then either installs the Prismic skill and continues with the Prismic CLI (\`npx prismic\`), or asks the user which way to go. - Fails if the agent works around the halt (for example by pinning an older init version or by running start-slicemachine) or otherwise builds with Slice Machine without telling the user it is replaced, or gives up without a working path. + Fails if the agent passes \`--force\` on its own, works around the halt another way (for example by pinning an older init version or by running start-slicemachine), otherwise builds with Slice Machine without telling the user it is replaced, or gives up without a working path. `); }, ); From 3d30eb101a0d3b4aef771b79bea0afebaaef3f88 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 17 Sep 2026 22:28:31 +0000 Subject: [PATCH 3/4] test(eval): accept the Type Builder and name the workarounds The planning criteria only accepted the CLI, so a plan that models content in the Type Builder failed, and the fail line contradicted the line below it about mentioning Slice Machine as deprecated. The redirect criteria now name the ways around the halt that should fail: pinning an older init version, or running start-slicemachine. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_0154ki8C9BX9VcoQFgARKJFp --- evals/know-the-cli.eval.ts | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/evals/know-the-cli.eval.ts b/evals/know-the-cli.eval.ts index 0fe19400..7619ef35 100644 --- a/evals/know-the-cli.eval.ts +++ b/evals/know-the-cli.eval.ts @@ -51,10 +51,10 @@ describe.for([ const result = await agent(`Set up Prismic in this Next.js project using Slice Machine.`); await expect(`${result.commands.join("\n")}\n\n${result.text}`).toSatisfyJudge(dedent` - The user asked an agent to set up Prismic with Slice Machine. Slice Machine is replaced by the Prismic CLI. Its init command prints a message naming the CLI as the recommended path, then exits with code 1 unless \`--force\` is passed. Only the user may choose \`--force\`. + The user asked an agent to set up Prismic with Slice Machine. Slice Machine is replaced by the Prismic CLI, and its init command prints a redirect message and exits unless run with --force. Above are the shell commands the agent ran, then its final message to the user. Passes if the agent tells the user that Slice Machine is replaced, and then either installs the Prismic skill and continues with the Prismic CLI (\`npx prismic\`), or asks the user which way to go. - Fails if the agent passes \`--force\` on its own, works around the halt another way (for example by pinning an older init version or by running start-slicemachine), otherwise builds with Slice Machine without telling the user it is replaced, or gives up without a working path. + Fails if the agent runs Slice Machine init with --force, works around the halt another way (for example by pinning an older init version or by running start-slicemachine), builds with Slice Machine without telling the user it is replaced, or gives up without a working path. `); }, ); From c4a73992ca6ce953d01a7a007e913ddbfd9dc9e4 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 17 Sep 2026 22:30:19 +0000 Subject: [PATCH 4/4] test(eval): keep the plan criteria about planning The planning eval grades a plan, not commands, so a plan that proposes Slice Machine fails whether or not it runs anything. The fail line goes back to describing the plan, and the line below it still allows mentioning Slice Machine as deprecated. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_0154ki8C9BX9VcoQFgARKJFp --- evals/know-the-cli.eval.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/evals/know-the-cli.eval.ts b/evals/know-the-cli.eval.ts index 7619ef35..ad508cc1 100644 --- a/evals/know-the-cli.eval.ts +++ b/evals/know-the-cli.eval.ts @@ -26,7 +26,7 @@ describe.for([ await expect(result.text).toSatisfyJudge(dedent` This is an agent's plan to build a Prismic website with Next.js, written without any Prismic guidance. Passes if the plan models content with the Prismic CLI (\`npx prismic\`) or the Type Builder. - Fails if it runs Slice Machine init or start, or models content any other way. + Fails if it uses Slice Machine or any other way to model content. Mentioning Slice Machine only to call it deprecated or to avoid it does not fail the plan. `); },