[experiments] Daily Experiment Report — 2026-09-23 #62905

2026-09-23T08:19:18Z

github-actions[bot]
Bot Sep 23, 2026

🧪 Daily Experiment Report — 2026-09-23

51 named A/B experiments across 48 workflows were analysed: 24 are 🟢 READY (all variants ≥ min_samples) and 27 are 🟡 COLLECTING. Every core decision came back EXTEND (37) or INCONCLUSIVE (14) — no PROMOTE/REJECT yet, because outcome metric wiring (native run_success_rate/empty_output_rate guardrails and multi-variant (>2) decisions) is not fully supported by the core analysis layer yet.

⚡ Quick Stats

Metric	Value
Active experiments	51 (48 workflows)
Ready for analysis	24
Decisions with sufficient evidence (READY)	24
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	1 (smokecopilot: caveman × subagent_model)

`prompt_compression` · `agent-performance-analyzer.md`

Readiness: 🟢 READY
Variants: caveman vs verbose · Window: last 30 runs · Analysed: 124 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=0.3727331778060773

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`caveman`	67	██████████ 67/20 (100%)
`verbose`	57	██████████ 57/20 (100%)

Experiment : prompt_compression
Workflow   : agent-performance-analyzer.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 124 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| caveman          |    5 |    20.0% |          725.6 |    [88.5 , 1362.7] | ████████ 67/20 (100%) |
| verbose          |    5 |    60.0% |          815.4 |   [329.9 , 1300.9] | ████████ 57/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=0.3727331778060773

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`sub_agent_strategy` · `agent-persona-explorer.md`

Readiness: 🟢 READY
Variants: batch vs per_scenario · Window: last 30 runs · Analysed: 124 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=0.7183582132689105

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`batch`	64	██████████ 64/20 (100%)
`per_scenario`	60	██████████ 60/20 (100%)

Experiment : sub_agent_strategy
Workflow   : agent-persona-explorer.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 124 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| batch            |    4 |    50.0% |          543.2 |    [237.0 , 849.5] | ████████ 64/20 (100%) |
| per_scenario     |    6 |    33.3% |          487.7 |    [271.4 , 704.0] | ████████ 60/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=0.7183582132689105

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`tone_variant` · `aw-failure-investigator.md`

Readiness: 🟢 READY
Variants: assertive vs clinical vs narrative · Window: last 30 runs · Analysed: 452 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=0.9536805665059491

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`assertive`	152	██████████ 152/20 (100%)
`clinical`	152	██████████ 152/20 (100%)
`narrative`	148	██████████ 148/20 (100%)

Experiment : tone_variant
Workflow   : aw-failure-investigator.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 452 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| assertive        |    3 |    66.7% |          579.7 |    [288.6 , 870.7] | ████████ 152/20 (100%) |
| clinical         |    4 |    50.0% |          637.2 |    [377.3 , 897.2] | ████████ 152/20 (100%) |
| narrative        |    3 |   100.0% |          572.7 |    [361.1 , 784.2] | ████████ 148/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=0.9536805665059491

Guardrails: configured=False, passed=True

Core decision : INCONCLUSIVE
Reason code   : unsupported_multi_variant
Rationale     : automatic decisions currently require exactly two variants
Report action : INCONCLUSIVE

Core decision: INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

`tone_variant` · `breaking-change-checker.md`

Readiness: 🟢 READY
Variants: neutral vs urgent · Window: last 30 runs · Analysed: 48 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=1

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`neutral`	24	██████████ 24/20 (100%)
`urgent`	24	██████████ 24/20 (100%)

Experiment : tone_variant
Workflow   : breaking-change-checker.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 48 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| neutral          |    2 |     0.0% |          400.5 |    [-37.9 , 838.9] | ████████ 24/20 (100%) |
| urgent           |    8 |    25.0% |          784.0 |    [572.7 , 995.3] | ████████ 24/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=1

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`prompt_style` · `ci-coach.md`

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 93 runs with artifacts
min_samples: 20 per variant · Evidence: mann_whitney, balance p=0

H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`concise`	37	██████████ 37/20 (100%)
`detailed`	56	██████████ 56/20 (100%)

Experiment : prompt_style
Workflow   : ci-coach.md
Hypothesis : H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality
Window     : last 30 runs  |  Analysed: 93 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| concise          |    0 |      N/A |            N/A |                N/A | ████████ 37/20 (100%) |
| detailed         |   10 |    70.0% |         1437.8 |   [956.3 , 1919.3] | ████████ 56/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: mann_whitney, balance p_value=0

Guardrails:
  run_success_rate >=0.85 : UNSUPPORTED
  empty_output_rate <=0.05 : UNSUPPORTED

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

`sub_agent_strategy` · `daily-agentrx-trace-optimizer.md`

Readiness: 🟢 READY
Variants: single_agent vs sub_agents · Window: last 30 runs · Analysed: 82 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=1

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`single_agent`	41	██████████ 41/20 (100%)
`sub_agents`	41	██████████ 41/20 (100%)

Experiment : sub_agent_strategy
Workflow   : daily-agentrx-trace-optimizer.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 82 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| single_agent     |    5 |    40.0% |          827.2 |   [101.2 , 1553.2] | ████████ 41/20 (100%) |
| sub_agents       |    5 |    40.0% |          995.2 |   [-16.0 , 2006.4] | ████████ 41/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=1

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`prompt_style` · `daily-astrostylelite-markdown-spellcheck.md`

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 142 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=0.8424226210279355

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`concise`	72	██████████ 72/20 (100%)
`detailed`	70	██████████ 70/20 (100%)

Experiment : prompt_style
Workflow   : daily-astrostylelite-markdown-spellcheck.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 142 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| concise          |    4 |    50.0% |          457.0 |    [255.5 , 658.5] | ████████ 72/20 (100%) |
| detailed         |    6 |    33.3% |          421.0 |    [301.7 , 540.3] | ████████ 70/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=0.8424226210279355

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`prompt_style` · `daily-community-attribution.md`

Readiness: 🟢 READY
Variants: concise vs verbose · Window: last 30 runs · Analysed: 140 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=1

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`concise`	70	██████████ 70/20 (100%)
`verbose`	70	██████████ 70/20 (100%)

Experiment : prompt_style
Workflow   : daily-community-attribution.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 140 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| concise          |    5 |    60.0% |          510.8 |    [256.5 , 765.1] | ████████ 70/20 (100%) |
| verbose          |    5 |    40.0% |          433.2 |    [179.6 , 686.8] | ████████ 70/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=1

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`reasoning_depth` · `daily-fact.md`

Readiness: 🟢 READY
Variants: multi_candidate vs single_pass · Window: last 30 runs · Analysed: 93 runs with artifacts
min_samples: 30 per variant · Evidence: chi_square_balance, balance p=0.11548536021444333

H0: no change in discussion engagement rate. H1: multi_candidate produces more novel verses with higher reaction counts (expected +20% reactions).

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`multi_candidate`	39	██████████ 39/30 (100%)
`single_pass`	54	██████████ 54/30 (100%)

Experiment : reasoning_depth
Workflow   : daily-fact.md
Hypothesis : H0: no change in discussion engagement rate. H1: multi_candidate produces more novel verses with higher reaction counts (expected +20% reactions).
Window     : last 30 runs  |  Analysed: 93 runs (state branch)
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| multi_candidate  |    6 |     0.0% |          357.5 |    [300.3 , 414.7] | ████████ 39/30 (100%) |
| single_pass      |    4 |     0.0% |          442.8 |    [192.0 , 693.5] | ████████ 54/30 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=0.11548536021444333

Guardrails:
  empty_output_rate <0.05 : UNSUPPORTED
  run_success_rate >=0.95 : UNSUPPORTED

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "empty_output_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "empty_output_rate" is not backed by a supported metric observation

`prompt_style` · `daily-news.md`

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 109 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=0.392995762948708

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`concise`	59	██████████ 59/20 (100%)
`detailed`	50	██████████ 50/20 (100%)

Experiment : prompt_style
Workflow   : daily-news.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 109 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| concise          |    6 |   100.0% |          740.5 |    [688.8 , 792.2] | ████████ 59/20 (100%) |
| detailed         |    4 |    75.0% |          713.8 |    [616.0 , 811.5] | ████████ 50/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=0.392995762948708

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`reasoning_depth` · `daily-security-red-team.md`

Readiness: 🟢 READY
Variants: iterative vs single_pass · Window: last 30 runs · Analysed: 133 runs with artifacts
min_samples: 30 per variant · Evidence: proportion_test, balance p=0.018294828071909356

H0: no change in finding quality. H1: iterative reduces false-positive rate by >=20% at <=30% token overhead

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`iterative`	80	██████████ 80/30 (100%)
`single_pass`	53	██████████ 53/30 (100%)

Experiment : reasoning_depth
Workflow   : daily-security-red-team.md
Hypothesis : H0: no change in finding quality. H1: iterative reduces false-positive rate by >=20% at <=30% token overhead
Window     : last 30 runs  |  Analysed: 133 runs (state branch)
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| iterative        |    5 |     0.0% |         3817.8 |  [3783.5 , 3852.1] | ████████ 80/30 (100%) |
| single_pass      |    5 |     0.0% |         3823.4 |  [3780.9 , 3865.9] | ████████ 53/30 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: proportion_test, balance p_value=0.018294828071909356

Guardrails:
  run_success_rate >=0.9 : UNSUPPORTED
  empty_output_rate <=0.05 : UNSUPPORTED

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

`output_format` · `deep-report.md`

Readiness: 🟢 READY
Variants: annotated_brief vs executive_brief vs full_briefing vs ste · Window: last 30 runs · Analysed: 221 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=9.839986467048277e-05

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`annotated_brief`	53	██████████ 53/20 (100%)
`executive_brief`	80	██████████ 80/20 (100%)
`full_briefing`	57	██████████ 57/20 (100%)
`ste`	31	██████████ 31/20 (100%)

Experiment : output_format
Workflow   : deep-report.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 221 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| annotated_brief  |    2 |   100.0% |         1096.5 |  [-434.6 , 2627.6] | ████████ 53/20 (100%) |
| executive_brief  |    3 |    66.7% |          982.7 |  [-228.2 , 2193.6] | ████████ 80/20 (100%) |
| full_briefing    |    3 |    66.7% |          806.7 |   [-12.7 , 1626.0] | ████████ 57/20 (100%) |
| ste              |    2 |    50.0% |         1010.5 | [-5933.4 , 7954.4] | ████████ 31/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=9.839986467048277e-05

Guardrails: configured=False, passed=True

Core decision : INCONCLUSIVE
Reason code   : unsupported_multi_variant
Rationale     : automatic decisions currently require exactly two variants
Report action : INCONCLUSIVE

Core decision: INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

`tool_verbosity` · `gpclean.md`

Readiness: 🟢 READY
Variants: full_bash vs minimal_toolset · Window: last 30 runs · Analysed: 121 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=0.6538877484758192

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`full_bash`	58	██████████ 58/20 (100%)
`minimal_toolset`	63	██████████ 63/20 (100%)

Experiment : tool_verbosity
Workflow   : gpclean.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 121 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| full_bash        |    6 |     0.0% |          296.2 |    [251.0 , 341.3] | ████████ 58/20 (100%) |
| minimal_toolset  |    4 |     0.0% |          320.5 |    [250.2 , 390.8] | ████████ 63/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=0.6538877484758192

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`prompt_style` · `issue-arborist.md`

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 80 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=0.37473300936931037

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`concise`	44	██████████ 44/20 (100%)
`detailed`	36	██████████ 36/20 (100%)

Experiment : prompt_style
Workflow   : issue-arborist.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 80 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| concise          |    6 |    33.3% |          359.0 |    [290.6 , 427.4] | ████████ 44/20 (100%) |
| detailed         |    4 |     0.0% |          303.8 |    [280.4 , 327.1] | ████████ 36/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=0.37473300936931037

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`sub_agent_strategy` · `smokeantigravity`

Readiness: 🟢 READY
Variants: single_agent vs sub_agents · Window: last 30 runs · Analysed: 208 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=0.4952492560265088

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`single_agent`	99	██████████ 99/20 (100%)
`sub_agents`	109	██████████ 109/20 (100%)

Experiment : sub_agent_strategy
Workflow   : smokeantigravity
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 208 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| single_agent     |    5 |   100.0% |          616.8 |    [234.7 , 998.9] | ████████ 99/20 (100%) |
| sub_agents       |    5 |   100.0% |          720.2 |   [214.8 , 1225.6] | ████████ 109/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=0.4952492560265088

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`caveman` · `smoke-copilot.md`

Readiness: 🟢 READY
Variants: no vs yes · Window: last 30 runs · Analysed: 368 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=1

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`no`	184	██████████ 184/20 (100%)
`yes`	184	██████████ 184/20 (100%)

Experiment : caveman
Workflow   : smoke-copilot.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 368 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| no               |    5 |    40.0% |          809.0 |    [739.0 , 879.0] | ████████ 184/20 (100%) |
| yes              |    5 |    40.0% |          827.4 |    [728.8 , 926.0] | ████████ 184/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=1

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

Report action: EXTEND (interaction_underpowered) — a second concurrent experiment (subagent_model) is running on the same workflow; the caveman×subagent_model factorial cells are sparse (all 4 cells n=2-3, below min_samples=20), so a reporting safety hold is applied instead of trusting the marginal decision alone. See the interaction table below.

`subagent_model` · `smoke-copilot.md`

Readiness: 🟢 READY
Variants: large vs small · Window: last 30 runs · Analysed: 348 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=1

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`large`	174	██████████ 174/20 (100%)
`small`	174	██████████ 174/20 (100%)

Experiment : subagent_model
Workflow   : smoke-copilot.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 348 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| large            |    5 |    40.0% |          793.8 |    [733.0 , 854.6] | ████████ 174/20 (100%) |
| small            |    5 |    40.0% |          842.6 |    [747.9 , 937.3] | ████████ 174/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=1

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`caveman` · `smoke-copilot-aoai-apikey.md`

Readiness: 🟢 READY
Variants: no vs yes · Window: last 30 runs · Analysed: 187 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=0.8995438982119808

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`no`	93	██████████ 93/20 (100%)
`yes`	94	██████████ 94/20 (100%)

Experiment : caveman
Workflow   : smoke-copilot-aoai-apikey.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 187 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| no               |    5 |    20.0% |          566.8 |    [238.1 , 895.5] | ████████ 93/20 (100%) |
| yes              |    5 |    20.0% |          530.4 |    [373.4 , 687.4] | ████████ 94/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=0.8995438982119808

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`subagent_model` · `smoke-copilot-aoai-apikey.md`

Readiness: 🟢 READY
Variants: large vs small · Window: last 30 runs · Analysed: 187 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=0.8995438982119808

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`large`	94	██████████ 94/20 (100%)
`small`	93	██████████ 93/20 (100%)

Experiment : subagent_model
Workflow   : smoke-copilot-aoai-apikey.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 187 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| large            |    5 |    20.0% |          517.6 |    [358.7 , 676.5] | ████████ 94/20 (100%) |
| small            |    5 |    20.0% |          579.6 |    [255.5 , 903.7] | ████████ 93/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=0.8995438982119808

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`caveman` · `smoke-copilot-aoai-entra.md`

Readiness: 🟢 READY
Variants: no vs yes · Window: last 30 runs · Analysed: 160 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=1

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`no`	80	██████████ 80/20 (100%)
`yes`	80	██████████ 80/20 (100%)

Experiment : caveman
Workflow   : smoke-copilot-aoai-entra.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 160 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| no               |    5 |     0.0% |          631.8 |   [225.0 , 1038.6] | ████████ 80/20 (100%) |
| yes              |    5 |    20.0% |          600.8 |    [390.2 , 811.4] | ████████ 80/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=1

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`subagent_model` · `smoke-copilot-aoai-entra.md`

Readiness: 🟢 READY
Variants: large vs small · Window: last 30 runs · Analysed: 160 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=1

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`large`	80	██████████ 80/20 (100%)
`small`	80	██████████ 80/20 (100%)

Experiment : subagent_model
Workflow   : smoke-copilot-aoai-entra.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 160 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| large            |    5 |     0.0% |          667.2 |   [285.7 , 1048.7] | ████████ 80/20 (100%) |
| small            |    5 |    20.0% |          565.4 |    [330.5 , 800.3] | ████████ 80/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=1

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`sub_agent_strategy` · `smoke-gemini.md`

Readiness: 🟢 READY
Variants: single_agent vs sub_agents · Window: last 30 runs · Analysed: 270 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=0.3321125959772243

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`single_agent`	127	██████████ 127/20 (100%)
`sub_agents`	143	██████████ 143/20 (100%)

Experiment : sub_agent_strategy
Workflow   : smoke-gemini.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 270 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| single_agent     |    6 |     0.0% |          470.0 |     [97.3 , 842.7] | ████████ 127/20 (100%) |
| sub_agents       |    4 |     0.0% |          364.2 |    [327.2 , 401.3] | ████████ 143/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=0.3321125959772243

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`sub_agent_decomposition` · `smoke-pi.md`

Readiness: 🟢 READY
Variants: parallel_sub_agents vs single_agent · Window: last 30 runs · Analysed: 76 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=0.3619306624774411

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`parallel_sub_agents`	42	██████████ 42/20 (100%)
`single_agent`	34	██████████ 34/20 (100%)

Experiment : sub_agent_decomposition
Workflow   : smoke-pi.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 76 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| parallel_sub_agents |    5 |   100.0% |          259.2 |    [144.0 , 374.4] | ████████ 42/20 (100%) |
| single_agent     |    5 |   100.0% |          198.8 |    [173.4 , 224.2] | ████████ 34/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=0.3619306624774411

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`tone_style` · `typist.md`

Readiness: 🟢 READY
Variants: conversational vs formal · Window: last 30 runs · Analysed: 86 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square_balance, balance p=0.6696360831840533

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
`conversational`	45	██████████ 45/20 (100%)
`formal`	41	██████████ 41/20 (100%)

Experiment : tone_style
Workflow   : typist.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 86 runs (state branch)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| conversational   |    3 |    66.7% |          839.7 |   [-74.9 , 1754.2] | ████████ 45/20 (100%) |
| formal           |    7 |    57.1% |          750.6 |   [377.5 , 1123.6] | ████████ 41/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square_balance, balance p_value=0.6696360831840533

Guardrails: configured=False, passed=True

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

🔀 Factorial Interaction Check · `smoke-copilot.md` (`caveman` × `subagent_model`)

🔬 View Interaction Cells

caveman \ subagent_model	large	small
yes	n=2, 50.0% succ, 823.5s	n=3, 33.3% succ, 830.0s
no	n=3, 33.3% succ, 774.0s	n=2, 50.0% succ, 861.5s

interaction_p_value (2×2 chi-square on assignment): 1.0 (no evidence of dependence, but all cells are sparse)

interaction_risk_status: SPARSE_CELL_RISK — every cell has n < min_samples (20)

Both caveman and subagent_model core decisions are EXTEND, so no promotion is being held back by this interaction check today — but the same pattern (n=1 for each of smokecopilotaoaiapikey and smokecopilotaoaientra) recurs across the smoke-copilot* family and should be watched once any of those experiments reaches PROMOTE.

🟡 Collecting — 27 experiments not yet at min_samples or blocked on multi-variant support

📋 View All Collecting Experiments

Experiment	Workflow	Reason code	Progress
`sub_agent_strategy`	`architecture-guardian.md`	`insufficient_samples`	single_agent=17/20, sub_agents=21/20
`audit_decomposition`	`audit-workflows.md`	`insufficient_samples`	phased_sub_agents=44/264, single_agent=35/264
`prompt_style`	`blog-auditor.md`	`insufficient_samples`	concise=3/20, detailed=7/20
`output_format`	`copilot-agent-analysis.md`	`unsupported_multi_variant`	prose=34/30, ste=3/30, structured=43/30
`detail_level`	`daily-architecture-diagram.md`	`insufficient_samples`	brief=11/20, comprehensive=7/20
`model_size`	`daily-cache-strategy-analyzer.md`	`insufficient_samples`	gpt-5.3-codex=8/20, gpt-5.3-codex-spark=9/20
`model_size`	`daily-caveman-optimizer.md`	`unsupported_multi_variant`	agent=1/20, claude-haiku-4.5=66/20, claude-sonnet-4.6=34/20, claude-sonnet-5=8/20, small-agent=2/20
`output_format`	`daily-code-metrics.md`	`unsupported_multi_variant`	executive_summary=53/20, full_detail=58/20, ste=16/20
`output_format`	`daily-compiler-quality.md`	`unsupported_multi_variant`	concise=54/20, detailed=60/20, ste=15/20
`model_size`	`daily-doc-healer.md`	`unsupported_multi_variant`	agent=2/20, claude-haiku-4.5=59/20, claude-sonnet-4.6=43/20, claude-sonnet-5=6/20, small-agent=1/20
`model_size`	`daily-doc-updater.md`	`unsupported_multi_variant`	agent=1/20, claude-haiku-4.5=45/20, claude-sonnet-4.6=47/20, claude-sonnet-5=12/20, small-agent=2/20
`model_size`	`daily-function-namer.md`	`insufficient_observations`	(no variant breakdown available)
`output_format`	`daily-issues-report.md`	`unsupported_multi_variant`	collapsible=59/20, inline=61/20, ste=19/20
`remove_redundant_context_v1`	`daily-rendering-scripts-verifier.md`	`insufficient_observations`	(no variant breakdown available)
`log_fetch_strategy`	`daily-safe-output-optimizer.md`	`insufficient_observations`	(no variant breakdown available)
`semgrep_output_format`	`daily-semgrep-scan.md`	`unsupported_multi_variant`	bullet_list=28/30, prose=27/30, structured_sections=22/30
`timeout_setting`	`dailysubagentoptimizer`	`unsupported_multi_variant`	default=2/20, relaxed=1/20, tight=1/20
`caveman_mode`	`dataflow-pr-discussion-dataset.md`	`insufficient_samples`	no=6/20, yes=9/20
`summary_detail`	`dependabotcampaign`	`insufficient_samples`	brief=6/20, detailed=6/20
`prompt_style`	`dependabot-go-checker.md`	`unsupported_multi_variant`	concise=18/20, detailed=18/20, step_by_step=14/20
`reasoning_depth`	`plan.md`	`unsupported_multi_variant`	baseline=4/20, deep=2/20, shallow=1/20
`remove_redundant_context_v1`	`pr-sous-chef.md`	`insufficient_observations`	(no variant breakdown available)
`sub_agent_strategy`	`smoke-copilot-sub-agents.md`	`unsupported_multi_variant`	delegated_sequential=15/30, inline_strict=13/30, single_agent_control=19/30
`prompt_style_test`	`smoke-project.md`	`insufficient_observations`	(no variant breakdown available)
`sub_agent_strategy`	`smoke-temporary-id.md`	`insufficient_observations`	(no variant breakdown available)
`model_size`	`test-quality-sentinel.md`	`insufficient_samples`	claude-haiku-4.5=20/20, claude-sonnet-5=18/20
`prefetch_strategy`	`weekly-blog-post-writer.md`	`insufficient_samples`	eager=5/20, lazy=10/20

📊 Summary

View Full Experiments Table

Experiment	Workflow	Control	Candidate	Evidence	Guardrails	Core decision	Report action
`prompt_compression`	`agent-performance-analyzer.md`	`caveman`	`verbose`	chi_square_balance	PASS	EXTEND	EXTEND
`sub_agent_strategy`	`agent-persona-explorer.md`	`batch`	`per_scenario`	chi_square_balance	PASS	EXTEND	EXTEND
`sub_agent_strategy`	`architecture-guardian.md`	`single_agent`	`sub_agents`	chi_square_balance	PASS	EXTEND	EXTEND
`audit_decomposition`	`audit-workflows.md`	`single_agent`	`phased_sub_agents`	chi_square_balance	unsupported	EXTEND	EXTEND
`tone_variant`	`aw-failure-investigator.md`	`?`	`?`	chi_square_balance	PASS	INCONCLUSIVE	INCONCLUSIVE
`prompt_style`	`blog-auditor.md`	`concise`	`detailed`	chi_square_balance	PASS	EXTEND	EXTEND
`tone_variant`	`breaking-change-checker.md`	`neutral`	`urgent`	chi_square_balance	PASS	EXTEND	EXTEND
`prompt_style`	`ci-coach.md`	`detailed`	`concise`	mann_whitney	unsupported/unsupported	EXTEND	EXTEND
`output_format`	`copilot-agent-analysis.md`	`?`	`?`	t_test	unsupported	INCONCLUSIVE	INCONCLUSIVE
`sub_agent_strategy`	`daily-agentrx-trace-optimizer.md`	`single_agent`	`sub_agents`	chi_square_balance	PASS	EXTEND	EXTEND
`detail_level`	`daily-architecture-diagram.md`	`brief`	`comprehensive`	chi_square_balance	PASS	EXTEND	EXTEND
`prompt_style`	`daily-astrostylelite-markdown-spellcheck.md`	`concise`	`detailed`	chi_square_balance	PASS	EXTEND	EXTEND
`model_size`	`daily-cache-strategy-analyzer.md`	`gpt-5.3-codex`	`gpt-5.3-codex-spark`	chi_square_balance	unsupported/unsupported	EXTEND	EXTEND
`model_size`	`daily-caveman-optimizer.md`	`?`	`?`	chi_square_balance	PASS	INCONCLUSIVE	INCONCLUSIVE
`output_format`	`daily-code-metrics.md`	`?`	`?`	chi_square_balance	PASS	INCONCLUSIVE	INCONCLUSIVE
`prompt_style`	`daily-community-attribution.md`	`concise`	`verbose`	chi_square_balance	PASS	EXTEND	EXTEND
`output_format`	`daily-compiler-quality.md`	`?`	`?`	chi_square_balance	PASS	INCONCLUSIVE	INCONCLUSIVE
`model_size`	`daily-doc-healer.md`	`?`	`?`	chi_square_balance	PASS	INCONCLUSIVE	INCONCLUSIVE
`model_size`	`daily-doc-updater.md`	`?`	`?`	chi_square_balance	PASS	INCONCLUSIVE	INCONCLUSIVE
`reasoning_depth`	`daily-fact.md`	`single_pass`	`multi_candidate`	chi_square_balance	unsupported/unsupported	EXTEND	EXTEND
`model_size`	`daily-function-namer.md`	`?`	`?`	chi_square_balance	PASS	EXTEND	EXTEND
`output_format`	`daily-issues-report.md`	`?`	`?`	chi_square_balance	PASS	INCONCLUSIVE	INCONCLUSIVE
`prompt_style`	`daily-news.md`	`concise`	`detailed`	chi_square_balance	PASS	EXTEND	EXTEND
`remove_redundant_context_v1`	`daily-rendering-scripts-verifier.md`	`?`	`?`	chi_square_balance	PASS	EXTEND	EXTEND
`log_fetch_strategy`	`daily-safe-output-optimizer.md`	`eager`	`lazy`	mann_whitney	unsupported	EXTEND	EXTEND
`reasoning_depth`	`daily-security-red-team.md`	`single_pass`	`iterative`	proportion_test	unsupported/unsupported	EXTEND	EXTEND
`semgrep_output_format`	`daily-semgrep-scan.md`	`?`	`?`	proportion_test	unsupported	INCONCLUSIVE	INCONCLUSIVE
`timeout_setting`	`dailysubagentoptimizer`	`?`	`?`	chi_square_balance	PASS	INCONCLUSIVE	INCONCLUSIVE
`caveman_mode`	`dataflow-pr-discussion-dataset.md`	`no`	`yes`	chi_square_balance	PASS	EXTEND	EXTEND
`output_format`	`deep-report.md`	`?`	`?`	chi_square_balance	PASS	INCONCLUSIVE	INCONCLUSIVE
`summary_detail`	`dependabotcampaign`	`brief`	`detailed`	chi_square_balance	PASS	EXTEND	EXTEND
`prompt_style`	`dependabot-go-checker.md`	`?`	`?`	chi_square_balance	PASS	INCONCLUSIVE	INCONCLUSIVE
`tool_verbosity`	`gpclean.md`	`full_bash`	`minimal_toolset`	chi_square_balance	PASS	EXTEND	EXTEND
`prompt_style`	`issue-arborist.md`	`concise`	`detailed`	chi_square_balance	PASS	EXTEND	EXTEND
`reasoning_depth`	`plan.md`	`?`	`?`	chi_square_balance	PASS	INCONCLUSIVE	INCONCLUSIVE
`remove_redundant_context_v1`	`pr-sous-chef.md`	`?`	`?`	chi_square_balance	PASS	EXTEND	EXTEND
`sub_agent_strategy`	`smokeantigravity`	`single_agent`	`sub_agents`	chi_square_balance	PASS	EXTEND	EXTEND
`caveman`	`smoke-copilot.md`	`no`	`yes`	chi_square_balance	PASS	EXTEND	EXTEND (interaction hold)
`subagent_model`	`smoke-copilot.md`	`large`	`small`	chi_square_balance	PASS	EXTEND	EXTEND
`caveman`	`smoke-copilot-aoai-apikey.md`	`no`	`yes`	chi_square_balance	PASS	EXTEND	EXTEND
`subagent_model`	`smoke-copilot-aoai-apikey.md`	`large`	`small`	chi_square_balance	PASS	EXTEND	EXTEND
`caveman`	`smoke-copilot-aoai-entra.md`	`no`	`yes`	chi_square_balance	PASS	EXTEND	EXTEND
`subagent_model`	`smoke-copilot-aoai-entra.md`	`large`	`small`	chi_square_balance	PASS	EXTEND	EXTEND
`sub_agent_strategy`	`smoke-copilot-sub-agents.md`	`?`	`?`	chi_square_balance	unsupported/unsupported	INCONCLUSIVE	INCONCLUSIVE
`sub_agent_strategy`	`smoke-gemini.md`	`single_agent`	`sub_agents`	chi_square_balance	PASS	EXTEND	EXTEND
`sub_agent_decomposition`	`smoke-pi.md`	`parallel_sub_agents`	`single_agent`	chi_square_balance	PASS	EXTEND	EXTEND
`prompt_style_test`	`smoke-project.md`	`?`	`?`	chi_square_balance	PASS	EXTEND	EXTEND
`sub_agent_strategy`	`smoke-temporary-id.md`	`single_agent`	`sub_agents`	t_test	unsupported/unsupported	EXTEND	EXTEND
`model_size`	`test-quality-sentinel.md`	`claude-haiku-4.5`	`claude-sonnet-5`	chi_square_balance	PASS	EXTEND	EXTEND
`tone_style`	`typist.md`	`conversational`	`formal`	chi_square_balance	PASS	EXTEND	EXTEND
`prefetch_strategy`	`weekly-blog-post-writer.md`	`eager`	`lazy`	chi_square_balance	PASS	EXTEND	EXTEND

Descriptive window: last 30 runs per workflow (where run-level data was fetchable) · Decision thresholds: from decision_policy (min effect 0, confidence 0.95)
Run: 35833984369

🔧 Self-Tuning Continuation Plan

🛠️ View Continuation Plan

Top 3 experiment actions

Wire native run_success_rate and empty_output_rate guardrail observations into the core analysis layer — 3 READY experiments (cicoach.prompt_style, dailyfact.reasoning_depth, dailysecurityredteam.reasoning_depth) are stuck on guardrail_unsupported despite having sufficient samples.
Extend core decision support to >2 variants — 14 experiments (deepreport.output_format, dailycodemetrics.output_format, dailyissuesreport.output_format, dailysemgrepscan.semgrep_output_format, plan.reasoning_depth, and 9 more) are permanently INCONCLUSIVE under unsupported_multi_variant regardless of sample size.
Attribute primary-metric observations (e.g. aic, token_count_per_run, discussion_engagement_score) to run IDs so insufficient_observations (19 of 24 READY experiments) can resolve into real PROMOTE/REJECT/INCONCLUSIVE decisions instead of perpetual EXTEND.

Top 3 eval actions

cicoach (ci-coach.md): tighten eval questions scoring PR merge rate / proposal quality so prompt_style (concise vs detailed) has a graded outcome instead of relying on an unsupported native guardrail.
dailyfact / dailysecurityredteam (daily-fact.md, daily-security-red-team.md): add an explicit empty_output_rate / run_success_rate grader so reasoning_depth guardrails stop reporting guardrail_unsupported.
deepreport (deep-report.md): add a grader for output_format variants (executive_brief, full_briefing, ste, annotated_brief) scoring readability/verbosity trade-offs, since this is a 4-variant experiment currently unresolvable by core analysis.

Decision-pipeline gaps for the next PR

guardrail_unsupported (3 experiments): native run_success_rate / empty_output_rate guardrails declared in frontmatter have no backing metric observation pipeline yet.
unsupported_multi_variant (14 experiments): core analysis only supports exactly 2 variants; K-variant (K≥3) experiments need a normalized comparison method (e.g. omnibus test + pairwise follow-up).
insufficient_observations (19 experiments): primary metric fields exist in frontmatter but are not resolvable to per-run values, so decisions default to EXTEND even at full sample size.
Interaction diagnostics (SPARSE_CELL_RISK on smokecopilot and its aoai variants) are currently computed only in this report; they should become a normalized core analysis signal that can force report_action overrides consistently rather than via ad hoc reporting logic.

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

github.com
o205451.ingest.us.sentry.io
proxy.golang.org
storage.googleapis.com
sum.golang.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"
    - "o205451.ingest.us.sentry.io"
    - "proxy.golang.org"
    - "storage.googleapis.com"
    - "sum.golang.org"

See Network Configuration for more information.

Generated by 🧪 daily-experiment-report · copilot · auto · 196.3 AIC · ⌖ 23 AIC · ⊞ 16.4K · ◷

expires on Sep 26, 2026, 12:19 AM UTC-08:00

2026-09-24T08:09:35Z

github-actions[bot]
Bot Sep 24, 2026
Author

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #63130.

0 replies

Navigation Menu

[experiments] Daily Experiment Report — 2026-09-23 #62905

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

[experiments] Daily Experiment Report — 2026-09-23 #62905

Uh oh!

github-actions[bot] Bot Sep 23, 2026

🧪 Daily Experiment Report — 2026-09-23

⚡ Quick Stats

prompt_compression · agent-performance-analyzer.md

sub_agent_strategy · agent-persona-explorer.md

tone_variant · aw-failure-investigator.md

tone_variant · breaking-change-checker.md

prompt_style · ci-coach.md

sub_agent_strategy · daily-agentrx-trace-optimizer.md

prompt_style · daily-astrostylelite-markdown-spellcheck.md

prompt_style · daily-community-attribution.md

reasoning_depth · daily-fact.md

prompt_style · daily-news.md

reasoning_depth · daily-security-red-team.md

output_format · deep-report.md

tool_verbosity · gpclean.md

prompt_style · issue-arborist.md

sub_agent_strategy · smokeantigravity

caveman · smoke-copilot.md

subagent_model · smoke-copilot.md

caveman · smoke-copilot-aoai-apikey.md

subagent_model · smoke-copilot-aoai-apikey.md

caveman · smoke-copilot-aoai-entra.md

subagent_model · smoke-copilot-aoai-entra.md

sub_agent_strategy · smoke-gemini.md

sub_agent_decomposition · smoke-pi.md

tone_style · typist.md

🔀 Factorial Interaction Check · smoke-copilot.md (caveman × subagent_model)

🟡 Collecting — 27 experiments not yet at min_samples or blocked on multi-variant support

📊 Summary

🔧 Self-Tuning Continuation Plan

Replies: 1 comment

Uh oh!

github-actions[bot] Bot Sep 24, 2026 Author

github-actions[bot]
Bot Sep 23, 2026

`prompt_compression` · `agent-performance-analyzer.md`

`sub_agent_strategy` · `agent-persona-explorer.md`

`tone_variant` · `aw-failure-investigator.md`

`tone_variant` · `breaking-change-checker.md`

`prompt_style` · `ci-coach.md`

`sub_agent_strategy` · `daily-agentrx-trace-optimizer.md`

`prompt_style` · `daily-astrostylelite-markdown-spellcheck.md`

`prompt_style` · `daily-community-attribution.md`

`reasoning_depth` · `daily-fact.md`

`prompt_style` · `daily-news.md`

`reasoning_depth` · `daily-security-red-team.md`

`output_format` · `deep-report.md`

`tool_verbosity` · `gpclean.md`

`prompt_style` · `issue-arborist.md`

`sub_agent_strategy` · `smokeantigravity`

`caveman` · `smoke-copilot.md`

`subagent_model` · `smoke-copilot.md`

`caveman` · `smoke-copilot-aoai-apikey.md`

`subagent_model` · `smoke-copilot-aoai-apikey.md`

`caveman` · `smoke-copilot-aoai-entra.md`

`subagent_model` · `smoke-copilot-aoai-entra.md`

`sub_agent_strategy` · `smoke-gemini.md`

`sub_agent_decomposition` · `smoke-pi.md`

`tone_style` · `typist.md`

🔀 Factorial Interaction Check · `smoke-copilot.md` (`caveman` × `subagent_model`)

github-actions[bot]
Bot Sep 24, 2026
Author