Issues

• Drop the schedule gate on the regression check: it is what produced the 1 October false green

  harness
  hookdeck/evals#99 · Filter by author leggetter opened on Oct 7, 2026

• The committed regression results file is overwritten by every non-merge run, and cannot contain a failing cell

  harness
  hookdeck/evals#98 · Filter by author leggetter opened on Oct 7, 2026

• A cell that errors on a later attempt still discards the earlier attempts' transcripts

  harness
  hookdeck/evals#97 · Filter by author leggetter opened on Oct 7, 2026

• provenance never reaches a published row, so --drop-stale and the whole staleness check are inert

  harness
  hookdeck/evals#96 · Filter by author leggetter opened on Oct 7, 2026

• regression-filtering-001 fails correct answers for the word they open with

  harnessregression-alert
  hookdeck/evals#92 · Filter by author github-actions opened on Oct 5, 2026
  2

• Operator events list endpoints advertise filters and paging they silently ignore

  product
  hookdeck/evals#88 · Filter by author leggetter opened on Oct 1, 2026 · 5. Sustain

• Outpost answers 502 for an unknown operator event destination id

  product
  hookdeck/evals#87 · Filter by author leggetter opened on Oct 1, 2026 · 5. Sustain

• Docs-call reporting counts unobserved search results as evidence, in both directions

  harness
  hookdeck/evals#83 · Filter by author leggetter opened on Sep 22, 2026 · 4. Trust the numbers
  1

• One errored cell stops a whole run from publishing

  harness
  hookdeck/evals#80 · Filter by author leggetter opened on Sep 21, 2026 · 4. Trust the numbers

• Agents wire a placeholder verification secret into a live source, and nothing surfaces it

  product
  hookdeck/evals#75 · Filter by author leggetter opened on Sep 2, 2026 · 5. Sustain

• Re-running publish-results on a held run fails to push

  harness
  hookdeck/evals#74 · Filter by author leggetter opened on Sep 2, 2026 · 4. Trust the numbers
  1

• Documentation access differs by agent and by arm, and the search path mostly returns nothing

  harness
  hookdeck/evals#61 · Filter by author leggetter opened on Aug 26, 2026 · 2. Prove the skills earn their place
  4

• A merged snapshot does not say it is merged, and the page renders one date for it

  harness
  hookdeck/evals#60 · Filter by author leggetter opened on Aug 26, 2026 · 4. Trust the numbers
  2

• Five benchmark scenarios have never been failed by any agent

  scenario
  hookdeck/evals#47 · Filter by author leggetter opened on Aug 24, 2026 · 1. Cover the product
  5

• Hookdeck's webhook notification config cannot be read back, and the natural way to set it returns 200 while doing nothing

  documentationproductscenario
  hookdeck/evals#43 · Filter by author leggetter opened on Aug 21, 2026 · 5. Sustain

• The provider-key preflight only runs in CI, not for a local run

  harness
  hookdeck/evals#42 · Filter by author leggetter opened on Aug 21, 2026 · 4. Trust the numbers

• The Outpost skill never says you need a key for an Outpost project

  product
  hookdeck/evals#40 · Filter by author leggetter opened on Aug 21, 2026 · 5. Sustain
  1

• The Outpost API answers "Not Found" when your key belongs to a non-Outpost project

  product
  hookdeck/evals#39 · Filter by author leggetter opened on Aug 21, 2026 · 5. Sustain
  1

• Nothing detects an agent reading this repository during a run

  harness
  hookdeck/evals#38 · Filter by author leggetter opened on Aug 21, 2026 · 4. Trust the numbers

• The operator events API is undocumented: absent from both OpenAPI definitions and from the docs

  documentationproduct
  hookdeck/evals#34 · Filter by author leggetter opened on Aug 20, 2026 · 5. Sustain
  2

• Outpost docs give the wrong env var name for every operator events setting

  documentationproduct
  hookdeck/evals#32 · Filter by author leggetter opened on Aug 20, 2026 · 5. Sustain
  1

• A leftover from a crashed run becomes permanent project state

  harness
  hookdeck/evals#28 · Filter by author leggetter opened on Aug 19, 2026 · 4. Trust the numbers
  2

• Does documenting hookdeck ci stop agents giving up on CLI authentication?

  product
  hookdeck/evals#27 · Filter by author leggetter opened on Aug 19, 2026 · 2. Prove the skills earn their place

• The skills submodule pin drifts silently, and moving it will change results with no record

  harness
  hookdeck/evals#26 · Filter by author leggetter opened on Aug 19, 2026 · 4. Trust the numbers

• No signal for when a configuration change is in force

  product
  hookdeck/evals#25 · Filter by author leggetter opened on Aug 19, 2026 · 5. Sustain
  1