{
  "schema": "coreai-kit-supervisor-measurements/1",
  "provenance": "Supervisor supplied in rounds/02.instruction.md; measured through coreai-kit, not rerun in this conversion round",
  "supplied_text": "Facts the supervisor measured on the kit side (2026-09-23 05:2x–05:4x JST, worktree ~/code/coreai-kit-models-wt,\n`Decision.Format.sharedState`, the sequential engine, the kit's own chat-template rendering) — quote as \"measured\nthrough coreai-kit\", do not re-derive:\n- `decide-cli parity` on results/fixtures.json: the 40 `choice` and `noul` rows — tokens identical 40/40 (the kit's\n  chat-template rendering equals the author's `compile()` byte for byte), label slots 40/40, option argmax 40/40,\n  max |Δp| 0.0055, mean 0.0002. The 8 `score_level` rows are skipped: the kit's question kinds have no primitive of\n  that name (the author's score_level is a yes/no on one proposition, the kit's `noul`, but the JSON says\n  `\"primitive\": \"score_level\"` and the kit renders `\"noul\"`). Median 2.8 s per fixture question over those 40 rows\n  (124–1,672 compiled tokens, S=1 prefill), 24 s for the 1,672-token row.\n- A two-question workflow request (a 20-token state, a 3-criteria choice and a noul): 2,069 ms for the 134-token\n  row and 1,540 ms for the 107-token row, 0 tokens reused (the recurrent hybrid re-prefills every row) — about\n  15 ms per token on the sequential engine. Answers: close 0.999; refund asked P(yes) 0.000.\n- SemIf's authored144 (144 English rows, 3 options each, SemIf's gold labels, its unchanged `benchmarks/evaluate.py`),\n  the kit's own rendering of each row as a `choice`: raw 131/144, **mean family balanced accuracy 0.906**, median\n  1.96 s per decision. For scale on the same rows and evaluator the kit README reports Qwen3.5-4B int8 zero-shot\n  0.821 and MiniCPM5-2B int8 0.681, and the OpenThai run 0.725. Numbers, not a ranking.\n- The kit's own \"which model\" mistake, for the knowledge page's Swift paragraph: an earlier `ask` measurement of\n  the OpenThai sibling had silently used the catalog default model because the command ignored `--bundle`; the\n  command was fixed and every figure above was taken with the model line printed (`model: apus_openjev_… format: sharedState`).\n",
  "measurement_date": "2026-09-23 JST",
  "worktree_read_only": "~/code/coreai-kit-models-wt",
  "format": "Decision.Format.sharedState",
  "engine": "coreai-sequential",
  "chat_rendering": "kit own chat template; thinking closed",
  "parity": {
    "rows": 40,
    "token_identical": 40,
    "label_slots": 40,
    "option_argmax": 40,
    "max_abs_probability_error": 0.0055,
    "mean_error": 0.0002,
    "skipped_score_level_rows": 8,
    "skip_reason": "Kit has no score_level question kind; although semantically a yes/no proposition, rendering noul changes primitive JSON and the prompt.",
    "median_seconds_per_question": 2.8,
    "compiled_tokens_min": 124,
    "compiled_tokens_max": 1672,
    "wide_row_tokens": 1672,
    "wide_row_seconds": 24
  },
  "workflow": {
    "state_tokens": 20,
    "questions": [
      {
        "primitive": "choice",
        "criteria": 3,
        "tokens": 134,
        "milliseconds": 2069,
        "answer": "close",
        "probability": 0.999
      },
      {
        "primitive": "noul",
        "tokens": 107,
        "milliseconds": 1540,
        "proposition": "refund asked",
        "yes_probability": 0.0
      }
    ],
    "reused_tokens": 0,
    "approx_sequential_milliseconds_per_token": 15
  },
  "semif": {
    "dataset": "SemIf authored144",
    "rows": 144,
    "language": "en",
    "options_per_row": 3,
    "labels": "SemIf gold",
    "evaluator": "unchanged benchmarks/evaluate.py",
    "rendered_primitive": "choice",
    "raw_correct": 131,
    "mean_family_balanced_accuracy": 0.906,
    "median_seconds_per_decision": 1.96,
    "same_rows_and_evaluator_context": {
      "Qwen3.5-4B int8 zero-shot": 0.821,
      "MiniCPM5-2B int8": 0.681,
      "OpenThai run": 0.725
    },
    "comparison_note": "Numbers for scale, not a ranking."
  },
  "model_selection_trap": "An earlier OpenThai ask measurement ignored --bundle and used the catalog default; the command was fixed, and all supplied measurements printed model: apus_openjev_… format: sharedState."
}