{"schema":"coreai-kev-fixtures/1","what":"The fixture and held-out records of the kev-4b port with the author's fp32 readout (the oracle) of every question and the shipped fp16 graph's probabilities on the Mac GPU. 384 records / 434 questions and a held-out file of 130; records written for the port and the SemIf authored144 items carry their text, the transfer-v4 records are references (the author's file, line and row hashes) to be rebuilt with conversion/kev/make_fixtures.py, and the records withheld by the name screen keep only their numbers.","model":{"hf_id":"jaredpalmer/kev-4b","tag":"v1.0","base":"Qwen/Qwen3.5-4B-Base","oracle":{"code":"github.com/jaredpalmer/kev tag kev-1.0 (kev.checkpoint.Checkpoint.load cpu fp32, kev.model.admit + model.forward, row form)","threads":1,"temperature":2.406050072164233,"versions":{"python":"3.12.13","torch":"2.8.0","transformers":"5.17.0","peft":"0.21.0","numpy":"2.5.3","safetensors":"0.8.0"},"checkpoint":"jaredpalmer/kev-4b@591dcb5bd6d05eb0b5131ea6608f93f10243335c"}},"graph":{"bundle":"kev_4b_decode_fp16_metal_pf128","asset":"AOT h16c .aimodelc of the bundle's .aimodel on an Apple M4 Max GPU (macOS 27.0 26A428), SpecializationOptions.default(), Python runtime","p":"the fp16 hidden rows read through the author's fp32 pointer head, temperature applied","gate_transcript":"gate-kev-4b-readout.json (fixture and held-out)","gate_sources":["results/r12_fixture_K128_4b.json","results/r12_heldout_K128_4b.json"]},"contract":{"row":"one causal row per question: [<|fim_prefix|>] + state + [<|fim_middle|>] + instructions + for each option ([<|box_start|>] + option + [<|box_end|>]) + [<|fim_suffix|>], positions 0..L-1, fresh zero states","delimiters":{"state":{"token":"<|fim_prefix|>","id":248060},"q":{"token":"<|fim_middle|>","id":248061},"opt":{"token":"<|box_start|>","id":248049},"opt_end":{"token":"<|box_end|>","id":248050},"decide":{"token":"<|fim_suffix|>","id":248062}},"pad":{"token":"<|endoftext|>","id":248044},"user_tokens":"re.sub(r'<\\|([A-Za-z0-9_]+)\\|>', r'<¦\\1¦>', text), tokenized without special tokens","render":"a string as is; an object or array as 'key: value' lines ('- ' for array items, nesting indented by two spaces), values as Python's str(); noul options 'no[: <criteria.false>]', 'yes[: <criteria.true>]' in that order; choice 'name' or 'name: <description>' in criteria order; score the levels in order (kev.api.to_record)","head":"z_k = ((W_k h_opt_k + b_k) . (W_q h_decide + b_q)) / sqrt(256); p = softmax(z / T) within the question","temperature":2.406050072164233,"readout_index":"decide = the <|fim_suffix|> index (the row's last token); opts = each <|box_end|> index","graph_row_limit_tokens":3968,"keys":"choice: criteria names in request order; noul: ['false', 'true']; score: '0'..'n-1'","near_tie":"oracle top-2 margin <= 0.02","graph_static_query_len":128,"host_reads_the_graph":"one row per question, fresh zero states per row; a row of T ids runs as ceil(T / 128) calls of 'main' (static S = 128): call k gets ids[128k : 128k + 128] with position_ids 0..128k+127; the last call is padded with <|endoftext|> (248044) and the hidden rows of the padded positions are discarded (causal: they cannot reach a real position). The hidden rows [1, 128, 2560] of every call, concatenated and cut to T, are the backbone's final-norm last_hidden_state [T, 2560] the head reads. A row fits when ceil(T / 128) * 128 <= 4095 (the position dim's upper bound; the KV sequence dim is allocated at 4096)."},"sources":{"transfer_v4":{"path_in_repo":"evals/v4/transfer-v4/development.jsonl","sha256":"ff374c49c6c9f15f8a56fb274b4a4857d20497eb8dd1ac07ce01560e682a5f2e","bytes":803309,"lines":764,"author_repo":{"repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","commit":"6b719c3c3f367295f6ef336f4f751cf5ff970abc"},"slices":{"tv4":"first 60 lines (all mmlu)","tv4x":"first 20 records of each of ['emotion', 'tweet_offensive', 'qnli', 'paws', 'sciq', 'legacy_holdout', 'composition_holdout'] (by _meta.source), file order","tv4s":"first 20 records whose question is a score, file order"},"source_licenses":{"mmlu":{"dataset":"cais/mmlu","license":"mit"},"emotion":{"dataset":"dair-ai/emotion","license":"other (card)"},"tweet_offensive":{"dataset":"cardiffnlp/tweet_eval","license":"unknown (card)"},"qnli":{"dataset":"nyu-mll/glue","license":"other (card; QNLI derives from SQuAD, CC BY-SA 4.0)"},"paws":{"dataset":"google-research-datasets/paws","license":"other (card)"},"sciq":{"dataset":"allenai/sciq","license":"cc-by-nc-3.0"},"legacy_holdout":{"dataset":null,"license":"generated by the author (Apache-2.0 repository)"},"composition_holdout":{"dataset":null,"license":"generated by the author (Apache-2.0 repository)"}}},"transfer_v4_heldout_slices":{"tv4h":"mmlu records 61-100, file order","tv4xh":"records 21-30 of each of ['emotion', 'tweet_offensive', 'qnli', 'paws', 'sciq', 'legacy_holdout', 'composition_holdout'] (by _meta.source), file order","tv4sh":"score records 21-40, file order"},"transfer_v4_what_is_redistributed":"nothing of the rows' text or token ids: each record names the file, the line and the row's _meta (ids and sha256); make_fixtures.py rebuilds the requests from the author's file at tag kev-1.0 and checks row_ids_sha256","semif_authored144":{"upstream_repo":"github.com/TheoLeeCJ/SemIf","upstream_path":"benchmarks/data/authored144.jsonl","upstream_commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","rows":144,"license":"MIT","license_file_sha256":"f765f2140f8507a8f0d81ec0fd2c4bd72fe6a066841ef27883ff876a76bf61be","license_notice":"MIT License\n\nCopyright (c) 2026 TheoLeeCJ\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n","mapping":"state -> state; question -> questions['decision'].instructions; options -> criteria {id: description}; type choice; gold = options[label].id","what_is_redistributed":"the 144 rows' state, question and option texts in the request shape above (records with source 'semif_authored144'), with the MIT copyright and permission notice above"},"own":{"license":"written for this port (Apache-2.0, as the conversion code)","names_screen":{"date":"2026-10-04","method":["exact-phrase web search per stem and per name the records use (stem + suffix, product names)","DNS A records of <stem>.com/.net/.io/.co.uk/.app and of the concatenated compound names","App Store search API (itunes.apple.com/search, entity software)","Companies House register search"],"stems_published":["brindlecourt","fennistone","ostlebury","pennimere","quennick","wendlecote"],"published_result":"DNS, App Store and Companies House: no entity of these names; exact-phrase web search: no page using these names. One near match is recorded, not counted: Companies House lists a dissolved company whose name extends one stem by a letter (a different word).","withheld":{"records":["own_t02","own_t03","own_t05","own_t08","own_t09","own_j03","own_j04","own_j05","own_L02"],"questions":28,"why":"four invented stems used in these records were found in use on the web (a book series, a product, places in games and fan wikis); the records are kept in the gates, their text is not published and the names are not changed (the oracle would change). Each withheld record keeps its id, request_sha256 and numbers."},"earlier_screen":"the stems were first screened by the clef-flash lane on 2026-10-03 (DNS; App Store; Companies House; fixtures-clef-flash.json names_screen)"}}},"records":[{"id":"tv4_000","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":0,"_meta":{"row":13292,"text_sha256":"e4f765d62c6ed44999ddbaa502f1b865f19197d1e492af2abef30ac5dfd47750","row_sha256":"010a5fda50061ef619d06b589cf689715a9774f817996fb54d6d627b8499f882","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13292","group_id":"mmlu/test/13292","variant":"clean"}},"state_tokens":30,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":94,"state_len":30,"decide":93,"opts":[55,66,77,92],"row_ids_sha256":"2000abddb81002f7b36b029b8aef585cd384badd145187f08d811a1fbadddbda","oracle":{"probs":[0.3150305449962616,0.1274043768644333,0.2856283187866211,0.2719367742538452],"logits_raw":[-1.6387466192245483,-3.8169519901275635,-1.8744869232177734,-1.9926766157150269],"argmax":0,"top2_margin":0.029402226209640503,"near_tie":false},"graph":{"p":[0.3147282004356384,0.1275910884141922,0.2873123586177826,0.2703683376312256],"max_abs_dp":0.001684039831161499,"argmax_equal":true}}]},{"id":"tv4_001","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":1,"_meta":{"row":588,"text_sha256":"6f26999eaee4ab138243c58aa5c0360817cb583c90caaac957438ec0b09f9d90","row_sha256":"010f9883d7f34928b7cf5b3fe2ecf5f7e662cf3cc29611c5790da5e6b28ef96a","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/588","group_id":"mmlu/test/588","variant":"clean"}},"state_tokens":16,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":63,"state_len":16,"decide":62,"opts":[33,42,52,61],"row_ids_sha256":"1167f1b340ce2f23e018030ac0676fb206c548907f19783a2a11471e6e086aec","oracle":{"probs":[0.0011141307186335325,0.003409974742680788,0.9864944219589233,0.008981414139270782],"logits_raw":[-10.638294219970703,-7.946813583374023,5.689361572265625,-5.616668224334717],"argmax":2,"top2_margin":0.9775130152702332,"near_tie":false},"graph":{"p":[0.0011441382812336087,0.00346803804859519,0.9863178730010986,0.009069874882698059],"max_abs_dp":0.00017654895782470703,"argmax_equal":true}}]},{"id":"tv4_002","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":2,"_meta":{"row":4589,"text_sha256":"b21eb6a47835c694dbb4c55d0c6eaf127143103b5be6e901b3bc081a1875e03a","row_sha256":"0140ae7a6c63a4b72a607fe028843ded2c762ed70839dd64f1e26c4f29fd8813","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4589","group_id":"mmlu/test/4589","variant":"clean"}},"state_tokens":18,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":96,"state_len":18,"decide":95,"opts":[39,53,74,94],"row_ids_sha256":"d1353ef0e56096aef885705eebdd6c352a134fd8201d4b4c55a488b0ab26b3b1","oracle":{"probs":[0.9461724758148193,0.017222043126821518,0.01631193421781063,0.02029358223080635],"logits_raw":[2.4616715908050537,-7.177529335021973,-7.308160781860352,-6.782661437988281],"argmax":0,"top2_margin":0.9258788824081421,"near_tie":false},"graph":{"p":[0.945670485496521,0.017428243532776833,0.016417725011706352,0.020483531057834625],"max_abs_dp":0.0005019903182983398,"argmax_equal":true}}]},{"id":"tv4_003","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":3,"_meta":{"row":6099,"text_sha256":"d11693badfd9d9aeb495b1bbd2b030536f3badf884e3c5c968350c222af0c1bf","row_sha256":"018fc4e2f2bc5d033d1203c9298ea91deb852be048cc2234325c3f764b24db98","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6099","group_id":"mmlu/test/6099","variant":"clean"}},"state_tokens":20,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":75,"state_len":20,"decide":74,"opts":[39,52,64,73],"row_ids_sha256":"1d58ded5471ce2d76dd8e83723daa86b0b9fa7d56b895a722870b7af02095b52","oracle":{"probs":[0.7948135137557983,0.12265517562627792,0.03986677527427673,0.0426645390689373],"logits_raw":[-0.7092422842979431,-5.205501079559326,-7.909501075744629,-7.746311187744141],"argmax":0,"top2_margin":0.6721583604812622,"near_tie":false},"graph":{"p":[0.7906588315963745,0.12575477361679077,0.04044175148010254,0.04314456135034561],"max_abs_dp":0.004154682159423828,"argmax_equal":true}}]},{"id":"tv4_004","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":4,"_meta":{"row":6139,"text_sha256":"67fe59443c44fef770b56204f7dd4ac9a42a3d1d492a4249ec7e742c7c97566f","row_sha256":"022dbb0545abb6ee6edcb5b28dbfdc7099785c2a5b106aa0536906349b11988d","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6139","group_id":"mmlu/test/6139","variant":"clean"}},"state_tokens":14,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":66,"state_len":14,"decide":65,"opts":[31,41,54,64],"row_ids_sha256":"f7f4804691594fc7a83f6eb1c8cb6d5f820b72b7cb60a59de6fa6ea5b4f96997","oracle":{"probs":[0.0047936588525772095,0.005961189512163401,0.9870544075965881,0.0021907698828727007],"logits_raw":[-7.469605922698975,-6.945143699645996,5.348459720611572,-9.353641510009766],"argmax":2,"top2_margin":0.9810932278633118,"near_tie":false},"graph":{"p":[0.00491651427000761,0.006067235954105854,0.9867876768112183,0.002228585770353675],"max_abs_dp":0.00026673078536987305,"argmax_equal":true}}]},{"id":"tv4_005","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":5,"_meta":{"row":13810,"text_sha256":"d6ab903bb64d97ef4252906a9c35b394c2b135cc82173b9a98a562ae4941e714","row_sha256":"03bd43e6287a70dfe5cc0e5fe2bf7bfe2cdf49e4a895d91a13fdde9bbb4ff5d9","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13810","group_id":"mmlu/test/13810","variant":"clean"}},"state_tokens":24,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":81,"state_len":24,"decide":80,"opts":[43,52,69,79],"row_ids_sha256":"f6389bae11472feac961a23bfbc86817f77aba956c54bc218fbf4aeae5703284","oracle":{"probs":[0.9009091258049011,0.0497448593378067,0.014532082714140415,0.03481397405266762],"logits_raw":[2.3841583728790283,-4.584959030151367,-7.545720100402832,-5.443649768829346],"argmax":0,"top2_margin":0.8511642813682556,"near_tie":false},"graph":{"p":[0.900649905204773,0.04982937499880791,0.014521854929625988,0.03499884530901909],"max_abs_dp":0.00025922060012817383,"argmax_equal":true}}]},{"id":"tv4_006","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":6,"_meta":{"row":12434,"text_sha256":"6415833a28d2dcfe5791e61f69f47411f55fa901fdbcb63f0f2a39b718ad434b","row_sha256":"04184bb438912221313ece06f12d94cc5b9eb0500c9a978a14d7e05f5fcbbedf","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12434","group_id":"mmlu/test/12434","variant":"clean"}},"state_tokens":194,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":237,"state_len":194,"decide":236,"opts":[209,217,228,235],"row_ids_sha256":"b2a8a29fc34ec49ff80d45dc8bd98b7532e65f324fec3a7d0c1b82ea9b587ff8","oracle":{"probs":[0.01970069296658039,0.13868305087089539,0.020422756671905518,0.8211934566497803],"logits_raw":[-9.362539291381836,-4.667043209075928,-9.275931358337402,-0.3877207338809967],"argmax":3,"top2_margin":0.6825103759765625,"near_tie":false},"graph":{"p":[0.019811978563666344,0.13736490905284882,0.02070142887532711,0.8221216797828674],"max_abs_dp":0.0013181418180465698,"argmax_equal":true}}]},{"id":"tv4_007","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":7,"_meta":{"row":123,"text_sha256":"b91ae5229f181bae8ead6b7d5617ebf84c8a0a6e86eb7b45f9f04ff55413c613","row_sha256":"041edcea28c4c2a8b1b57e50a75d175e7efa26cce57a5e8dc554add9a2369644","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/123","group_id":"mmlu/test/123","variant":"clean"}},"state_tokens":13,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":80,"state_len":13,"decide":79,"opts":[34,49,63,78],"row_ids_sha256":"50f6dfe74a8f3fac36c562d796bb87b9a1574190a28572003bd489d64a8ec709","oracle":{"probs":[0.2620672583580017,0.11499524861574173,0.5438520312309265,0.07908540964126587],"logits_raw":[-5.081377029418945,-7.063265800476074,-3.3247780799865723,-7.964000701904297],"argmax":2,"top2_margin":0.2817847728729248,"near_tie":false},"graph":{"p":[0.26423901319503784,0.1162101998925209,0.5392643809318542,0.08028635382652283],"max_abs_dp":0.004587650299072266,"argmax_equal":true}}]},{"id":"tv4_008","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":8,"_meta":{"row":2785,"text_sha256":"15d7296ec5506ce45f273de8dca692fd7ee3b091af4b89c5d8ff60e3aa972fd4","row_sha256":"0434623eedfb30e0c938e9f5342d1ebc1849aa07e10549d9bfccffc0d06e8ed6","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/2785","group_id":"mmlu/test/2785","variant":"clean"}},"state_tokens":37,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":96,"state_len":37,"decide":95,"opts":[55,66,76,94],"row_ids_sha256":"ebf13644b09d8ff3701d3e7fbd42ad1eb28000ae8c58f5d7ffea6e0be8f47ac9","oracle":{"probs":[0.10677852481603622,0.8779287934303284,0.0029180501587688923,0.01237452868372202],"logits_raw":[-0.3907793164253235,4.67830753326416,-9.052176475524902,-5.57609748840332],"argmax":1,"top2_margin":0.7711502909660339,"near_tie":false},"graph":{"p":[0.1073785349726677,0.8771418333053589,0.00296025606803596,0.012519370764493942],"max_abs_dp":0.0007869601249694824,"argmax_equal":true}}]},{"id":"tv4_009","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":9,"_meta":{"row":5717,"text_sha256":"ae688490966c0852a0980ac7935c42ea98308efbfaca49a1cd5589064dba6cac","row_sha256":"05d282bcb34f55ca7b59ecafcb9a63c6d36d97a015a1c48979f80a7d6c7a4ea6","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/5717","group_id":"mmlu/test/5717","variant":"clean"}},"state_tokens":259,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":321,"state_len":259,"decide":320,"opts":[279,294,309,319],"row_ids_sha256":"f8587e17cf0e7c98c00d851776836241a1abcac295f41a4b82c9e8ff30d52d11","oracle":{"probs":[0.003395455190911889,0.0023475931957364082,0.978935718536377,0.015321257524192333],"logits_raw":[-10.074394226074219,-10.962340354919434,3.5535407066345215,-6.448948860168457],"argmax":2,"top2_margin":0.9636144638061523,"near_tie":false},"graph":{"p":[0.003457693150267005,0.0024363098200410604,0.9786092042922974,0.01549683790653944],"max_abs_dp":0.00032651424407958984,"argmax_equal":true}}]},{"id":"tv4_010","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":10,"_meta":{"row":8037,"text_sha256":"6eb96bdf834743053b122379db03e7f056ad3dc8ff3f5513b290c40620d2555f","row_sha256":"064a938d631d75f6147d58bb15659a7e76b9afd06b060eced5705d607167a4dd","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/8037","group_id":"mmlu/test/8037","variant":"clean"}},"state_tokens":29,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":64,"state_len":29,"decide":63,"opts":[43,49,56,62],"row_ids_sha256":"75d8a68f1077318821be6fb102f94ab200f20286f0c5f36e0cd69715f65c08bd","oracle":{"probs":[0.08209438621997833,0.34313106536865234,0.38211092352867126,0.19266363978385925],"logits_raw":[-8.138072967529297,-4.696837425231934,-4.437950134277344,-6.085529327392578],"argmax":2,"top2_margin":0.03897985816001892,"near_tie":false},"graph":{"p":[0.08106456696987152,0.34112125635147095,0.3834644556045532,0.19434967637062073],"max_abs_dp":0.0020098090171813965,"argmax_equal":true}}]},{"id":"tv4_011","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":11,"_meta":{"row":1047,"text_sha256":"2a0d61f454a3b0592fa563c96cd9d4ec0981bf856112ed46603214e9573eac02","row_sha256":"065c2a3f0ad371d06e17d82fe4bc8bfc5855bca2a04de9c1c9adb08d551a8639","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1047","group_id":"mmlu/test/1047","variant":"clean"}},"state_tokens":70,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":105,"state_len":70,"decide":104,"opts":[83,89,95,103],"row_ids_sha256":"c0f2fbf0e7c8da5318c150fea8c7dca5e577765482150045adb619dcd1984d66","oracle":{"probs":[0.04895943030714989,0.1945311427116394,0.3305221498012543,0.4259873330593109],"logits_raw":[-4.379034042358398,-1.059646725654602,0.21575535833835602,0.8262568116188049],"argmax":3,"top2_margin":0.09546518325805664,"near_tie":false},"graph":{"p":[0.04978971928358078,0.19574706256389618,0.3294092118740082,0.42505404353141785],"max_abs_dp":0.001215919852256775,"argmax_equal":true}}]},{"id":"tv4_012","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":12,"_meta":{"row":4518,"text_sha256":"72be257ffdd59cdd7a74367581cd0aa5d75169cacfe687ab8b9bfbacafc817d0","row_sha256":"06f9edd28b068873d96ccfdb1df2848b00d64e1fd317e529fc71ac37a979eb39","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4518","group_id":"mmlu/test/4518","variant":"clean"}},"state_tokens":37,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":83,"state_len":37,"decide":82,"opts":[54,62,73,81],"row_ids_sha256":"5595e60356ee14e848ab01adaa3940149131b3cc59a3ef99a682871814f2737a","oracle":{"probs":[0.20165841281414032,0.011728987097740173,0.08439213037490845,0.7022204399108887],"logits_raw":[-3.947751522064209,-10.791789054870605,-6.043664455413818,-0.94579017162323],"argmax":3,"top2_margin":0.5005620121955872,"near_tie":false},"graph":{"p":[0.20111782848834991,0.011911937966942787,0.08428137004375458,0.7026888132095337],"max_abs_dp":0.0005405843257904053,"argmax_equal":true}}]},{"id":"tv4_013","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":13,"_meta":{"row":1492,"text_sha256":"c76022d4ee47dfc12b1e1ba7ff196c857d494ffbcd679e8b1d655f668fd72500","row_sha256":"070ed9de720c112fbd8b78c4e6dbbd24c1734d23867fb249de507295794e6cdf","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1492","group_id":"mmlu/test/1492","variant":"clean"}},"state_tokens":33,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":163,"state_len":33,"decide":162,"opts":[63,93,121,161],"row_ids_sha256":"a43f683d90f97b85d9e67007afa2adf43ae2760e31d4b8ab7f39249642253e2d","oracle":{"probs":[0.25900205969810486,0.08372330665588379,0.44022154808044434,0.2170531153678894],"logits_raw":[-0.044351786375045776,-2.7615489959716797,1.2319183349609375,-0.469486266374588],"argmax":2,"top2_margin":0.18121948838233948,"near_tie":false},"graph":{"p":[0.2600085735321045,0.0843493714928627,0.4382520914077759,0.21738992631435394],"max_abs_dp":0.001969456672668457,"argmax_equal":true}}]},{"id":"tv4_014","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":14,"_meta":{"row":4063,"text_sha256":"5cc13884a1a177fa42853b29a986afb3d31c76715d41b3ef3263fd69b6438330","row_sha256":"0733d390eb549c8c29ff274b7f6c2ef73f6dcc0629dd53b1b19b34e43f31f116","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4063","group_id":"mmlu/test/4063","variant":"clean"}},"state_tokens":22,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":87,"state_len":22,"decide":86,"opts":[46,59,76,85],"row_ids_sha256":"4d37606e1d7fd4174482706f16b783471416b69e70cc304f0e127405043cf16f","oracle":{"probs":[0.5154111385345459,0.13634498417377472,0.23513613641262054,0.11310772597789764],"logits_raw":[0.5502175688743591,-2.649291515350342,-1.3380509614944458,-3.098855972290039],"argmax":0,"top2_margin":0.28027498722076416,"near_tie":false},"graph":{"p":[0.5159146785736084,0.13610517978668213,0.23529358208179474,0.11268651485443115],"max_abs_dp":0.0005035400390625,"argmax_equal":true}}]},{"id":"tv4_015","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":15,"_meta":{"row":11310,"text_sha256":"08836e0191901e64734fbe0f9a786efaf3709406e58f7fa495078bcb610c53a9","row_sha256":"07cea382ce1e459ddb0a46898382abcb1762fd9b713a087f06aca048f0923b2a","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/11310","group_id":"mmlu/test/11310","variant":"clean"}},"state_tokens":105,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":177,"state_len":105,"decide":176,"opts":[126,144,161,175],"row_ids_sha256":"f07a3716b76ecf9d9aa2c622a2f6f596f2e5027582c1f7ed208373a072cf6955","oracle":{"probs":[0.4475664496421814,0.08508399873971939,0.3801879286766052,0.08716163784265518],"logits_raw":[-2.045011043548584,-6.039501190185547,-2.4375803470611572,-5.981454849243164],"argmax":0,"top2_margin":0.06737852096557617,"near_tie":false},"graph":{"p":[0.4449280798435211,0.08600359410047531,0.3807871341705322,0.08828119933605194],"max_abs_dp":0.0026383697986602783,"argmax_equal":true}}]},{"id":"tv4_016","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":16,"_meta":{"row":6610,"text_sha256":"230af30dc37acdac5cace354b74414e19f977465f005f72fb6f13ce1c83fc01f","row_sha256":"080972877925a0e9071a647836ab7bc1b9f6fb860fd37fd74b9d115eb1a8a150","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6610","group_id":"mmlu/test/6610","variant":"clean"}},"state_tokens":24,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":82,"state_len":24,"decide":81,"opts":[40,59,69,80],"row_ids_sha256":"66e7a7aaedcfcec1400a3b1c1d8204f8cb8a1bb35f6ac0bec62d0387938863c6","oracle":{"probs":[0.0537070631980896,0.07393914461135864,0.07500778138637543,0.7973459959030151],"logits_raw":[-5.2811174392700195,-4.511908054351807,-4.477382659912109,1.2097898721694946],"argmax":3,"top2_margin":0.7223381996154785,"near_tie":false},"graph":{"p":[0.05440974235534668,0.0742761641740799,0.07547152787446976,0.7958425879478455],"max_abs_dp":0.0015034079551696777,"argmax_equal":true}}]},{"id":"tv4_017","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":17,"_meta":{"row":9462,"text_sha256":"f658cc8e1232787da72bf6e03aa3f71f61d9b2e1cbd9cb34e78e29b12865dcfb","row_sha256":"0813758be42e718265867e9b104410830958994506ff946e678204da674ab60e","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/9462","group_id":"mmlu/test/9462","variant":"clean"}},"state_tokens":14,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":76,"state_len":14,"decide":75,"opts":[35,46,65,74],"row_ids_sha256":"f52d80020f620fe9b5fa2a7cd9fe64da0ef86b04aff33b7dbc1902617fced76a","oracle":{"probs":[0.10457127541303635,0.07979969680309296,0.747214138507843,0.0684148371219635],"logits_raw":[-2.3416171073913574,-2.9920907020568848,2.389838933944702,-3.3624541759490967],"argmax":2,"top2_margin":0.6426428556442261,"near_tie":false},"graph":{"p":[0.10509437322616577,0.07968582212924957,0.7466570138931274,0.06856272369623184],"max_abs_dp":0.0005571246147155762,"argmax_equal":true}}]},{"id":"tv4_018","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":18,"_meta":{"row":5208,"text_sha256":"31a70389d9e44cad3395616eac1504fa0ce169118cd1d71dce3763e10a62c45c","row_sha256":"08cdef9dcb2506c626af30d1df6d606b616e510de7a393b065c19acf5da24497","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/5208","group_id":"mmlu/test/5208","variant":"clean"}},"state_tokens":27,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":67,"state_len":27,"decide":66,"opts":[44,51,59,65],"row_ids_sha256":"aad24101aae2b7f2bd5903fa7a75bed8bd65242929b0eddadd26b46b4a5300ff","oracle":{"probs":[0.003768309485167265,0.0019068237161263824,0.9856775999069214,0.008647325448691845],"logits_raw":[-8.177544593811035,-9.816516876220703,5.216219902038574,-6.179022789001465],"argmax":2,"top2_margin":0.9770302772521973,"near_tie":false},"graph":{"p":[0.0038229450583457947,0.0019594086334109306,0.9855656027793884,0.008652067743241787],"max_abs_dp":0.00011199712753295898,"argmax_equal":true}}]},{"id":"tv4_019","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":19,"_meta":{"row":9213,"text_sha256":"ad7d058864a902202cc16c98a71221e2eee376a5cb319653c0581de66893e99f","row_sha256":"0ad24336bd4a190abb0870d37ed56901090b8c288422d98b2b8dae935e0a5951","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/9213","group_id":"mmlu/test/9213","variant":"clean"}},"state_tokens":89,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":130,"state_len":89,"decide":129,"opts":[103,111,119,128],"row_ids_sha256":"81df69e328d36495eab8cb665d45581f9d57a648f9a6a4229adee4367591c0c9","oracle":{"probs":[0.26913294196128845,0.23663587868213654,0.13414302468299866,0.36008816957473755],"logits_raw":[-2.948969602584839,-3.258586883544922,-4.6242995262146,-2.2484638690948486],"argmax":3,"top2_margin":0.0909552276134491,"near_tie":false},"graph":{"p":[0.26928088068962097,0.23844462633132935,0.13415317237377167,0.35812124609947205],"max_abs_dp":0.001966923475265503,"argmax_equal":true}}]},{"id":"tv4_020","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":20,"_meta":{"row":12914,"text_sha256":"5afdeb67f2af871c752771dd9425be2bc6967bcb203b35de05f057220a27789a","row_sha256":"0ada8ef3402be468e4a35329742fa3232f1ff88aac6612afb0d092090c50e3f3","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12914","group_id":"mmlu/test/12914","variant":"clean"}},"state_tokens":34,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":82,"state_len":34,"decide":81,"opts":[50,56,67,80],"row_ids_sha256":"9553fef6d998f387dbc013c283ac97999c54f4c7b90e4580b28ee2349bf9ebc6","oracle":{"probs":[0.07224544137716293,0.01614832691848278,0.8871472477912903,0.024458957836031914],"logits_raw":[-2.0663676261901855,-5.671238899230957,3.967865228652954,-4.672295093536377],"argmax":2,"top2_margin":0.8149018287658691,"near_tie":false},"graph":{"p":[0.07280680537223816,0.016375767067074776,0.886083722114563,0.024733686819672585],"max_abs_dp":0.001063525676727295,"argmax_equal":true}}]},{"id":"tv4_021","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":21,"_meta":{"row":724,"text_sha256":"e38af88bca0cd26b62120a5e330698e74c26820357a06f0fd8da19c39a867706","row_sha256":"0adf991fd97e7801b25429b4babec11df58b8d5dc9f6e81989c1ae1694338f59","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/724","group_id":"mmlu/test/724","variant":"clean"}},"state_tokens":16,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":65,"state_len":16,"decide":64,"opts":[32,41,52,63],"row_ids_sha256":"550b9ad66a5a1b3263631348b4f76879808641a8076e45ed8bd897fff0b4fcf7","oracle":{"probs":[0.28640973567962646,0.1525333970785141,0.3049657344818115,0.25609108805656433],"logits_raw":[-0.8528628349304199,-2.368770122528076,-0.7018205523490906,-1.1220760345458984],"argmax":2,"top2_margin":0.01855599880218506,"near_tie":true},"graph":{"p":[0.286985844373703,0.15404559671878815,0.30470913648605347,0.2542594075202942],"max_abs_dp":0.0018316805362701416,"argmax_equal":true}}]},{"id":"tv4_022","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":22,"_meta":{"row":4454,"text_sha256":"28649f1c96e825de7daa47754c989b906cd75b4644525f722544a5d298b8468c","row_sha256":"0b7df30f69dd9f462fbac0d3c7662a9b4b38a59a4d4f11310fc1969525ae8ae8","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4454","group_id":"mmlu/test/4454","variant":"clean"}},"state_tokens":105,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":144,"state_len":105,"decide":143,"opts":[119,127,134,142],"row_ids_sha256":"4d9a261b166c2979b2617bc4a1532dddced5a96177ad49713e91ec4839b90a15","oracle":{"probs":[0.473575621843338,0.3190898895263672,0.16468609869480133,0.04264833405613899],"logits_raw":[-1.7489427328109741,-2.6989448070526123,-4.290381908416748,-7.541083335876465],"argmax":0,"top2_margin":0.15448573231697083,"near_tie":false},"graph":{"p":[0.47394245862960815,0.31681400537490845,0.16613127291202545,0.04311222583055496],"max_abs_dp":0.0022758841514587402,"argmax_equal":true}}]},{"id":"tv4_023","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":23,"_meta":{"row":8228,"text_sha256":"5eda8abb1505371de513af404a66c64321f24733a74fe9d7997751869896daca","row_sha256":"0bec9e299974e1bbd0299dc5b542170ed2e7f0c823d095e8c95bd34dd66c5ceb","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/8228","group_id":"mmlu/test/8228","variant":"clean"}},"state_tokens":41,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":92,"state_len":41,"decide":91,"opts":[58,69,79,90],"row_ids_sha256":"b11d078163153bdf62a081e2b9de43ec251e2a69ef8618955f2f429321a2b33e","oracle":{"probs":[0.7337514758110046,0.08191145211458206,0.022401990368962288,0.16193503141403198],"logits_raw":[1.0759371519088745,-4.199403762817383,-7.318820953369141,-2.559544563293457],"argmax":0,"top2_margin":0.5718164443969727,"near_tie":false},"graph":{"p":[0.7350694537162781,0.08242347836494446,0.022592445835471153,0.15991458296775818],"max_abs_dp":0.0020204484462738037,"argmax_equal":true}}]},{"id":"tv4_024","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":24,"_meta":{"row":13238,"text_sha256":"5c97128bd214131f1ddac5218553879a4bd91db54f482e32c57da5913e6d0f09","row_sha256":"0cace9a6eff08851fc6f07a62a24fcb1f1c06c96dceebc71c4dc67fec7a6bf10","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13238","group_id":"mmlu/test/13238","variant":"clean"}},"state_tokens":23,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":107,"state_len":23,"decide":106,"opts":[55,76,97,105],"row_ids_sha256":"8cde7efbf4c454f1c385ee41c2e6eda2e53eb29bcb56e72ccf7f1048c8e303e1","oracle":{"probs":[0.398024320602417,0.18030424416065216,0.04451369121670723,0.37715765833854675],"logits_raw":[0.043987780809402466,-1.861284852027893,-5.226984977722168,-0.08557763695716858],"argmax":0,"top2_margin":0.02086666226387024,"near_tie":false},"graph":{"p":[0.39657843112945557,0.18163682520389557,0.04473681002855301,0.37704795598983765],"max_abs_dp":0.0014458894729614258,"argmax_equal":true}}]},{"id":"tv4_025","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":25,"_meta":{"row":11404,"text_sha256":"c27531d6a1741c33d2f26d6e9fddc7936f384064c89cde64dcc2ccdb8481cf78","row_sha256":"0cd28fb4615e5cff84b7f6c042098abed66dd1a265c7c45364224dbff09e2c84","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/11404","group_id":"mmlu/test/11404","variant":"clean"}},"state_tokens":132,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":180,"state_len":132,"decide":179,"opts":[148,155,162,178],"row_ids_sha256":"4a4a4f6cec088e8dffd8a035ee52f1770968722b0f512e220123272c9d49b339","oracle":{"probs":[0.4803442358970642,0.1125757023692131,0.23228369653224945,0.17479631304740906],"logits_raw":[-0.9961614608764648,-4.487044334411621,-2.7442617416381836,-3.4283933639526367],"argmax":0,"top2_margin":0.24806053936481476,"near_tie":false},"graph":{"p":[0.4802953004837036,0.11296635866165161,0.23158466815948486,0.1751537173986435],"max_abs_dp":0.0006990283727645874,"argmax_equal":true}}]},{"id":"tv4_026","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":26,"_meta":{"row":6593,"text_sha256":"fa18d8eb79c698b5e014bb3484bc1ac5132f0d94874455a0cead140a36871b4b","row_sha256":"0d02b58d02f3e95f74b821635dfdddc34a8b2ace30bdbc508f05f915cd44bfb3","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6593","group_id":"mmlu/test/6593","variant":"clean"}},"state_tokens":34,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":92,"state_len":34,"decide":91,"opts":[55,67,77,90],"row_ids_sha256":"a8221610bb42dd3209252c7b9313c2e3abe6c65ca81d4a741045f6c80e3df9f7","oracle":{"probs":[0.2253187596797943,0.3078656792640686,0.19752155244350433,0.26929402351379395],"logits_raw":[-2.1312203407287598,-1.3801777362823486,-2.4480209350585938,-1.7022510766983032],"argmax":1,"top2_margin":0.03857165575027466,"near_tie":false},"graph":{"p":[0.22615092992782593,0.3067304193973541,0.1983584761619568,0.2687602639198303],"max_abs_dp":0.0011352598667144775,"argmax_equal":true}}]},{"id":"tv4_027","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":27,"_meta":{"row":3841,"text_sha256":"080abd0a3265a45f0136d1fa3f9b3eb724571503c178f9c082ca331d2550ee66","row_sha256":"0d43bd015715be6a656242ba73f2e7cb6e82355ad11e9c12914a2d9052eccd2b","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/3841","group_id":"mmlu/test/3841","variant":"clean"}},"state_tokens":15,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":77,"state_len":15,"decide":76,"opts":[32,49,62,75],"row_ids_sha256":"c8d5579fed65ad3cb06b933224a93421edb4e34bf2c0b2b29952e6db7cb1fed3","oracle":{"probs":[0.02962842397391796,0.35287588834762573,0.4597041606903076,0.15779142081737518],"logits_raw":[-9.177024841308594,-3.2163193225860596,-2.5799989700317383,-5.152810096740723],"argmax":2,"top2_margin":0.10682827234268188,"near_tie":false},"graph":{"p":[0.029976055026054382,0.35305407643318176,0.4615814685821533,0.15538844466209412],"max_abs_dp":0.002402976155281067,"argmax_equal":true}}]},{"id":"tv4_028","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":28,"_meta":{"row":11705,"text_sha256":"73ee9198bb27004a774c7defd4636b24f6931b85320a3fb826f31dbd7d2bbe02","row_sha256":"0eb66519306fa8a8ecc1873b57d6bbf252d623dc5481e64207120b290c2359aa","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/11705","group_id":"mmlu/test/11705","variant":"clean"}},"state_tokens":234,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":290,"state_len":234,"decide":289,"opts":[253,263,276,288],"row_ids_sha256":"c5eedf726eb20deb22a377ef082dfd38ebe607d3247fde00f4817b1485a5f599","oracle":{"probs":[0.34704673290252686,0.3146056830883026,0.20348362624645233,0.13486401736736298],"logits_raw":[-3.0610198974609375,-3.2971479892730713,-4.345547199249268,-5.335200309753418],"argmax":0,"top2_margin":0.03244104981422424,"near_tie":false},"graph":{"p":[0.3477550148963928,0.3137400150299072,0.20375652611255646,0.13474838435649872],"max_abs_dp":0.0008656680583953857,"argmax_equal":true}}]},{"id":"tv4_029","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":29,"_meta":{"row":2862,"text_sha256":"d429e7cd3c0676ad51fe99eb08872dabbb5f9f457cfed19084009276fcfb0ab5","row_sha256":"0f055897f863481cb6f33301380858951cd7a3fad82fec2a605fcfe4551d0596","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/2862","group_id":"mmlu/test/2862","variant":"clean"}},"state_tokens":30,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":64,"state_len":30,"decide":63,"opts":[43,49,55,62],"row_ids_sha256":"f2e4a0985d6e856fbce30d79df845a9c65d244bc342547013366ee5facb58c41","oracle":{"probs":[0.02507132478058338,0.0006970107788220048,0.011386982165277004,0.9628446698188782],"logits_raw":[-2.757366180419922,-11.377471923828125,-4.656351089477539,6.020306587219238],"argmax":3,"top2_margin":0.9377733469009399,"near_tie":false},"graph":{"p":[0.025091392919421196,0.0007113218889571726,0.011436780914664268,0.9627604484558105],"max_abs_dp":8.422136306762695e-05,"argmax_equal":true}}]},{"id":"tv4_030","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":30,"_meta":{"row":9772,"text_sha256":"18cbaea8b014f81f1a3cae404be60d6cde12ff38a648f3ec08b8c51464b57137","row_sha256":"0f24142b228b4133d31115ae881abcd37fd4efd0afe4f91de60e11fb598211d1","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/9772","group_id":"mmlu/test/9772","variant":"clean"}},"state_tokens":13,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":78,"state_len":13,"decide":77,"opts":[30,40,59,76],"row_ids_sha256":"9d507cb8e8c6c2478177bb86bf3ed78701252e692ff32920fc5783627266d05f","oracle":{"probs":[0.10892936587333679,0.03823283687233925,0.7892667055130005,0.06357110291719437],"logits_raw":[-2.1623501777648926,-4.681496620178223,2.602602481842041,-3.45810604095459],"argmax":2,"top2_margin":0.6803373098373413,"near_tie":false},"graph":{"p":[0.1094495952129364,0.038677867501974106,0.7880014777183533,0.06387104839086533],"max_abs_dp":0.0012652277946472168,"argmax_equal":true}}]},{"id":"tv4_031","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":31,"_meta":{"row":5094,"text_sha256":"2bafd22f31a9fff5917ac28f483d154f2a964ff090176b1ba0b4c9434403801c","row_sha256":"0f69624a2c67dc837033e8037a6c0e7f8748b6c94ed6ef265ec82be45c5cb8e7","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/5094","group_id":"mmlu/test/5094","variant":"clean"}},"state_tokens":22,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":67,"state_len":22,"decide":66,"opts":[37,46,56,65],"row_ids_sha256":"92caa742b07fbf513d8f3f6b1f842b3572f4eb7013b12356d7a6b40e179a0884","oracle":{"probs":[0.9014595150947571,0.07842501997947693,0.009546308778226376,0.010569173842668533],"logits_raw":[3.61226749420166,-2.2629988193511963,-7.330112457275391,-7.085206985473633],"argmax":0,"top2_margin":0.8230345249176025,"near_tie":false},"graph":{"p":[0.9018208980560303,0.07796106487512589,0.009578626602888107,0.010639401152729988],"max_abs_dp":0.0004639551043510437,"argmax_equal":true}}]},{"id":"tv4_032","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":32,"_meta":{"row":10172,"text_sha256":"e063a35bc796e0c4eb85c16dc5c8b1fbb82701c11168fb320be646ba28476bfc","row_sha256":"0f9a5d1f0ef7225fd0f2bd67d68d0ebc47bb8ce1d98e3a082d23ca27d4cd5182","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/10172","group_id":"mmlu/test/10172","variant":"clean"}},"state_tokens":21,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":135,"state_len":21,"decide":134,"opts":[55,81,107,133],"row_ids_sha256":"83d4890753eec1d5f4aefa15d53f8618bec6ca58103addc60545098b37e2b76e","oracle":{"probs":[0.006398226134479046,0.006537633016705513,0.020407356321811676,0.9666568040847778],"logits_raw":[-8.268241882324219,-8.216381072998047,-5.477525234222412,3.8048906326293945],"argmax":3,"top2_margin":0.9462494254112244,"near_tie":false},"graph":{"p":[0.0065604886040091515,0.006699197459965944,0.02084340527653694,0.9658969044685364],"max_abs_dp":0.0007598996162414551,"argmax_equal":true}}]},{"id":"tv4_033","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":33,"_meta":{"row":3349,"text_sha256":"2727ef590d6a7b74fc042faafcf9c9570b374400f88c35e1069784efabaa7ef7","row_sha256":"111ff188a463868073a49590f1be6da85ca9649c068f5dd639fcce931036c5c8","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/3349","group_id":"mmlu/test/3349","variant":"clean"}},"state_tokens":198,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":293,"state_len":198,"decide":292,"opts":[226,248,270,291],"row_ids_sha256":"3d8d625aeee2bd962934909891a9a2ea53d919015334d154a7c09379493cd71a","oracle":{"probs":[0.00243237498216331,0.0050327470526099205,0.043879956007003784,0.9486548900604248],"logits_raw":[-9.695302963256836,-7.945869445800781,-2.735588550567627,4.659616470336914],"argmax":3,"top2_margin":0.9047749042510986,"near_tie":false},"graph":{"p":[0.0024810214526951313,0.005143973045051098,0.043534521013498306,0.9488404989242554],"max_abs_dp":0.0003454349935054779,"argmax_equal":true}}]},{"id":"tv4_034","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":34,"_meta":{"row":5191,"text_sha256":"526fc9864f083866076d59ec085a5840d2dc3a51de7d6140a37f0b06b082b8c2","row_sha256":"1176e707d52212efe4a01cc0c680f05ffac10f895743a6f755297e64ab15f122","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/5191","group_id":"mmlu/test/5191","variant":"clean"}},"state_tokens":43,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":77,"state_len":43,"decide":76,"opts":[55,61,68,75],"row_ids_sha256":"11f16b69dd0685e28f995b6635b5960e9d7415b4afba12225ea23e3296cbb053","oracle":{"probs":[0.1682184785604477,0.059066176414489746,0.4283755123615265,0.3443398177623749],"logits_raw":[-3.4019522666931152,-5.920136451721191,-1.1529291868209839,-1.6783411502838135],"argmax":2,"top2_margin":0.08403569459915161,"near_tie":false},"graph":{"p":[0.16898736357688904,0.05970156192779541,0.4270980954170227,0.34421297907829285],"max_abs_dp":0.0012774169445037842,"argmax_equal":true}}]},{"id":"tv4_035","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":35,"_meta":{"row":1808,"text_sha256":"f770a51b608eda349ee8aa4125bde15c974a7c110b05eec9df0f7392b935225a","row_sha256":"123bf64fbafcd2f6fec39cf01f12d33a9bf995d806e619aa81bc351e8915a2af","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1808","group_id":"mmlu/test/1808","variant":"clean"}},"state_tokens":36,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":108,"state_len":36,"decide":107,"opts":[63,74,94,106],"row_ids_sha256":"d4e3d3c6b21d62cea995c6a72bd73ac7e5237f4466cb2b36220b44153a402bf1","oracle":{"probs":[0.9208319783210754,0.013207502663135529,0.05247688293457031,0.01348357368260622],"logits_raw":[2.1043319702148438,-8.108129501342773,-4.7887725830078125,-8.058355331420898],"argmax":0,"top2_margin":0.8683550953865051,"near_tie":false},"graph":{"p":[0.9214127659797668,0.013112693093717098,0.0520024336874485,0.013472135178744793],"max_abs_dp":0.0005807876586914062,"argmax_equal":true}}]},{"id":"tv4_036","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":36,"_meta":{"row":10509,"text_sha256":"f56bf336aa556755eac37356eee98ec47feea09017a3d173deca0090a0671730","row_sha256":"14383703de5b130b888fbfa3e419839116b75a84137a6acc94b9f9da1a536863","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/10509","group_id":"mmlu/test/10509","variant":"clean"}},"state_tokens":101,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":150,"state_len":101,"decide":149,"opts":[118,128,138,148],"row_ids_sha256":"4334d7668aa909cac6ba11bdb8969004d06134fb4192a782846e810025957936","oracle":{"probs":[0.11167816072702408,0.22471334040164948,0.42167386412620544,0.2419346570968628],"logits_raw":[-4.41905403137207,-2.7367334365844727,-1.2223494052886963,-2.559065341949463],"argmax":2,"top2_margin":0.17973920702934265,"near_tie":false},"graph":{"p":[0.11139114201068878,0.2252723127603531,0.42428600788116455,0.23905061185359955],"max_abs_dp":0.0028840452432632446,"argmax_equal":true}}]},{"id":"tv4_037","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":37,"_meta":{"row":13713,"text_sha256":"5f54dd5414d94f4e5d9b41b3446a051b265c3b6c54c95b2e5ad393b9186d495c","row_sha256":"14910a65dfdb4ef6cf2108d5ff4522050ab8408c532b968aae2b7de7aa64727b","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13713","group_id":"mmlu/test/13713","variant":"clean"}},"state_tokens":24,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":67,"state_len":24,"decide":66,"opts":[37,43,50,65],"row_ids_sha256":"04e308660f844f083dcb2fb187a332451a552f8c772938d14dfbf9b6e3bc355a","oracle":{"probs":[0.1309981793165207,0.09336290508508682,0.4930292069911957,0.2826097905635834],"logits_raw":[-4.587323188781738,-5.402226448059082,-1.3983807563781738,-2.7373507022857666],"argmax":2,"top2_margin":0.2104194164276123,"near_tie":false},"graph":{"p":[0.13190481066703796,0.09303531795740128,0.4911605715751648,0.2838992774486542],"max_abs_dp":0.0018686354160308838,"argmax_equal":true}}]},{"id":"tv4_038","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":38,"_meta":{"row":1807,"text_sha256":"63104fec8e11568f820894e0eb75b8c1e47b4bf437e5d78f65047e8c92ad6cfb","row_sha256":"149cc7e282a8181766092e95741e7d74584e56717f89aa15d7fb2de472b1776d","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1807","group_id":"mmlu/test/1807","variant":"clean"}},"state_tokens":35,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":130,"state_len":35,"decide":129,"opts":[63,90,110,128],"row_ids_sha256":"924313310c33b48ff4275930f00fd339dde968ebae2808aa4ca11bde10308e9f","oracle":{"probs":[0.04206390678882599,0.025341540575027466,0.8831757307052612,0.04941878840327263],"logits_raw":[-4.834688186645508,-6.053942680358887,2.4901316165924072,-4.446976184844971],"argmax":2,"top2_margin":0.8337569236755371,"near_tie":false},"graph":{"p":[0.04188543185591698,0.025597305968403816,0.8833905458450317,0.049126699566841125],"max_abs_dp":0.0002920888364315033,"argmax_equal":true}}]},{"id":"tv4_039","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":39,"_meta":{"row":4720,"text_sha256":"d9f86719ffa9dc8954a412b58b7482a7dc5bd0ea176d0f0ec31398637cf1648b","row_sha256":"14e515224ce2587fad9cf05bbfab4b5e27f211f949db0c413468ecb5c63e2ea0","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4720","group_id":"mmlu/test/4720","variant":"clean"}},"state_tokens":26,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":69,"state_len":26,"decide":68,"opts":[41,50,58,67],"row_ids_sha256":"a039229b628a584148c4ef1165be67627782265a91b293918e53969c93b2c7d9","oracle":{"probs":[0.4805123209953308,0.4267076849937439,0.0757506862282753,0.01702931337058544],"logits_raw":[-0.8418197631835938,-1.127547025680542,-5.286769390106201,-8.877825736999512],"argmax":0,"top2_margin":0.053804636001586914,"near_tie":false},"graph":{"p":[0.4826391339302063,0.42243656516075134,0.07733429223299026,0.017590096220374107],"max_abs_dp":0.004271119832992554,"argmax_equal":true}}]},{"id":"tv4_040","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":40,"_meta":{"row":12568,"text_sha256":"a3a223bcafcf86355939d886cbe5cffcd1499240ff3007125d1629cfb6e6ce05","row_sha256":"15ee481e12ac1182a5e6741fc1dc006dd5acd23711420945c5cfddec3de8c7e5","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12568","group_id":"mmlu/test/12568","variant":"clean"}},"state_tokens":100,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":208,"state_len":100,"decide":207,"opts":[132,160,181,206],"row_ids_sha256":"050bd3614c5086ba32d76befad242b5b31e272ff6704073f0cbf9045db5ba3f3","oracle":{"probs":[0.07466527819633484,0.19247709214687347,0.08866479992866516,0.6441928744316101],"logits_raw":[-3.351006507873535,-1.0725691318511963,-2.937532424926758,1.8339899778366089],"argmax":3,"top2_margin":0.45171576738357544,"near_tie":false},"graph":{"p":[0.07587786763906479,0.19447734951972961,0.0898149386048317,0.6398298144340515],"max_abs_dp":0.004363059997558594,"argmax_equal":true}}]},{"id":"tv4_041","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":41,"_meta":{"row":12163,"text_sha256":"357bec3dde33cccb1b6d97f805743e7804c397287ffbe3cfc449c9f3ce049e8c","row_sha256":"16592e464ce99da5a7f2f2e16f0251575aea7595e69d2fc1092656e96904297e","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12163","group_id":"mmlu/test/12163","variant":"clean"}},"state_tokens":126,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":208,"state_len":126,"decide":207,"opts":[148,163,193,206],"row_ids_sha256":"cad0c73530463ee6f59f2e2b0a3bfac82e0f2c20bc7c6cd2ee79bce02d0047a9","oracle":{"probs":[0.1233932226896286,0.04569873958826065,0.2979303002357483,0.5329777598381042],"logits_raw":[-2.2131381034851074,-4.603080749511719,-0.09224504232406616,1.3071621656417847],"argmax":3,"top2_margin":0.23504745960235596,"near_tie":false},"graph":{"p":[0.12451154738664627,0.04603496566414833,0.30357566475868225,0.5258779525756836],"max_abs_dp":0.007099807262420654,"argmax_equal":true}}]},{"id":"tv4_042","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":42,"_meta":{"row":6773,"text_sha256":"ad92e1941705bfa22d357ace87e80108d4f552db2d04d78c4c7dbcff69e62991","row_sha256":"16f57b256c903af68e1f29638cdea0c98d34cdfe847a6899b3974dfceb24522f","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6773","group_id":"mmlu/test/6773","variant":"clean"}},"state_tokens":21,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":59,"state_len":21,"decide":58,"opts":[36,43,51,57],"row_ids_sha256":"6c96c2b06935f2f5ac29d7db24c138340901d7f54839cc1603211643fb7e4058","oracle":{"probs":[0.4437820613384247,0.13237157464027405,0.2593693137168884,0.16447709500789642],"logits_raw":[-6.571156024932861,-9.481803894042969,-7.863398551940918,-8.959310531616211],"argmax":0,"top2_margin":0.18441274762153625,"near_tie":false},"graph":{"p":[0.44270724058151245,0.1329076737165451,0.2587140202522278,0.16567102074623108],"max_abs_dp":0.0011939257383346558,"argmax_equal":true}}]},{"id":"tv4_043","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":43,"_meta":{"row":1982,"text_sha256":"d9ffe57f3bac0b6b5e36685e5fc5c980ca2c277cafee5911504bac119dc8a19b","row_sha256":"173b236e9e6235053ace7bfa297ab81a9dc1d8098f84e6605736ea0497db23d8","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1982","group_id":"mmlu/test/1982","variant":"clean"}},"state_tokens":22,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":60,"state_len":22,"decide":59,"opts":[39,45,51,58],"row_ids_sha256":"03ac9c7af4229dc0469db0941087f3d701fad2a598763f9e1a59c49e39ec500d","oracle":{"probs":[0.0750739574432373,0.8144273161888123,0.04362407326698303,0.06687461584806442],"logits_raw":[-2.567723274230957,3.168327569961548,-3.873882293701172,-2.8459932804107666],"argmax":1,"top2_margin":0.739353358745575,"near_tie":false},"graph":{"p":[0.07495091110467911,0.8157790899276733,0.04350856691598892,0.06576145440340042],"max_abs_dp":0.001351773738861084,"argmax_equal":true}}]},{"id":"tv4_044","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":44,"_meta":{"row":3758,"text_sha256":"209b997de0aab123d791890481a71f42a3242b0de191ce264d3cc16192c12b6e","row_sha256":"173e90ff9c8e472cbc5633766bc1e4310718fa6cfc5738327921b238c04c4b16","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/3758","group_id":"mmlu/test/3758","variant":"clean"}},"state_tokens":32,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":82,"state_len":32,"decide":81,"opts":[45,59,67,80],"row_ids_sha256":"08fa0c75bdfa00975668800cee18adda23546f18f90637c0afdcb4eeaed97e53","oracle":{"probs":[0.0017190277576446533,0.0001701900764601305,0.0006938357255421579,0.9974169731140137],"logits_raw":[-11.240362167358398,-16.804588317871094,-13.423320770263672,4.0703206062316895],"argmax":3,"top2_margin":0.9956979751586914,"near_tie":false},"graph":{"p":[0.0017471344908699393,0.00017366126121487468,0.0007208112510852516,0.9973583817481995],"max_abs_dp":5.8591365814208984e-05,"argmax_equal":true}}]},{"id":"tv4_045","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":45,"_meta":{"row":1857,"text_sha256":"d1c63be268917217f4bf97cdd3bce1c57767d755fe12f1771a173baaf1eb0455","row_sha256":"1742bf69e66dfd9afd72eca8a80efdfea705670aa814f7c847d3d1a3e82d5bae","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1857","group_id":"mmlu/test/1857","variant":"clean"}},"state_tokens":49,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":90,"state_len":49,"decide":89,"opts":[64,71,79,88],"row_ids_sha256":"940928f9b10c85f521cbc39a81f334d28d3c1dda2d9b639a59ab35a97455d202","oracle":{"probs":[0.058799877762794495,0.010207188315689564,0.8951600193977356,0.035832930356264114],"logits_raw":[-3.8954057693481445,-8.108513832092285,2.655938148498535,-5.087055683135986],"argmax":2,"top2_margin":0.8363601565361023,"near_tie":false},"graph":{"p":[0.057948652654886246,0.01015380583703518,0.8965857028961182,0.03531194478273392],"max_abs_dp":0.0014256834983825684,"argmax_equal":true}}]},{"id":"tv4_046","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":46,"_meta":{"row":2806,"text_sha256":"47d3aa72b23e7d16cbb68a5870a03acdf468c693fccc545e641995d1b9d9b59a","row_sha256":"1797ea252aabf8e7b5d92080fb1c53b73873154cfa2f9e0ddedaae07c3b25986","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/2806","group_id":"mmlu/test/2806","variant":"clean"}},"state_tokens":32,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":68,"state_len":32,"decide":67,"opts":[46,54,60,66],"row_ids_sha256":"babf37ef5d0f13c3251a6bb8051eea3249242807a062df0635df2f0577284036","oracle":{"probs":[0.9806473255157471,0.005689522251486778,0.009510239586234093,0.0041529834270477295],"logits_raw":[4.7715349197387695,-7.61862850189209,-6.3825364112854,-8.37605094909668],"argmax":0,"top2_margin":0.9711371064186096,"near_tie":false},"graph":{"p":[0.9804789423942566,0.005781146697700024,0.00953555479645729,0.004204354248940945],"max_abs_dp":0.00016838312149047852,"argmax_equal":true}}]},{"id":"tv4_047","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":47,"_meta":{"row":12630,"text_sha256":"c4fbf107fefb1392e03a9e502108a2e09e06a2848b1b1e732af643146f9ae716","row_sha256":"17b6e12c658cc0b30180f9170090381c8cf88caf34f84ef346ea61c0076d328f","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12630","group_id":"mmlu/test/12630","variant":"clean"}},"state_tokens":36,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":67,"state_len":36,"decide":66,"opts":[49,55,60,65],"row_ids_sha256":"f2a73e0d265347d39aadd408ca94c9d7c8c17eac0b787113d8d94b885d9a5885","oracle":{"probs":[0.053366243839263916,0.09565497934818268,0.695864737033844,0.15511402487754822],"logits_raw":[-9.013313293457031,-7.609216690063477,-2.8346328735351562,-6.446101188659668],"argmax":2,"top2_margin":0.5407507419586182,"near_tie":false},"graph":{"p":[0.05314522236585617,0.09677331149578094,0.6950143575668335,0.15506711602210999],"max_abs_dp":0.0011183321475982666,"argmax_equal":true}}]},{"id":"tv4_048","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":48,"_meta":{"row":8523,"text_sha256":"c2d9284847102813614e2479684b859b9ad6a71aaf310c15851ed05a0c67e047","row_sha256":"17f07d503ffa0494aa586d7a4a18911b4b0d62da344c0dca33d1f6be0a925804","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/8523","group_id":"mmlu/test/8523","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":119,"state_len":78,"decide":118,"opts":[92,100,108,117],"row_ids_sha256":"b9ca798e56f775708b12e758b3436d71f9cc9e8d85819a96ab44a437a7917472","oracle":{"probs":[0.19524699449539185,0.4933403432369232,0.07672601193189621,0.23468662798404694],"logits_raw":[-5.255789279937744,-3.025540351867676,-7.50309944152832,-4.813110828399658],"argmax":1,"top2_margin":0.2586537003517151,"near_tie":false},"graph":{"p":[0.19612433016300201,0.4945650100708008,0.07651552557945251,0.23279507458209991],"max_abs_dp":0.0018915534019470215,"argmax_equal":true}}]},{"id":"tv4_049","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":49,"_meta":{"row":13067,"text_sha256":"0b689018d753bf1e0bab2566ca49fd325829eb7f0cce0eaf324421c61f7eb07b","row_sha256":"18d05dc22cd35d391f093924dbfcf17482a64e1d2d9780a90fd9b43ca842618f","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13067","group_id":"mmlu/test/13067","variant":"clean"}},"state_tokens":17,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":86,"state_len":17,"decide":85,"opts":[36,52,67,84],"row_ids_sha256":"bbea807c8128e322141fc729eca8bb2ed26345555ddb58922d2d93c5a9f158a0","oracle":{"probs":[0.030564919114112854,0.19830618798732758,0.01773613877594471,0.75339275598526],"logits_raw":[-5.2215399742126465,-0.722324013710022,-6.531028747558594,2.4892101287841797],"argmax":3,"top2_margin":0.5550865530967712,"near_tie":false},"graph":{"p":[0.030796267092227936,0.19932417571544647,0.01787295937538147,0.7520065903663635],"max_abs_dp":0.0013861656188964844,"argmax_equal":true}}]},{"id":"tv4_050","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":50,"_meta":{"row":2282,"text_sha256":"ca192f34bda6eb5a55ae413e76503d8d9a8481bf2c86e8bad24fcf2cadfd75d2","row_sha256":"18dffaae14139a6757aa0e55718041a4ea0795baf3588fae0d85de5093163620","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/2282","group_id":"mmlu/test/2282","variant":"clean"}},"state_tokens":28,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":73,"state_len":28,"decide":72,"opts":[43,51,61,71],"row_ids_sha256":"b88865e49c919e191b5bde05ad9bf73c65f3eb16885ea1df37472a63fd03dbba","oracle":{"probs":[0.8219895362854004,0.1754050850868225,0.0008795431349426508,0.0017258162843063474],"logits_raw":[-0.1622534990310669,-3.878709554672241,-16.619829177856445,-14.998023986816406],"argmax":0,"top2_margin":0.6465844511985779,"near_tie":false},"graph":{"p":[0.8231116533279419,0.17423541843891144,0.0009033011156134307,0.0017496501095592976],"max_abs_dp":0.0011696666479110718,"argmax_equal":true}}]},{"id":"tv4_051","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":51,"_meta":{"row":10179,"text_sha256":"ea4396ca54518fad07b8b5961da2009fae7b812110014f09cb227ca35c027a9c","row_sha256":"1953357a0b60b7cc6360565c6031204ac39ec21a61b1f3fb6c3cf9af4632a3cb","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/10179","group_id":"mmlu/test/10179","variant":"clean"}},"state_tokens":35,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":136,"state_len":35,"decide":135,"opts":[63,94,125,134],"row_ids_sha256":"2d204d8ec15232566ce7a734fdc991e6fdb6bb715f183cfe2fc32955f411bf9e","oracle":{"probs":[0.08638008683919907,0.03159793093800545,0.661391019821167,0.22063098847866058],"logits_raw":[0.2494482845067978,-2.1702334880828857,5.14717435836792,2.5056838989257812],"argmax":2,"top2_margin":0.4407600164413452,"near_tie":false},"graph":{"p":[0.08746010810136795,0.03160879388451576,0.6588364839553833,0.22209462523460388],"max_abs_dp":0.0025545358657836914,"argmax_equal":true}}]},{"id":"tv4_052","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":52,"_meta":{"row":2982,"text_sha256":"98874b205296b4f10427e4fd13b34e861d28b2cc1eb15e895246a288e4223ed3","row_sha256":"197fc93a439226b164e6b58a655da65fbb98aa5fa9efd63e0d8adcae40ae2b9a","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/2982","group_id":"mmlu/test/2982","variant":"clean"}},"state_tokens":29,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":132,"state_len":29,"decide":131,"opts":[54,80,101,130],"row_ids_sha256":"34f4b51bd52aeb4916f7fccddb7f714c79cbdef6997c9ea85a8fc9bcca823e3f","oracle":{"probs":[0.9397506713867188,0.020810909569263458,0.01043710857629776,0.02900139056146145],"logits_raw":[-0.5200018882751465,-9.687382698059082,-11.347821235656738,-8.888895034790039],"argmax":0,"top2_margin":0.9107492566108704,"near_tie":false},"graph":{"p":[0.9431018233299255,0.019451646134257317,0.009971019811928272,0.027475491166114807],"max_abs_dp":0.003351151943206787,"argmax_equal":true}}]},{"id":"tv4_053","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":53,"_meta":{"row":10687,"text_sha256":"8ad6ed1ee05fb239397328c42c3635969b0116c8cd706783db3b539d69442150","row_sha256":"1afab0887f5eed8238a62ad38710101c9a670e8fc7c0e297ee8870bc7da16ca5","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/10687","group_id":"mmlu/test/10687","variant":"clean"}},"state_tokens":209,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":358,"state_len":209,"decide":357,"opts":[247,291,333,356],"row_ids_sha256":"33f5f00b22fd8282092fd51dc0e2550d901fa84853eb4693ee1b200def3a2ed8","oracle":{"probs":[0.28537091612815857,0.100547194480896,0.1014336571097374,0.5126482844352722],"logits_raw":[-1.7862299680709839,-4.296131134033203,-4.2750115394592285,-0.3767654001712799],"argmax":3,"top2_margin":0.22727736830711365,"near_tie":false},"graph":{"p":[0.2854436933994293,0.10184714198112488,0.10175660997629166,0.5109524726867676],"max_abs_dp":0.0016958117485046387,"argmax_equal":true}}]},{"id":"tv4_054","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":54,"_meta":{"row":931,"text_sha256":"bffdafb5dfcb1e38747797c28c9240ba377547c3a3b7918284db1bf3bcf291f0","row_sha256":"1b19c3c4efa3abc6ba91131741026207079d43fb02ec9d8da8d87d08eeae05ea","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/931","group_id":"mmlu/test/931","variant":"clean"}},"state_tokens":28,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":113,"state_len":28,"decide":112,"opts":[54,73,92,111],"row_ids_sha256":"9c1e426c83e5fae677ff24bf445e50fbc1a2e64fc7563dcf5aacdd52bc093867","oracle":{"probs":[0.25471827387809753,0.23990139365196228,0.2584570646286011,0.24692322313785553],"logits_raw":[-1.6546567678451538,-1.798851728439331,-1.61959707736969,-1.7294385433197021],"argmax":2,"top2_margin":0.00373879075050354,"near_tie":true},"graph":{"p":[0.2544136345386505,0.23997844755649567,0.2588590383529663,0.24674883484840393],"max_abs_dp":0.0004019737243652344,"argmax_equal":true}}]},{"id":"tv4_055","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":55,"_meta":{"row":10372,"text_sha256":"600f6c0edf1596b591557e7d464d168fee58a76d37bccd04899ea67277ca8440","row_sha256":"1d98a7be5205e8cdca077a6afd8d4b19c680d2b6dbeed56307facafc6211e788","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/10372","group_id":"mmlu/test/10372","variant":"clean"}},"state_tokens":132,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":189,"state_len":132,"decide":188,"opts":[151,163,175,187],"row_ids_sha256":"891c725a3cf6f51633b6161f48f35a9aab3efab8af50d1c341f12cd86f6449c6","oracle":{"probs":[0.2524462044239044,0.20208807289600372,0.323058545589447,0.22240713238716125],"logits_raw":[-2.5850064754486084,-3.120339870452881,-1.9915895462036133,-2.8898258209228516],"argmax":2,"top2_margin":0.0706123411655426,"near_tie":false},"graph":{"p":[0.25381752848625183,0.20171837508678436,0.32363641262054443,0.22082772850990295],"max_abs_dp":0.0015794038772583008,"argmax_equal":true}}]},{"id":"tv4_056","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":56,"_meta":{"row":3523,"text_sha256":"0a55d50ec128876b1b93a1817c50be49da2b8d9ba9da57fe6b8e68bc7fc1f804","row_sha256":"1f67a2db1cb8de1eb8994cb200cca72336ee543d0d712feeac0f7377fc9ccc0b","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/3523","group_id":"mmlu/test/3523","variant":"clean"}},"state_tokens":26,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":69,"state_len":26,"decide":68,"opts":[42,51,60,67],"row_ids_sha256":"454556680a4aa5876baf41895722621ea62dd4299aba297af68a24711c2ce24a","oracle":{"probs":[0.1683899313211441,0.21268104016780853,0.18917499482631683,0.4297540485858917],"logits_raw":[-2.2238306999206543,-1.6619908809661865,-1.9437901973724365,0.0304715596139431],"argmax":3,"top2_margin":0.2170730084180832,"near_tie":false},"graph":{"p":[0.16918087005615234,0.21167050302028656,0.18922710418701172,0.42992156744003296],"max_abs_dp":0.0010105371475219727,"argmax_equal":true}}]},{"id":"tv4_057","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":57,"_meta":{"row":6245,"text_sha256":"232d792150da6346024c6becbeb1ed6e41d7b8225d3afc5408dc51c7d2883b3f","row_sha256":"1fa4a3b6a91ab2aeb7bb098782fa05e9bad721b7a1db556e4fbb62d55ab70c04","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6245","group_id":"mmlu/test/6245","variant":"clean"}},"state_tokens":27,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":63,"state_len":27,"decide":62,"opts":[41,46,55,61],"row_ids_sha256":"46eeacb8dea027a3ddd3b0234eb0cc30fcb434638dc8316c85e19190c7102b11","oracle":{"probs":[0.19384539127349854,0.27376386523246765,0.2139548808336258,0.3184359073638916],"logits_raw":[-2.369206190109253,-1.538625717163086,-2.131718873977661,-1.174938440322876],"argmax":3,"top2_margin":0.04467204213142395,"near_tie":false},"graph":{"p":[0.19396117329597473,0.27357563376426697,0.21385209262371063,0.31861114501953125],"max_abs_dp":0.0001882314682006836,"argmax_equal":true}}]},{"id":"tv4_058","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":58,"_meta":{"row":9182,"text_sha256":"c041a95d73eaf3622c37781715d7651e4e11b576acfd78959990153d966a28b4","row_sha256":"202af939f40c369cf54f711791efd619354fd75d0a8159f50bff9d69acd88401","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/9182","group_id":"mmlu/test/9182","variant":"clean"}},"state_tokens":88,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":129,"state_len":88,"decide":128,"opts":[102,110,118,127],"row_ids_sha256":"bb4cca9c4746cfa4c349e9c849b0391961ba77b3616d6b4c91873b862cb39cad","oracle":{"probs":[0.21142752468585968,0.2194308489561081,0.4589100778102875,0.11023150384426117],"logits_raw":[-2.066162347793579,-1.9767658710479736,-0.201541006565094,-3.6332218647003174],"argmax":2,"top2_margin":0.23947922885417938,"near_tie":false},"graph":{"p":[0.2100546509027481,0.21851417422294617,0.46197232604026794,0.10945888608694077],"max_abs_dp":0.0030622482299804688,"argmax_equal":true}}]},{"id":"tv4_059","source":"transfer_v4_dev_head","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":59,"_meta":{"row":5150,"text_sha256":"521d509885166a58de2627c79846a9aa42a22d22d83f8ffcfadf1e28d3e106e3","row_sha256":"2036db6f4f015742ccc14fe5697c9e51c566c5dd072f2b40b6f44fd892f4b76c","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/5150","group_id":"mmlu/test/5150","variant":"clean"}},"state_tokens":20,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":102,"state_len":20,"decide":101,"opts":[47,64,81,100],"row_ids_sha256":"1f563ddc9348b71c1e853b5f77322f2a29ad9a8b77a45a0ddbb035769b97e14c","oracle":{"probs":[0.9421905279159546,0.023069903254508972,0.010752439498901367,0.023987090215086937],"logits_raw":[2.760894298553467,-6.164777755737305,-8.001546859741211,-6.0709733963012695],"argmax":0,"top2_margin":0.9182034134864807,"near_tie":false},"graph":{"p":[0.9417498707771301,0.023240400478243828,0.010779148899018764,0.02423059567809105],"max_abs_dp":0.0004406571388244629,"argmax_equal":true}}]},{"id":"tv4x_emotion_00","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":80,"_meta":{"row":1012,"text_sha256":"6aa999c7c36768c0b24075e2e822f1b740d65573d11fb4a480891f642000d34d","row_sha256":"00004983dc48fe12fb4541c466c19f3119120f8c61b237a660984044be80c62f","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1012","group_id":"emotion/test/1012","variant":"clean"}},"state_tokens":12,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":42,"state_len":12,"decide":41,"opts":[23,26,29,32,36,40],"row_ids_sha256":"91b69dc52a9f42df303442655b992693e102559bcb75c2161196caa3fb946902","oracle":{"probs":[0.06544811278581619,0.18583767116069794,0.5508414506912231,0.02951323240995407,0.03811100125312805,0.13024857640266418],"logits_raw":[-7.030908584594727,-4.519916534423828,-1.9055664539337158,-8.947132110595703,-8.331991195678711,-5.375096321105957],"argmax":2,"top2_margin":0.365003764629364,"near_tie":false},"graph":{"p":[0.06569533050060272,0.18564553558826447,0.5497280359268188,0.029661018401384354,0.0382767952978611,0.13099327683448792],"max_abs_dp":0.0011134147644042969,"argmax_equal":true}}]},{"id":"tv4x_emotion_01","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":81,"_meta":{"row":876,"text_sha256":"d1f1ee7d007de6f0a8f54aac3f50b417a1486ef7e7978ce1b94b83fe3e7281da","row_sha256":"00a2a1d85a3a3b058f34f6fe3cbe33246abe45ec564ad5a1c00a6e62a653436c","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/876","group_id":"emotion/test/876","variant":"clean"}},"state_tokens":8,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":38,"state_len":8,"decide":37,"opts":[19,22,25,28,32,36],"row_ids_sha256":"b09c2ad57738602426fb2b7431b5b2a36fb93a2af4888d65c574c7538a79ef11","oracle":{"probs":[0.20891447365283966,0.017065487802028656,0.013920889236032963,0.531512975692749,0.06585066020488739,0.1627354770898819],"logits_raw":[-6.966256618499756,-12.993090629577637,-13.483125686645508,-4.719480514526367,-9.744126319885254,-7.567285060882568],"argmax":3,"top2_margin":0.32259851694107056,"near_tie":false},"graph":{"p":[0.21095582842826843,0.017088163644075394,0.013955294154584408,0.530514121055603,0.06597243994474411,0.16151416301727295],"max_abs_dp":0.002041354775428772,"argmax_equal":true}}]},{"id":"tv4x_emotion_02","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":82,"_meta":{"row":230,"text_sha256":"d4f0ae6c4554b83a2c946c3889646c017a9cb83069617197c6ceeb2e80e6da83","row_sha256":"017bb61de48393f84791a2a30f5106a6c15b572d2b180da2ab81f431b9185733","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/230","group_id":"emotion/test/230","variant":"clean"}},"state_tokens":6,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":36,"state_len":6,"decide":35,"opts":[17,20,23,26,30,34],"row_ids_sha256":"dd80cd0ab2975685816385b980b84ab9bff5aba55ceaf0ead02e985a4a46a857","oracle":{"probs":[0.8262603878974915,0.0025129031855612993,0.00191875163000077,0.05043194442987442,0.10073330253362656,0.018142683431506157],"logits_raw":[0.23545977473258972,-13.708732604980469,-14.35779857635498,-6.492542266845703,-4.827912330627441,-8.952384948730469],"argmax":0,"top2_margin":0.7255271077156067,"near_tie":false},"graph":{"p":[0.8257903456687927,0.0025415339041501284,0.0019467315869405866,0.050620414316654205,0.10100150108337402,0.01809948869049549],"max_abs_dp":0.00047004222869873047,"argmax_equal":true}}]},{"id":"tv4x_emotion_03","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":83,"_meta":{"row":20,"text_sha256":"48561b46ae7c00bedfbe6e7560f1bc0c33837955c08902b3a62e5321eeb396ae","row_sha256":"01f0a333c9946da779ee21cf036b16ab5820bdeb220dba361488eec9aee20d57","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/20","group_id":"emotion/test/20","variant":"clean"}},"state_tokens":43,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":73,"state_len":43,"decide":72,"opts":[54,57,60,63,67,71],"row_ids_sha256":"29855e6fb709ed31c9e866334477dabf099f68da592a0173399c7cf820922298","oracle":{"probs":[0.209742933511734,0.09120327234268188,0.6777969002723694,0.006179373245686293,0.008222498930990696,0.006855050567537546],"logits_raw":[-1.4079370498657227,-3.4116759300231934,1.4142754077911377,-9.888459205627441,-9.201153755187988,-9.638786315917969],"argmax":2,"top2_margin":0.4680539667606354,"near_tie":false},"graph":{"p":[0.2104146033525467,0.09186331182718277,0.67625892162323,0.006235938984900713,0.008297285065054893,0.006929944735020399],"max_abs_dp":0.0015379786491394043,"argmax_equal":true}}]},{"id":"tv4x_emotion_04","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":84,"_meta":{"row":1635,"text_sha256":"25b4f38b3acba992e9f64c7a7ec53d190e099eb4a2618cb2428f157836f55f64","row_sha256":"024d94cd7aeffeb38fe329e8920ff3d127ddbf8d0d3b7b63d5dac2eebdedad67","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1635","group_id":"emotion/test/1635","variant":"clean"}},"state_tokens":9,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":39,"state_len":9,"decide":38,"opts":[20,23,26,29,33,37],"row_ids_sha256":"c16faa5fe08283f5aa691d1e617d347ff7baaf4595256b516b302284dc70321e","oracle":{"probs":[0.001362483948469162,0.9615828990936279,0.029116719961166382,0.0008227062062360346,0.000999560346826911,0.006115653086453676],"logits_raw":[-13.017724990844727,2.764209747314453,-5.650392532348633,-14.231493949890137,-13.762996673583984,-9.404939651489258],"argmax":1,"top2_margin":0.9324661493301392,"near_tie":false},"graph":{"p":[0.0013877680758014321,0.9612674713134766,0.029338641092181206,0.0008398788049817085,0.0010279767448082566,0.006138402968645096],"max_abs_dp":0.0003154277801513672,"argmax_equal":true}}]},{"id":"tv4x_emotion_05","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":85,"_meta":{"row":1435,"text_sha256":"665c7b7f8bc910f661810f975c8e50ae78eafc69f4331cbbfa4e0aa6ac02cc8a","row_sha256":"025d147634ffff48088a9ad95b2a2f87a3051a1fac9f6b88cacf7b9a02bc243e","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1435","group_id":"emotion/test/1435","variant":"clean"}},"state_tokens":22,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":52,"state_len":22,"decide":51,"opts":[33,36,39,42,46,50],"row_ids_sha256":"e7ce2d5577ade9a454f07ab45dad7c3af45d07533df885c4232319835826273e","oracle":{"probs":[0.22445592284202576,0.25750458240509033,0.06769024580717087,0.3542887568473816,0.044396210461854935,0.0516643151640892],"logits_raw":[-3.751203775405884,-3.420712947845459,-6.635425567626953,-2.653003215789795,-7.6502685546875,-7.285479545593262],"argmax":3,"top2_margin":0.09678417444229126,"near_tie":false},"graph":{"p":[0.22473759949207306,0.2569253444671631,0.06775875389575958,0.3542366623878479,0.04463055357336998,0.05171114206314087],"max_abs_dp":0.0005792379379272461,"argmax_equal":true}}]},{"id":"tv4x_emotion_06","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":86,"_meta":{"row":639,"text_sha256":"cc97f8dda8e945ac9d7aabd8ef5e7ff18662a31aaf94beb49d72434087a4ebca","row_sha256":"02ab69252c9985b80f1d7818cd086cd6b4b0a8a3d338d127d5b3c832c7930390","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/639","group_id":"emotion/test/639","variant":"clean"}},"state_tokens":7,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":37,"state_len":7,"decide":36,"opts":[18,21,24,27,31,35],"row_ids_sha256":"f151252ca40a36c0e41df6d1ab3e4008c9e9e4bcbee9da781147101d20bcc89f","oracle":{"probs":[0.005317712668329477,0.38509461283683777,0.5455541610717773,0.0064971125684678555,0.006884221453219652,0.050652142614126205],"logits_raw":[-11.078340530395508,-0.7745610475540161,0.06349751353263855,-10.596373558044434,-10.457124710083008,-5.655251502990723],"argmax":2,"top2_margin":0.16045954823493958,"near_tie":false},"graph":{"p":[0.005412975791841745,0.3864467442035675,0.5435004234313965,0.006582241039723158,0.0069719841703772545,0.05108564347028732],"max_abs_dp":0.0020537376403808594,"argmax_equal":true}}]},{"id":"tv4x_emotion_07","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":87,"_meta":{"row":281,"text_sha256":"2d284bed307dbadc63087b1cf19db78694e59001d29833dbb6b7677743978954","row_sha256":"02eeedfe34d1d5c5c4089939a0fbbb334f7c3400c4230a5677cda20d3220dbcc","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/281","group_id":"emotion/test/281","variant":"clean"}},"state_tokens":7,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":37,"state_len":7,"decide":36,"opts":[18,21,24,27,31,35],"row_ids_sha256":"fe2860d15b341556579e3101132bd4532539a6ecd1a0c5c9e0eac92356502909","oracle":{"probs":[0.9051250219345093,0.002383862156420946,0.0008609853684902191,0.05718396604061127,0.021863725036382675,0.012582341209053993],"logits_raw":[0.33205118775367737,-13.958324432373047,-16.408645629882812,-6.312976837158203,-8.626260757446289,-9.955686569213867],"argmax":0,"top2_margin":0.8479410409927368,"near_tie":false},"graph":{"p":[0.9044409990310669,0.0024015484377741814,0.0008833897300064564,0.05746317654848099,0.0221309382468462,0.012680006213486195],"max_abs_dp":0.0006840229034423828,"argmax_equal":true}}]},{"id":"tv4x_emotion_08","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":88,"_meta":{"row":790,"text_sha256":"f03cc3f062e67ad773241fc1f28f1bd6aa5b3fe45ed5c39046feb017591253c6","row_sha256":"0448b9b74ada8136d2e437a1d7d92c25eede055a11baa59da5ab93cd91cc64e8","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/790","group_id":"emotion/test/790","variant":"clean"}},"state_tokens":22,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":52,"state_len":22,"decide":51,"opts":[33,36,39,42,46,50],"row_ids_sha256":"ed3f31d77c86f319d2ed54ae2fd0765e2e8aa7475a0d781466d9d4fdedd8288f","oracle":{"probs":[0.018434368073940277,0.7135462164878845,0.22295646369457245,0.004825940355658531,0.006087275221943855,0.03414968028664589],"logits_raw":[-7.690692901611328,1.1058990955352783,-1.692988395690918,-10.915307998657227,-10.356636047363281,-6.207275390625],"argmax":1,"top2_margin":0.4905897378921509,"near_tie":false},"graph":{"p":[0.018702445551753044,0.7130109667778015,0.22301754355430603,0.004859312903136015,0.00614077877253294,0.03426893427968025],"max_abs_dp":0.0005352497100830078,"argmax_equal":true}}]},{"id":"tv4x_emotion_09","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":89,"_meta":{"row":1948,"text_sha256":"643e031f6d736b138b810ffcd5374c2edf907aad2227273fd3dac28ec809b77c","row_sha256":"067e3ced09e2e785889dbbf0883308f485c57906c95ba0be86a6eb2a67ecc2a9","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1948","group_id":"emotion/test/1948","variant":"clean"}},"state_tokens":31,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":61,"state_len":31,"decide":60,"opts":[42,45,48,51,55,59],"row_ids_sha256":"c58fa07f7e8a398d69fec85ba9834a6b22ab559f8d1544a3de2b30beac9eb9ca","oracle":{"probs":[0.5489015579223633,0.007121729198843241,0.006978669669479132,0.12479979544878006,0.1244736909866333,0.1877245008945465],"logits_raw":[-1.771899700164795,-12.225628852844238,-12.274454116821289,-5.335761070251465,-5.3420562744140625,-4.353455543518066],"argmax":0,"top2_margin":0.3611770570278168,"near_tie":false},"graph":{"p":[0.5474990010261536,0.007232126779854298,0.00705518526956439,0.12524455785751343,0.12408798933029175,0.18888124823570251],"max_abs_dp":0.0014025568962097168,"argmax_equal":true}}]},{"id":"tv4x_emotion_10","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":90,"_meta":{"row":616,"text_sha256":"4176ff72f271799d281cea4c289d899b8631b8843f7745fb1d57d3cee3ae5eb8","row_sha256":"06adb02feddb5c0bba16c41cced75756d41f423fbff9795916d1bdce28039f48","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/616","group_id":"emotion/test/616","variant":"clean"}},"state_tokens":10,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":40,"state_len":10,"decide":39,"opts":[21,24,27,30,34,38],"row_ids_sha256":"ea1c08cd4f09a9af6372fd075dbf818d96ac51170772a61c8a46a7d7a1d40f4c","oracle":{"probs":[0.7314901351928711,0.006917773280292749,0.0037303536664694548,0.1429818570613861,0.07975796610116959,0.03512194752693176],"logits_raw":[-1.1627123355865479,-12.377286911010742,-13.863241195678711,-5.090266704559326,-6.494729042053223,-8.468099594116211],"argmax":0,"top2_margin":0.5885082483291626,"near_tie":false},"graph":{"p":[0.7299392819404602,0.0069969226606190205,0.003791102673858404,0.14374887943267822,0.08012830466032028,0.035395581275224686],"max_abs_dp":0.0015508532524108887,"argmax_equal":true}}]},{"id":"tv4x_emotion_11","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":91,"_meta":{"row":1171,"text_sha256":"bfb1daa85dda1a6897035bac2a77661f7002d68a36cdd52b85c585af4d7958a8","row_sha256":"077454a516cd975420be15757456ce45684963fd916e47385f0f941d2f96518c","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1171","group_id":"emotion/test/1171","variant":"clean"}},"state_tokens":18,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":48,"state_len":18,"decide":47,"opts":[29,32,35,38,42,46],"row_ids_sha256":"167e4c25d3c90e05688d5168626520aafc46f9b5b3b9b4e26c4b2c1755715556","oracle":{"probs":[0.008580156601965427,0.8114964962005615,0.10067281872034073,0.007398229092359543,0.009630010463297367,0.06222233921289444],"logits_raw":[-12.456293106079102,-1.5101418495178223,-6.531578063964844,-12.812896728515625,-12.178557395935059,-7.689277648925781],"argmax":1,"top2_margin":0.710823655128479,"near_tie":false},"graph":{"p":[0.00861247070133686,0.8116050958633423,0.10092554986476898,0.007394245360046625,0.009575610980391502,0.061886951327323914],"max_abs_dp":0.0003353878855705261,"argmax_equal":true}}]},{"id":"tv4x_emotion_12","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":92,"_meta":{"row":1126,"text_sha256":"6f7a8045cf2af8513305b849b51ea6b978643f9b7d2006fb29851c172089324b","row_sha256":"0776ff3e8ecc84647559298dc85c267389e4d6fcf9238915d5c0f81b7715ad45","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1126","group_id":"emotion/test/1126","variant":"clean"}},"state_tokens":52,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":82,"state_len":52,"decide":81,"opts":[63,66,69,72,76,80],"row_ids_sha256":"8763d7c2054016aeb3c288ce69dd37964286c39dba30ea701e8bc0f3e271919c","oracle":{"probs":[0.10293987393379211,0.0880986824631691,0.7224677205085754,0.03678733482956886,0.025222107768058777,0.02448427677154541],"logits_raw":[-5.5576558113098145,-5.932248115539551,-0.8694007396697998,-8.03346061706543,-8.941582679748535,-9.013017654418945],"argmax":2,"top2_margin":0.6195278167724609,"near_tie":false},"graph":{"p":[0.10322819650173187,0.08926719427108765,0.7200989127159119,0.03700976073741913,0.025537027046084404,0.024858858436346054],"max_abs_dp":0.0023688077926635742,"argmax_equal":true}}]},{"id":"tv4x_emotion_13","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":93,"_meta":{"row":180,"text_sha256":"1f9892f77e7ed6d783b6d8da629f00bea0e96eed3f3ff7b7d7bbe32132e7aac9","row_sha256":"07ed2fa62f9c4bff059a7aa9e7d11ebb2b9cefb212f24b8950ae3e79f6adfe61","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/180","group_id":"emotion/test/180","variant":"clean"}},"state_tokens":30,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":60,"state_len":30,"decide":59,"opts":[41,44,47,50,54,58],"row_ids_sha256":"59e96b6412473bb5d67ed4f3df839fa1bc292a7460c723f6732e62e4bc1e009d","oracle":{"probs":[0.21903492510318756,0.17596998810768127,0.058871615678071976,0.012190203182399273,0.10542736947536469,0.42850589752197266],"logits_raw":[-2.7889621257781982,-3.3156893253326416,-5.950203895568848,-9.73907470703125,-4.548287391662598,-1.1743263006210327],"argmax":5,"top2_margin":0.2094709724187851,"near_tie":false},"graph":{"p":[0.22129987180233002,0.17464202642440796,0.059097304940223694,0.012389068491756916,0.1063733696937561,0.42619839310646057],"max_abs_dp":0.002307504415512085,"argmax_equal":true}}]},{"id":"tv4x_emotion_14","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":94,"_meta":{"row":1333,"text_sha256":"a5eb09875a8231481d14d26be9641d79af8f6515ea3c66d798dbd9188a5294d0","row_sha256":"08da79fda3a7a400c05c8070bd8424133272f026de32fbea20525eab986c845f","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1333","group_id":"emotion/test/1333","variant":"clean"}},"state_tokens":9,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":39,"state_len":9,"decide":38,"opts":[20,23,26,29,33,37],"row_ids_sha256":"fd1d51ba09159c2a60042c80a066d8042ac6fbce0c1d6c3355eaca5da7d17ff7","oracle":{"probs":[0.11501190066337585,0.059434328228235245,0.7777829170227051,0.014058646745979786,0.018043741583824158,0.015668459236621857],"logits_raw":[-4.114556789398193,-5.702943325042725,0.4843955934047699,-9.171586990356445,-8.571130752563477,-8.91074275970459],"argmax":2,"top2_margin":0.6627709865570068,"near_tie":false},"graph":{"p":[0.1139589473605156,0.059892140328884125,0.7780802249908447,0.014134276658296585,0.01810256391763687,0.015831902623176575],"max_abs_dp":0.00105295330286026,"argmax_equal":true}}]},{"id":"tv4x_emotion_15","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":95,"_meta":{"row":254,"text_sha256":"3504fdaa8caeae7312ea8043b6b1aca152a3b60668c977b418fb2238a0b67456","row_sha256":"0cdffbc03f8cf7c3a8869ab0159b89aa4c7460f30b36bdeaf1a398e0475bff7f","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/254","group_id":"emotion/test/254","variant":"clean"}},"state_tokens":14,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":44,"state_len":14,"decide":43,"opts":[25,28,31,34,38,42],"row_ids_sha256":"5c6a20e1433482070e2f06aa58b27796e9236057fa4f69ef977eeb3621a54aba","oracle":{"probs":[0.0005994108505547047,0.48866647481918335,0.4957684278488159,0.0008311826968565583,0.0008665233035571873,0.01326804794371128],"logits_raw":[-13.189797401428223,2.9391307830810547,2.973846912384033,-12.403253555297852,-12.303067207336426,-5.737858772277832],"argmax":2,"top2_margin":0.007101953029632568,"near_tie":true},"graph":{"p":[0.00060839339857921,0.49035248160362244,0.4939420521259308,0.0008408123976550996,0.0008809395367279649,0.013375346548855305],"max_abs_dp":0.0018263757228851318,"argmax_equal":true}}]},{"id":"tv4x_emotion_16","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":96,"_meta":{"row":1401,"text_sha256":"166ab5bc6613551e8ad7e2fcbd87a91c45d1fde27d0c2e39aa6c74a7463d13cc","row_sha256":"0d130863dde313fe1ebd0f91e6dacf58ff43921c9bd180d33e24a96e7f7012ca","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1401","group_id":"emotion/test/1401","variant":"clean"}},"state_tokens":20,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":50,"state_len":20,"decide":49,"opts":[31,34,37,40,44,48],"row_ids_sha256":"20e8328c52c4297f6eec5c3edac9acea00716a48e4bd49772142d25e731a301a","oracle":{"probs":[0.8155958652496338,0.004178383853286505,0.04384295269846916,0.07423114776611328,0.035556625574827194,0.026595033705234528],"logits_raw":[0.46340954303741455,-12.22608470916748,-6.5702080726623535,-5.303255081176758,-7.074245452880859,-7.772967338562012],"argmax":0,"top2_margin":0.7413647174835205,"near_tie":false},"graph":{"p":[0.8171647787094116,0.004138887394219637,0.042668331414461136,0.07425574213266373,0.03548067808151245,0.02629164047539234],"max_abs_dp":0.001568913459777832,"argmax_equal":true}}]},{"id":"tv4x_emotion_17","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":97,"_meta":{"row":1025,"text_sha256":"6e30edf5e02d8cf4daf79f0498e8c75658975419d72d4de78abd43fce0c12c9f","row_sha256":"0d5648bb81f5516a5b3241e749701d3f6e6b0f74fde6d8546526a7e2ae0da18a","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1025","group_id":"emotion/test/1025","variant":"clean"}},"state_tokens":26,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":56,"state_len":26,"decide":55,"opts":[37,40,43,46,50,54],"row_ids_sha256":"d9aa61badd05df2cd4b4979da3e0483034c7463788f62ade48a8963ab6a0b037","oracle":{"probs":[0.7420860528945923,0.006265836767852306,0.015443632379174232,0.12920740246772766,0.06856587529182434,0.03843125328421593],"logits_raw":[-1.3505859375,-12.83791732788086,-10.667457580566406,-5.5564727783203125,-7.081003665924072,-8.473923683166504],"argmax":0,"top2_margin":0.612878680229187,"near_tie":false},"graph":{"p":[0.7417684197425842,0.006292431149631739,0.01536698080599308,0.12882652878761292,0.06920205056667328,0.038543637841939926],"max_abs_dp":0.000636175274848938,"argmax_equal":true}}]},{"id":"tv4x_emotion_18","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":98,"_meta":{"row":1576,"text_sha256":"a3432a86c14e8fd2af65a2054d6f2b7b506dfd7b0fce0688701d813831942802","row_sha256":"0d969a10c0b2fe52d83ee11f899fbb5a0afec0690faca0c9dd67e77e0c208001","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1576","group_id":"emotion/test/1576","variant":"clean"}},"state_tokens":37,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":67,"state_len":37,"decide":66,"opts":[48,51,54,57,61,65],"row_ids_sha256":"7d691b2acd7d2993525f40d3420630bcae485701efa578bfef41c58e2736fb2d","oracle":{"probs":[0.8315562605857849,0.0009227924747392535,0.0010654598008841276,0.05575184151530266,0.08880231529474258,0.021901313215494156],"logits_raw":[1.8859492540359497,-14.48397159576416,-14.13808536529541,-4.616132736206055,-3.496110677719116,-6.864258289337158],"argmax":0,"top2_margin":0.7427539229393005,"near_tie":false},"graph":{"p":[0.8310702443122864,0.0009398700203746557,0.0010826775105670094,0.05596768483519554,0.08893387019634247,0.022005712613463402],"max_abs_dp":0.00048601627349853516,"argmax_equal":true}}]},{"id":"tv4x_emotion_19","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":99,"_meta":{"row":1022,"text_sha256":"1ddfafd83f06f7e23367fa275286cb4073ee3492459e8b893a15f320097224fe","row_sha256":"0dfffcf7f68dcd3b88fb7bf4eeb31f74e4f958e82c5eae9c104b8a36f10ba9d0","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1022","group_id":"emotion/test/1022","variant":"clean"}},"state_tokens":44,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":74,"state_len":44,"decide":73,"opts":[55,58,61,64,68,72],"row_ids_sha256":"2e04ad2f7cad9c5d925f6173fd0734a256ef29e12c344123dd35a79b659c20b5","oracle":{"probs":[0.8634060621261597,0.003866505343466997,0.0026103167328983545,0.05027718469500542,0.06137409433722496,0.018465865403413773],"logits_raw":[0.2156108021736145,-12.797592163085938,-13.742879867553711,-6.625592231750488,-6.145738124847412,-9.035557746887207],"argmax":0,"top2_margin":0.8020319938659668,"near_tie":false},"graph":{"p":[0.8630243539810181,0.003898011986166239,0.0026452350430190563,0.050353050231933594,0.06156615912914276,0.018513159826397896],"max_abs_dp":0.00038170814514160156,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_00","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":160,"_meta":{"row":468,"text_sha256":"042fddd95a5cb84ea94b5f130d7a3f2655b9dd2cc3c9d9734f3c6fe4a9eb0544","row_sha256":"00216449d3e3752e425208ef10a57a8247e28b345bc2fc599c3536e3c897a64a","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/468","group_id":"tweet_offensive/test/468","variant":"clean"}},"state_tokens":37,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":68,"state_len":37,"decide":67,"opts":[48,66],"row_ids_sha256":"03509090c09cadf79a04c88fa6e8e8ad5880f453441053b054537bba452292e0","oracle":{"probs":[0.40010854601860046,0.5998914837837219],"logits_raw":[-1.231427550315857,-0.25694629549980164],"argmax":1,"top2_margin":0.19978293776512146,"near_tie":false},"graph":{"p":[0.4002145528793335,0.5997854471206665],"max_abs_dp":0.00010603666305541992,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_01","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":161,"_meta":{"row":80,"text_sha256":"5de6a05f98731ba1d727c74f853a0499e394b7087ce9db7a7a8746c3d38218e4","row_sha256":"002524b5c03ec2a10836416e0d77f8e55757b6055748867bdb59308198897821","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/80","group_id":"tweet_offensive/test/80","variant":"clean"}},"state_tokens":47,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":78,"state_len":47,"decide":77,"opts":[58,76],"row_ids_sha256":"c705b18dff2dca56d543577e6b4cce90fed0bacc49a44d39fbadc1beeeb1acf2","oracle":{"probs":[0.49890169501304626,0.5010982751846313],"logits_raw":[-1.9133301973342896,-1.9027600288391113],"argmax":1,"top2_margin":0.002196580171585083,"near_tie":true},"graph":{"p":[0.4974946677684784,0.502505362033844],"max_abs_dp":0.0014070868492126465,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_02","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":162,"_meta":{"row":469,"text_sha256":"88ba35a0d8d13737edf221e1c5192f316705445420b69a23bc168e134d707b7f","row_sha256":"003ad009f2103f968bb8f94e1523565e9f993ab8438a83cbd1d52c535ac82068","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/469","group_id":"tweet_offensive/test/469","variant":"clean"}},"state_tokens":33,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":64,"state_len":33,"decide":63,"opts":[44,62],"row_ids_sha256":"39e5adb98348db4f133d9f0a76daaa28c1214cdc67a050e8284163e008d514db","oracle":{"probs":[0.9507721066474915,0.049227915704250336],"logits_raw":[-0.10096338391304016,-7.224828720092773],"argmax":0,"top2_margin":0.9015442132949829,"near_tie":false},"graph":{"p":[0.950571596622467,0.049428440630435944],"max_abs_dp":0.0002005249261856079,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_03","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":163,"_meta":{"row":266,"text_sha256":"d1e5b0cdfe48d6e1c4a554b925c9d5494d7c4a42a477e09ac81f12c3052544d5","row_sha256":"007175feb144c988dba800924852588051004622a5e48eeb94f3aef53ad2b20c","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/266","group_id":"tweet_offensive/test/266","variant":"clean"}},"state_tokens":62,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":93,"state_len":62,"decide":92,"opts":[73,91],"row_ids_sha256":"f4bf9c0f928fecfd2b23998fe92df0d889060e96c072b6530670571db860d238","oracle":{"probs":[0.7636762857437134,0.2363237738609314],"logits_raw":[-1.319419264793396,-4.141574382781982],"argmax":0,"top2_margin":0.527352511882782,"near_tie":false},"graph":{"p":[0.763839602470398,0.23616036772727966],"max_abs_dp":0.0001634061336517334,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_04","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":164,"_meta":{"row":190,"text_sha256":"2c87f8da612df68a2880c0dafd3e8d6d73a3e98d96362f4d06f85e5a47183567","row_sha256":"0085fb12775a0ac9206ffc87ad66cd58a290c0e568b1fde3bfa875c3ef8e6b5b","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/190","group_id":"tweet_offensive/test/190","variant":"clean"}},"state_tokens":39,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":70,"state_len":39,"decide":69,"opts":[50,68],"row_ids_sha256":"451be0c5bb8bf2c0b96fe8c20f2ad21183055edd028a703f34ab682a86066817","oracle":{"probs":[0.7378087639808655,0.2621912360191345],"logits_raw":[-0.931801974773407,-3.421126365661621],"argmax":0,"top2_margin":0.47561752796173096,"near_tie":false},"graph":{"p":[0.7367116808891296,0.26328831911087036],"max_abs_dp":0.0010970830917358398,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_05","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":165,"_meta":{"row":294,"text_sha256":"4e4b2f2f8e478ed51893e32c4d46a20b611896760a9c7de50213ef114423f32d","row_sha256":"008c73d5089aea50a11a44db9c482272d0fea99e21b1689b9eac27ef8e226afa","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/294","group_id":"tweet_offensive/test/294","variant":"clean"}},"state_tokens":26,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":57,"state_len":26,"decide":56,"opts":[37,55],"row_ids_sha256":"f90ea150e343213e573a13eab4db1c32f3654ed248be4df50943b54b02032093","oracle":{"probs":[0.884526789188385,0.115473173558712],"logits_raw":[-0.6039500832557678,-5.502702713012695],"argmax":0,"top2_margin":0.7690536379814148,"near_tie":false},"graph":{"p":[0.8838905096054077,0.1161094456911087],"max_abs_dp":0.0006362795829772949,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_06","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":166,"_meta":{"row":429,"text_sha256":"a4af81c32beb18b9bdf20df0327928494dee74916f28d3171b9ab470736566dd","row_sha256":"00d8d7edd3234418ef812978c7f06029994c032935ed57460d110c61b60bf4af","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/429","group_id":"tweet_offensive/test/429","variant":"clean"}},"state_tokens":38,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":69,"state_len":38,"decide":68,"opts":[49,67],"row_ids_sha256":"b7f92b73f348c3f361d3cc1dddf34518e01df02b1125472a808e5a3d17941e5c","oracle":{"probs":[0.5711539387702942,0.4288460910320282],"logits_raw":[-1.393140196800232,-2.0826194286346436],"argmax":0,"top2_margin":0.142307847738266,"near_tie":false},"graph":{"p":[0.5715132355690002,0.42848682403564453],"max_abs_dp":0.0003592967987060547,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_07","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":167,"_meta":{"row":104,"text_sha256":"30fe04abd0e281dc1f9a435c3024db5c6b3f56f6fbe48eed36857c8bccdaa970","row_sha256":"018a0f280d4eadc2f0285f6309d74f7ee58d9e9ffd5ef6058157e1a160e640b1","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/104","group_id":"tweet_offensive/test/104","variant":"clean"}},"state_tokens":28,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":59,"state_len":28,"decide":58,"opts":[39,57],"row_ids_sha256":"71328e249d87ba10ae6295e900315605b8ce04bf6ff6b752b7bacf23b656334f","oracle":{"probs":[0.5422791838645935,0.4577207565307617],"logits_raw":[-1.7990996837615967,-2.206977367401123],"argmax":0,"top2_margin":0.08455842733383179,"near_tie":false},"graph":{"p":[0.5440608263015747,0.4559391438961029],"max_abs_dp":0.0017816424369812012,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_08","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":168,"_meta":{"row":374,"text_sha256":"fd6d9208b2f20e18ec8ab60ed835e8784eca73be8842609f23ec445b8998343d","row_sha256":"02375fb3276b1ff2d02b0dc50c031d48387edb61ee76e64e8880778732f6d1a1","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/374","group_id":"tweet_offensive/test/374","variant":"clean"}},"state_tokens":64,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":95,"state_len":64,"decide":94,"opts":[75,93],"row_ids_sha256":"5d0a2bc42b4f279c49654b2f07641f1199647ec038d02bf3e435484416e7813c","oracle":{"probs":[0.43810319900512695,0.561896800994873],"logits_raw":[-1.1821672916412354,-0.5833886861801147],"argmax":1,"top2_margin":0.1237936019897461,"near_tie":false},"graph":{"p":[0.438068151473999,0.5619319081306458],"max_abs_dp":3.510713577270508e-05,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_09","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":169,"_meta":{"row":274,"text_sha256":"75b8f65838e92d47d261d31612fcd08877c01be3bac45100aa7ca838ca588f9e","row_sha256":"029044b9602c3dca939fb62961ecd2e3d1956469b0c646642b8341e08a3801d0","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/274","group_id":"tweet_offensive/test/274","variant":"clean"}},"state_tokens":56,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":87,"state_len":56,"decide":86,"opts":[67,85],"row_ids_sha256":"2e036fd5f9567256f335e19bc8568e3b4249d690ad2ac689586266b859fda8f1","oracle":{"probs":[0.5308529138565063,0.46914711594581604],"logits_raw":[-1.3137598037719727,-1.611072063446045],"argmax":0,"top2_margin":0.06170579791069031,"near_tie":false},"graph":{"p":[0.5295583009719849,0.47044169902801514],"max_abs_dp":0.0012946128845214844,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_10","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":170,"_meta":{"row":705,"text_sha256":"4b4065f8a854e325b7eac1d7659c71b5b04fb4572e8387069b33f23a5f0e55aa","row_sha256":"03bad7307ab2d5a081314e0c9d858e94c8c5e1f67928550463ee1aba99b643cc","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/705","group_id":"tweet_offensive/test/705","variant":"clean"}},"state_tokens":59,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":90,"state_len":59,"decide":89,"opts":[70,88],"row_ids_sha256":"073c3f554a267c93fbb5333f6f0a40788d7765c10da530e94a75b6b8e9f50ff8","oracle":{"probs":[0.5211887359619141,0.47881120443344116],"logits_raw":[-1.791275143623352,-1.995322346687317],"argmax":0,"top2_margin":0.0423775315284729,"near_tie":false},"graph":{"p":[0.5218141674995422,0.47818586230278015],"max_abs_dp":0.0006254315376281738,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_11","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":171,"_meta":{"row":565,"text_sha256":"f4ae10950b9f2304119b7325310552d6532704e776e0affca1b9d34738f121b9","row_sha256":"0471afc0b417cb9595d712b8141d80709b3189582fd8eb7da882d82ccc41a86b","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/565","group_id":"tweet_offensive/test/565","variant":"clean"}},"state_tokens":44,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":75,"state_len":44,"decide":74,"opts":[55,73],"row_ids_sha256":"09e92c67b5d1d8107da0d0332f0d952b599d4d230ab7e36086b9683d89a31cf4","oracle":{"probs":[0.9498008489608765,0.050199124962091446],"logits_raw":[-0.39756694436073303,-7.471966743469238],"argmax":0,"top2_margin":0.8996016979217529,"near_tie":false},"graph":{"p":[0.9494640827178955,0.050535909831523895],"max_abs_dp":0.00033678486943244934,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_12","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":172,"_meta":{"row":319,"text_sha256":"b56036f176fad759a87256a16e70fa1f45bb2897301e49e488a4d8d76e0aa45d","row_sha256":"04cc0f8ddcbaa3ce3f73c4ef716470eff6b1866ed39fc18da57d6a3cfad3594c","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/319","group_id":"tweet_offensive/test/319","variant":"clean"}},"state_tokens":31,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":62,"state_len":31,"decide":61,"opts":[42,60],"row_ids_sha256":"66e599c3261b48f7ff0f3876cb80ee7dab1c7d376709574b077d08d1c071aaf1","oracle":{"probs":[0.4602143168449402,0.5397857427597046],"logits_raw":[-1.6076929569244385,-1.2239761352539062],"argmax":1,"top2_margin":0.0795714259147644,"near_tie":false},"graph":{"p":[0.4600922763347626,0.5399076342582703],"max_abs_dp":0.0001220405101776123,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_13","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":173,"_meta":{"row":6,"text_sha256":"f7daa080888f2e96d477ff09591a39966d6b215c668edd5b39de0fa64e73b306","row_sha256":"053d0ce5428ac6c728129a3d7f0e49e158cf3cffc792faa1c02cdf0d58da9313","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/6","group_id":"tweet_offensive/test/6","variant":"clean"}},"state_tokens":12,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":43,"state_len":12,"decide":42,"opts":[23,41],"row_ids_sha256":"a791b043a64a2f0b17fd0bef34dd8b6f4e0a27a0c2b1838d5c3ddc9238fae70a","oracle":{"probs":[0.7055423855781555,0.29445770382881165],"logits_raw":[-2.141359806060791,-4.243842124938965],"argmax":0,"top2_margin":0.41108468174934387,"near_tie":false},"graph":{"p":[0.7042614221572876,0.29573854804039],"max_abs_dp":0.00128096342086792,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_14","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":174,"_meta":{"row":81,"text_sha256":"60c28e99188e2f96e8c09978e91ba91028b41721a9af4eb5f7d709cfe722da69","row_sha256":"05599192f542a8c676782bf2fc85c0747a12da0bf74736d5210e0c53ee26c974","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/81","group_id":"tweet_offensive/test/81","variant":"clean"}},"state_tokens":40,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":71,"state_len":40,"decide":70,"opts":[51,69],"row_ids_sha256":"8f243620849c81a22d540c5badc3e4950c9157e54d6de9446815c7d5da77b184","oracle":{"probs":[0.37133005261421204,0.6286699175834656],"logits_raw":[-1.4241447448730469,-0.15732315182685852],"argmax":1,"top2_margin":0.25733986496925354,"near_tie":false},"graph":{"p":[0.37112632393836975,0.6288736462593079],"max_abs_dp":0.00020372867584228516,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_15","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":175,"_meta":{"row":717,"text_sha256":"ad63bfb4131a61055cee25aa15da75ad7c9a4c04237350674bc7a0f732728529","row_sha256":"08244b08a72a4a768a5def4880d7bfdf9ff0bc9eb454b1ce64ed16be7b1a739c","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/717","group_id":"tweet_offensive/test/717","variant":"clean"}},"state_tokens":31,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":62,"state_len":31,"decide":61,"opts":[42,60],"row_ids_sha256":"af4f5e8c265b6cc818323d3985ce2aac5d4a509df1e817dc739429bf2b086c99","oracle":{"probs":[0.6485496163368225,0.3514503836631775],"logits_raw":[-1.045046329498291,-2.519160747528076],"argmax":0,"top2_margin":0.297099232673645,"near_tie":false},"graph":{"p":[0.6483249068260193,0.3516750931739807],"max_abs_dp":0.00022470951080322266,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_16","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":176,"_meta":{"row":342,"text_sha256":"bc17575176b83581b2aca7f78e592319321b4b7a93b6c99e7516c0f7268ea3ff","row_sha256":"0904016e53d21b759cbd0dae5eeac767cc91c55073a2d287835cbd933a1042fd","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/342","group_id":"tweet_offensive/test/342","variant":"clean"}},"state_tokens":73,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":104,"state_len":73,"decide":103,"opts":[84,102],"row_ids_sha256":"a4505f3c4d58a458d078c621d907e3f3f41c8c617b791ddcde3fec86ed870748","oracle":{"probs":[0.15444467961788177,0.845555305480957],"logits_raw":[-2.0479073524475098,2.0427567958831787],"argmax":1,"top2_margin":0.6911106109619141,"near_tie":false},"graph":{"p":[0.15388619899749756,0.8461138606071472],"max_abs_dp":0.0005585551261901855,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_17","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":177,"_meta":{"row":795,"text_sha256":"f6cbb342e60d8a7bc16acd810bd6808b125df7df68cdd25ef929c5e9ce82f44b","row_sha256":"090c2107045145b6feb53fe0fa6460da4d5d9d21fe62228fc2524ed969af2e31","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/795","group_id":"tweet_offensive/test/795","variant":"clean"}},"state_tokens":41,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":72,"state_len":41,"decide":71,"opts":[52,70],"row_ids_sha256":"a13da9de65fb8c97585b396f1abd82201e035a7efb7b3f7111b330af89a3e7fc","oracle":{"probs":[0.4318801164627075,0.5681198835372925],"logits_raw":[-1.4116806983947754,-0.7519792914390564],"argmax":1,"top2_margin":0.13623976707458496,"near_tie":false},"graph":{"p":[0.43118441104888916,0.5688155889511108],"max_abs_dp":0.0006957054138183594,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_18","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":178,"_meta":{"row":123,"text_sha256":"f7d04b1b9f7b432339f846df8a21ef457db7cfe2bdba0901e3f35ed7c5876ed1","row_sha256":"09526509d5650bae8be1c7e93f776850fbe7cbbfecd0ddbd3449572dea9c9516","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/123","group_id":"tweet_offensive/test/123","variant":"clean"}},"state_tokens":26,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":57,"state_len":26,"decide":56,"opts":[37,55],"row_ids_sha256":"480b26b11f53c4fd2f38e8bc6dcc59ad036109f1877ce6355a88971d2cb45099","oracle":{"probs":[0.5747702717781067,0.4252297580242157],"logits_raw":[-1.684709906578064,-2.4097509384155273],"argmax":0,"top2_margin":0.149540513753891,"near_tie":false},"graph":{"p":[0.5735874176025391,0.42641252279281616],"max_abs_dp":0.001182854175567627,"argmax_equal":true}}]},{"id":"tv4x_tweet_offensive_19","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":179,"_meta":{"row":456,"text_sha256":"eb89b9a749ab5e03f8b42fe72711f98dca27030ce88be865758e99841e623db1","row_sha256":"0978d0cb523437b15e499068c8d8585c988e6f0994c5ccc6de659f5a6c5eb9f8","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/456","group_id":"tweet_offensive/test/456","variant":"clean"}},"state_tokens":68,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":99,"state_len":68,"decide":98,"opts":[79,97],"row_ids_sha256":"fb4315b9527400f9d570bcf6dbb9468d800548bbce404f20fef7bc4d3348f760","oracle":{"probs":[0.8412155508995056,0.15878449380397797],"logits_raw":[-0.3315523564815521,-4.3431596755981445],"argmax":0,"top2_margin":0.6824310421943665,"near_tie":false},"graph":{"p":[0.8411219120025635,0.1588781177997589],"max_abs_dp":9.363889694213867e-05,"argmax_equal":true}}]},{"id":"tv4x_qnli_00","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":240,"_meta":{"row":1714,"text_sha256":"9845373b64960d3bca56c5c34aad2173004e49bb0ed8582d1aadc7ef9b6e19a8","row_sha256":"006ac40be6adb5b4046eb22f837b50966d8668fdcfdffd816563e7764bcb939d","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/1714","group_id":"qnli/validation/1714","variant":"clean"}},"state_tokens":67,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":94,"state_len":67,"decide":93,"opts":[89,92],"row_ids_sha256":"b7f4e2955a27a198bbaeb7ed9798ede2bdd684f63eca27ad3ed9e7c70a2ea6fd","oracle":{"probs":[0.010116765275597572,0.9898832440376282],"logits_raw":[-5.380301475524902,5.647571563720703],"argmax":1,"top2_margin":0.9797664880752563,"near_tie":false},"graph":{"p":[0.010213449597358704,0.9897865056991577],"max_abs_dp":9.673833847045898e-05,"argmax_equal":true}}]},{"id":"tv4x_qnli_01","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":241,"_meta":{"row":1900,"text_sha256":"27f0f252e66e0fac367b7361abfe1202b4b8f2f8905a8e843912e9ef3d4e8a84","row_sha256":"011c2da09bec40ee37ae0ab36f27fcedc599855eb59c4d52adb1a98352d98cd3","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/1900","group_id":"qnli/validation/1900","variant":"clean"}},"state_tokens":57,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":95,"state_len":57,"decide":94,"opts":[90,93],"row_ids_sha256":"09d410053044c7097a8571626079f16b56f810052f7bd5e1b29a8dcf648248d3","oracle":{"probs":[0.9825109839439392,0.01748907007277012],"logits_raw":[2.7758421897888184,-6.917015552520752],"argmax":0,"top2_margin":0.9650219082832336,"near_tie":false},"graph":{"p":[0.9819797277450562,0.018020249903202057],"max_abs_dp":0.0005312561988830566,"argmax_equal":true}}]},{"id":"tv4x_qnli_02","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":242,"_meta":{"row":2575,"text_sha256":"5e02a62572edafc14ce561623edf67e95b3b87fdd8029257eb585e419e0749a1","row_sha256":"020a27fa0285b3515c44f4ce37e48ebbff851a1859113e5ff3f3977fd6be5b7e","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/2575","group_id":"qnli/validation/2575","variant":"clean"}},"state_tokens":21,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":51,"state_len":21,"decide":50,"opts":[46,49],"row_ids_sha256":"f9ac77c0be284f3bbf150f598348286d39d786299574391af82e2cc23f276c46","oracle":{"probs":[0.1373477578163147,0.8626522421836853],"logits_raw":[-1.5717438459396362,2.8493621349334717],"argmax":1,"top2_margin":0.7253044843673706,"near_tie":false},"graph":{"p":[0.13790260255336761,0.862097442150116],"max_abs_dp":0.0005548447370529175,"argmax_equal":true}}]},{"id":"tv4x_qnli_03","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":243,"_meta":{"row":1998,"text_sha256":"e1e3ab832833b63a93997f399656af5edae92f9b8fbb7700c39754e7df3c4e92","row_sha256":"022665c82b545868033098220fff3ba296025e2f926e2cb9ce7f77402579a052","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/1998","group_id":"qnli/validation/1998","variant":"clean"}},"state_tokens":29,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":64,"state_len":29,"decide":63,"opts":[59,62],"row_ids_sha256":"77213d97cb09127af8c1135bd22601c01018ccb690389cff244b2d79323afbb9","oracle":{"probs":[0.9792577624320984,0.020742280408740044],"logits_raw":[2.1865246295928955,-7.08788537979126],"argmax":0,"top2_margin":0.958515465259552,"near_tie":false},"graph":{"p":[0.9785689115524292,0.02143106982111931],"max_abs_dp":0.0006888508796691895,"argmax_equal":true}}]},{"id":"tv4x_qnli_04","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":244,"_meta":{"row":4117,"text_sha256":"7d7d122f20925140db28dd81b60bffb3e59ca6587ad47a6f8ddca0083ed3d0ea","row_sha256":"022dedc816cb845ec1ce5b0606af8270a0a54e5cceeae4217ac0efe1b93fc996","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/4117","group_id":"qnli/validation/4117","variant":"clean"}},"state_tokens":98,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":137,"state_len":98,"decide":136,"opts":[132,135],"row_ids_sha256":"721fa2327714a2b19cb896a388275d97e2990557178b17513646f28a4f01ff4c","oracle":{"probs":[0.05259081721305847,0.9474091529846191],"logits_raw":[-2.9087748527526855,4.047571182250977],"argmax":1,"top2_margin":0.8948183059692383,"near_tie":false},"graph":{"p":[0.05300265923142433,0.9469972848892212],"max_abs_dp":0.0004118680953979492,"argmax_equal":true}}]},{"id":"tv4x_qnli_05","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":245,"_meta":{"row":3930,"text_sha256":"1140e42273d8c9d12f7eea7c5227dee7cbf302068f77c87fa21f7a4e2a78fe2f","row_sha256":"025fc0affd878c088f092574c86ffff48cc7bc65c60a9252b783a5c641fcbf19","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/3930","group_id":"qnli/validation/3930","variant":"clean"}},"state_tokens":50,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":83,"state_len":50,"decide":82,"opts":[78,81],"row_ids_sha256":"1bc55cb3a303b9144f1fca1d85c54c2681c11de3490e01a91e27fc5e0b8b1c38","oracle":{"probs":[0.12158852815628052,0.8784114718437195],"logits_raw":[-2.5603690147399902,2.197528839111328],"argmax":1,"top2_margin":0.756822943687439,"near_tie":false},"graph":{"p":[0.12228909879922867,0.8777109384536743],"max_abs_dp":0.0007005706429481506,"argmax_equal":true}}]},{"id":"tv4x_qnli_06","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":246,"_meta":{"row":5417,"text_sha256":"665140834a5729a9df029f9f5e07f1611018081116d85af5bf5bfbc638d639e8","row_sha256":"028ee36d7d2f94d0c48a2c94946cddbe624d57d570c501ecb0bf5ab83a3616fe","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/5417","group_id":"qnli/validation/5417","variant":"clean"}},"state_tokens":38,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":63,"state_len":38,"decide":62,"opts":[58,61],"row_ids_sha256":"82c57e59303c0febf54bac7479aebc3adcf51ed3b5d54c834bbdc0eb6a7a9b40","oracle":{"probs":[0.6725401878356934,0.32745978236198425],"logits_raw":[-0.8409817218780518,-2.5726077556610107],"argmax":0,"top2_margin":0.3450804054737091,"near_tie":false},"graph":{"p":[0.6736097931861877,0.32639026641845703],"max_abs_dp":0.0010696053504943848,"argmax_equal":true}}]},{"id":"tv4x_qnli_07","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":247,"_meta":{"row":182,"text_sha256":"5aa8e116927f50cb4d36a21dfb4bd20b369f3bbda41b163546571a75d9c26aa6","row_sha256":"028ff272234176332316caf860273564fb760bdef0a6d4af1472efec657ddbf1","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/182","group_id":"qnli/validation/182","variant":"clean"}},"state_tokens":28,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":54,"state_len":28,"decide":53,"opts":[49,52],"row_ids_sha256":"72054bef4770b58e4d570ab97df3237035f62d87fc7be1a789b6ce43f6db25ce","oracle":{"probs":[0.7513476610183716,0.2486523687839508],"logits_raw":[0.09672510623931885,-2.563915252685547],"argmax":0,"top2_margin":0.5026953220367432,"near_tie":false},"graph":{"p":[0.7469549179077148,0.2530450224876404],"max_abs_dp":0.004392743110656738,"argmax_equal":true}}]},{"id":"tv4x_qnli_08","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":248,"_meta":{"row":416,"text_sha256":"8b83c32333787aea3ae32141afadc2b8ba4d2b2e339362a161bd335ac007116d","row_sha256":"035a3573cc069f9e593d74cab4fe4385d061ea858ef2e3c92ee735923f3797aa","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/416","group_id":"qnli/validation/416","variant":"clean"}},"state_tokens":44,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":75,"state_len":44,"decide":74,"opts":[70,73],"row_ids_sha256":"bcf3f623de7c8b6820fb767c43b7f4cb19f169647589b0487552a99dd000add5","oracle":{"probs":[0.9943895936012268,0.005610350053757429],"logits_raw":[3.8640124797821045,-8.59334945678711],"argmax":0,"top2_margin":0.9887792468070984,"near_tie":false},"graph":{"p":[0.9942899942398071,0.005710018333047628],"max_abs_dp":9.966827929019928e-05,"argmax_equal":true}}]},{"id":"tv4x_qnli_09","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":249,"_meta":{"row":2695,"text_sha256":"35ec3f161936ec2c98c6f8e0107e2d489fb892440f08271624b9a6cb4a15a003","row_sha256":"038c9f863ce04d7beee4ee19a6bcb0ac3adbe0fa4a39204e13eabf91d0897954","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/2695","group_id":"qnli/validation/2695","variant":"clean"}},"state_tokens":27,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":54,"state_len":27,"decide":53,"opts":[49,52],"row_ids_sha256":"de53ee1725c41e4fb6523e9f4c5fa67e16e3bcd6b74b237c1f6c30ad33c658d5","oracle":{"probs":[0.9873384237289429,0.012661551125347614],"logits_raw":[3.2785794734954834,-7.203240394592285],"argmax":0,"top2_margin":0.9746768474578857,"near_tie":false},"graph":{"p":[0.9873732924461365,0.012626660987734795],"max_abs_dp":3.489013761281967e-05,"argmax_equal":true}}]},{"id":"tv4x_qnli_10","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":250,"_meta":{"row":3379,"text_sha256":"7dba31276b7e6cf3363d040279a05471c77cba50efd16cb0c38f23589b7c748a","row_sha256":"03aea18d3d6cfd05c282a386c17b876674e7951a3e20a1b0a8fa3a530d6ab0cb","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/3379","group_id":"qnli/validation/3379","variant":"clean"}},"state_tokens":44,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":72,"state_len":44,"decide":71,"opts":[67,70],"row_ids_sha256":"e86de004d2313cb034432359b520b6ec2efa6bd6ed15bdd1336309fe411eb1e4","oracle":{"probs":[0.9902093410491943,0.00979069247841835],"logits_raw":[3.5764400959014893,-7.5310516357421875],"argmax":0,"top2_margin":0.9804186224937439,"near_tie":false},"graph":{"p":[0.9900963306427002,0.00990368239581585],"max_abs_dp":0.00011301040649414062,"argmax_equal":true}}]},{"id":"tv4x_qnli_11","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":251,"_meta":{"row":4802,"text_sha256":"8afb77221b7ace20f32e4ecf513b5b903de006034311072ca64fa5c45c304d83","row_sha256":"042f68db744ac90ad8be2bb7b59bba1d5d1e1d3092cd11ee6a90fc1e05e243c6","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/4802","group_id":"qnli/validation/4802","variant":"clean"}},"state_tokens":24,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":50,"state_len":24,"decide":49,"opts":[45,48],"row_ids_sha256":"e22876fdd9c613c2c27a0d526ba1250b2e4a2a64286f74b8e566d0b88445061f","oracle":{"probs":[0.08970481157302856,0.9102951884269714],"logits_raw":[-3.012254238128662,2.5631520748138428],"argmax":1,"top2_margin":0.8205903768539429,"near_tie":false},"graph":{"p":[0.09066072851419449,0.9093392491340637],"max_abs_dp":0.0009559392929077148,"argmax_equal":true}}]},{"id":"tv4x_qnli_12","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":252,"_meta":{"row":1028,"text_sha256":"ad9ce13c16c3a55ee54267faae9122c64ba4aea143fb6ef493355ae74a22f1d2","row_sha256":"0445460bb33270740ff1d3b49dd52a808d7d0eadb94231c0647e40b14caccc65","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/1028","group_id":"qnli/validation/1028","variant":"clean"}},"state_tokens":51,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":84,"state_len":51,"decide":83,"opts":[79,82],"row_ids_sha256":"ed8e7ffe5bbc25356149e065dbc90249d892c29b96e61cebefa9eaf17120221c","oracle":{"probs":[0.06482027471065521,0.9351797103881836],"logits_raw":[-2.9016518592834473,3.5203847885131836],"argmax":1,"top2_margin":0.8703594207763672,"near_tie":false},"graph":{"p":[0.06370001286268234,0.9362999200820923],"max_abs_dp":0.0011202618479728699,"argmax_equal":true}}]},{"id":"tv4x_qnli_13","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":253,"_meta":{"row":1106,"text_sha256":"b22065dbd979956c062f2c27052ffbd3f408f3ab54ad3d943e955ab0dc87843f","row_sha256":"04820391cd042455592727f727b3e59250d725e0de10975a6105e863154454ec","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/1106","group_id":"qnli/validation/1106","variant":"clean"}},"state_tokens":27,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":52,"state_len":27,"decide":51,"opts":[47,50],"row_ids_sha256":"95314760fad9ae838f3b5de7535d4e1ac0d27a330786c95744363ef6751ef385","oracle":{"probs":[0.2717862129211426,0.7282137274742126],"logits_raw":[-1.2322254180908203,1.1391263008117676],"argmax":1,"top2_margin":0.45642751455307007,"near_tie":false},"graph":{"p":[0.2727051079273224,0.7272948622703552],"max_abs_dp":0.0009188950061798096,"argmax_equal":true}}]},{"id":"tv4x_qnli_14","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":254,"_meta":{"row":1945,"text_sha256":"371cbdf5201d1559b62f91d2bdab6ad2d2104d7ac5f4d3db87bce3b9f4e964a6","row_sha256":"04dd278821eadceac2c61ff4f7488c30b9e4c18a2eb4b007a1f485da9ffaf9c2","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/1945","group_id":"qnli/validation/1945","variant":"clean"}},"state_tokens":38,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":69,"state_len":38,"decide":68,"opts":[64,67],"row_ids_sha256":"065e80947ab146ced9aa1d88cf58995d7a30f50b7d669b0e108a4b8d4004d767","oracle":{"probs":[0.08353903144598007,0.9164609313011169],"logits_raw":[-3.0803308486938477,2.6826534271240234],"argmax":1,"top2_margin":0.8329219222068787,"near_tie":false},"graph":{"p":[0.08312524855136871,0.9168747663497925],"max_abs_dp":0.0004138350486755371,"argmax_equal":true}}]},{"id":"tv4x_qnli_15","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":255,"_meta":{"row":1749,"text_sha256":"ab21fd233b1d5a05e03dc9990df89cbceb3f1032a41c991231a917d6e602ba38","row_sha256":"056b7d64e157d80325e7e80c24a55671bf9d7b6022e3eb26087a49d991f39548","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/1749","group_id":"qnli/validation/1749","variant":"clean"}},"state_tokens":58,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":98,"state_len":58,"decide":97,"opts":[93,96],"row_ids_sha256":"ed7789a75717cc4c0c92006aaeded8727cf1a978dcc4e44d3b3c50f3ebb0f032","oracle":{"probs":[0.07898931950330734,0.9210107326507568],"logits_raw":[-3.0572872161865234,2.852354049682617],"argmax":1,"top2_margin":0.8420214056968689,"near_tie":false},"graph":{"p":[0.07837338000535965,0.921626627445221],"max_abs_dp":0.0006159394979476929,"argmax_equal":true}}]},{"id":"tv4x_qnli_16","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":256,"_meta":{"row":5119,"text_sha256":"da65511d9914134b1b614c8881820ae01d37cccc093bc155540bddc0a5b257d5","row_sha256":"05abda4d70d457b761fea0ff9c7fc6a5ccfa8d1dac548841905ccedec7e1e283","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/5119","group_id":"qnli/validation/5119","variant":"clean"}},"state_tokens":37,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":71,"state_len":37,"decide":70,"opts":[66,69],"row_ids_sha256":"8e1aa8827478029be24fec722b8777e0874734ec939bdd07f1aa498c5613dd2d","oracle":{"probs":[0.26064714789390564,0.7393527626991272],"logits_raw":[-0.971959114074707,1.536606788635254],"argmax":1,"top2_margin":0.47870561480522156,"near_tie":false},"graph":{"p":[0.2603912353515625,0.7396087050437927],"max_abs_dp":0.00025594234466552734,"argmax_equal":true}}]},{"id":"tv4x_qnli_17","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":257,"_meta":{"row":4476,"text_sha256":"7228a5f445d5d35b480913bdd3c648476a6e5743ba2d8ce524c8c864316ca8c2","row_sha256":"06010ded8e680dae0e0b8eec96d4a90a90d3f7ea17897dd4468d683b49f94dc1","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/4476","group_id":"qnli/validation/4476","variant":"clean"}},"state_tokens":51,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":80,"state_len":51,"decide":79,"opts":[75,78],"row_ids_sha256":"f192c0e0af36e0a8533d33057cfa81d3f6f43f9b1f0a7f03d3fdcaf6b7317bdd","oracle":{"probs":[0.016129255294799805,0.9838707447052002],"logits_raw":[-4.676390647888184,5.214543342590332],"argmax":1,"top2_margin":0.9677414894104004,"near_tie":false},"graph":{"p":[0.016270529478788376,0.9837294220924377],"max_abs_dp":0.00014132261276245117,"argmax_equal":true}}]},{"id":"tv4x_qnli_18","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":258,"_meta":{"row":2679,"text_sha256":"91255f5369751974f74ec78c071ea2fb8bfda176031828a6e6af872db6f95ac1","row_sha256":"060ce662495649b49fc2ff4109d4713187bba4cd8685a357c77a0234f4d1e26a","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/2679","group_id":"qnli/validation/2679","variant":"clean"}},"state_tokens":14,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":44,"state_len":14,"decide":43,"opts":[39,42],"row_ids_sha256":"3664fd9964efb49eb1b0e104bbd65604ef0b284659442981f907e228fe80b375","oracle":{"probs":[0.9470260143280029,0.052973948419094086],"logits_raw":[0.556502640247345,-6.381405830383301],"argmax":0,"top2_margin":0.8940520882606506,"near_tie":false},"graph":{"p":[0.9466527104377747,0.05334723740816116],"max_abs_dp":0.0003733038902282715,"argmax_equal":true}}]},{"id":"tv4x_qnli_19","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":259,"_meta":{"row":4070,"text_sha256":"5e1c285ed6a50a364098eeca9e5780115daa7cfb5bc9a9d2f51f006bd6a80033","row_sha256":"0639e94bbf5f42ba589154f51cb10fdf3e39f6abea72fff9db4d59d28eb92533","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/4070","group_id":"qnli/validation/4070","variant":"clean"}},"state_tokens":55,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":85,"state_len":55,"decide":84,"opts":[80,83],"row_ids_sha256":"cc9e736b69ed6e0f381d91cb535d66c84a598b6ca57095547d2a38df19be2b80","oracle":{"probs":[0.19602124392986298,0.8039787411689758],"logits_raw":[-2.7681312561035156,0.627646803855896],"argmax":1,"top2_margin":0.6079574823379517,"near_tie":false},"graph":{"p":[0.19581837952136993,0.8041815757751465],"max_abs_dp":0.000202864408493042,"argmax_equal":true}}]},{"id":"tv4x_paws_00","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":320,"_meta":{"row":856,"text_sha256":"b39024f1673d530e60fd0d97fa23dd8dfd7450e969715e17a362906e4e4ae287","row_sha256":"00807b6c80dc3f42f08d9dea69d9f943b048b6c5da3d8136157d9872c54d18e2","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/856","group_id":"paws/test/856","variant":"clean"}},"state_tokens":25,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":81,"state_len":25,"decide":80,"opts":[68,79],"row_ids_sha256":"a1ab5ca2fae02d31ffef5e2eeae709fb9ab7d748b2691aed8c4664ea1672c225","oracle":{"probs":[0.3804095685482025,0.6195904612541199],"logits_raw":[-2.2812862396240234,-1.1075903177261353],"argmax":1,"top2_margin":0.23918089270591736,"near_tie":false},"graph":{"p":[0.37558645009994507,0.6244135499000549],"max_abs_dp":0.004823118448257446,"argmax_equal":true}}]},{"id":"tv4x_paws_01","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":321,"_meta":{"row":2686,"text_sha256":"cdbfeff1f36776ebeccc77fca491ca4a854315378393b5883e67c710d8e1aae6","row_sha256":"01c37641efdb2bc558fcf5c870c7a77c26dd5ece7458d4a70da998b9a6ce600a","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/2686","group_id":"paws/test/2686","variant":"clean"}},"state_tokens":42,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":113,"state_len":42,"decide":112,"opts":[100,111],"row_ids_sha256":"c57b71fbd68616769466cd43ba1d2e78d00d3f0003d32947671c971051e04e3d","oracle":{"probs":[0.49646350741386414,0.5035365223884583],"logits_raw":[-1.24235200881958,-1.2083154916763306],"argmax":1,"top2_margin":0.007073014974594116,"near_tie":true},"graph":{"p":[0.4983747899532318,0.5016251802444458],"max_abs_dp":0.0019113421440124512,"argmax_equal":true}}]},{"id":"tv4x_paws_02","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":322,"_meta":{"row":4712,"text_sha256":"fc460bd5c072efdbf42379535d10451ba8e5babae251aab671dfe6d4d963f6e9","row_sha256":"0253c7c4a41cf8d64770b7e74a3fda4c265f663de0acb195275af4a9dca07196","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/4712","group_id":"paws/test/4712","variant":"clean"}},"state_tokens":33,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":87,"state_len":33,"decide":86,"opts":[74,85],"row_ids_sha256":"33666d4d605f1bd1f9f945b4e748f1b1b9faa2498501dfc3d0fa272f81d49c0e","oracle":{"probs":[0.9737703800201416,0.026229625567793846],"logits_raw":[2.386023759841919,-6.310128688812256],"argmax":0,"top2_margin":0.9475407600402832,"near_tie":false},"graph":{"p":[0.9732199311256409,0.02678000181913376],"max_abs_dp":0.0005504488945007324,"argmax_equal":true}}]},{"id":"tv4x_paws_03","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":323,"_meta":{"row":3056,"text_sha256":"25a8f4bce60585921fc7bb50e0660c0849b8a20c10fcd9f7197460986be184ef","row_sha256":"03554f9384a8812787f16fc356bff6adf4002a872cdc69f10ce4a97928ce4bd9","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/3056","group_id":"paws/test/3056","variant":"clean"}},"state_tokens":38,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":107,"state_len":38,"decide":106,"opts":[94,105],"row_ids_sha256":"b2eac19b59983632724a1e6dfb27166c4fc8237e2cfd337feceff1a8a53cdbc2","oracle":{"probs":[0.02570674940943718,0.9742932319641113],"logits_raw":[-4.70389461517334,4.041997909545898],"argmax":1,"top2_margin":0.9485864639282227,"near_tie":false},"graph":{"p":[0.025427615270018578,0.9745723605155945],"max_abs_dp":0.000279134139418602,"argmax_equal":true}}]},{"id":"tv4x_paws_04","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":324,"_meta":{"row":3990,"text_sha256":"efb066a292e49b3856fdb6ad9386a4002ab5685940c32dd10c116cad3fa166a8","row_sha256":"0396ad8f63e4a76587672c5b296a89c9cd22a9665a3e7dccefc3016d33b4f1ce","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/3990","group_id":"paws/test/3990","variant":"clean"}},"state_tokens":32,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":86,"state_len":32,"decide":85,"opts":[73,84],"row_ids_sha256":"5656b42e139fead4a19708153c0c825166e16edbde16f9ae85dc0aa60c873242","oracle":{"probs":[0.06338899582624435,0.9366110563278198],"logits_raw":[-4.772831916809082,1.7066075801849365],"argmax":1,"top2_margin":0.8732220530509949,"near_tie":false},"graph":{"p":[0.06280706077814102,0.937192976474762],"max_abs_dp":0.0005819350481033325,"argmax_equal":true}}]},{"id":"tv4x_paws_05","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":325,"_meta":{"row":529,"text_sha256":"c0a8c191543c31a01d1d7206f6177150e46e5078671c906d2c43a69e350be12f","row_sha256":"03a964f3ae534adfacfb83f756e93404c932ebc053ecbfd3e66e8a203775ecee","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/529","group_id":"paws/test/529","variant":"clean"}},"state_tokens":31,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":96,"state_len":31,"decide":95,"opts":[83,94],"row_ids_sha256":"7ecdf4903d3f46e3372f7bd86d15f625defd193bc3832d4b2058ef9b858345b8","oracle":{"probs":[0.19257213175296783,0.8074278831481934],"logits_raw":[-3.8427987098693848,-0.39400777220726013],"argmax":1,"top2_margin":0.6148557662963867,"near_tie":false},"graph":{"p":[0.19246318936347961,0.8075368404388428],"max_abs_dp":0.00010895729064941406,"argmax_equal":true}}]},{"id":"tv4x_paws_06","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":326,"_meta":{"row":116,"text_sha256":"e663c5b92e7a5cc9a1b2c6bcb939063a0295d062ed9c6d2933b5b513fdc5f3aa","row_sha256":"0417933bf84a525da6959a3548e9c533ce9d8655b267c55db8f6e423239fafa1","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/116","group_id":"paws/test/116","variant":"clean"}},"state_tokens":32,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":97,"state_len":32,"decide":96,"opts":[84,95],"row_ids_sha256":"77b23ff58b3cf7035fb6795fa4ee9e856edcf505bfc9c9b7c6c2f18fcabfe3fe","oracle":{"probs":[0.27180027961730957,0.7281997799873352],"logits_raw":[-2.291045904159546,0.0801355242729187],"argmax":1,"top2_margin":0.45639950037002563,"near_tie":false},"graph":{"p":[0.27079418301582336,0.729205846786499],"max_abs_dp":0.001006096601486206,"argmax_equal":true}}]},{"id":"tv4x_paws_07","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":327,"_meta":{"row":2414,"text_sha256":"5a17b5264afce263aeb32953a5d710f1cd45bf535d620989515c265f560b2181","row_sha256":"0455513e097769adc3aaefd350d551118caaecac19e6c47183293e3809388e4c","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/2414","group_id":"paws/test/2414","variant":"clean"}},"state_tokens":21,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":72,"state_len":21,"decide":71,"opts":[59,70],"row_ids_sha256":"fdefb711963f78581e4707d8ca882e714a3d81f1a42364d19441f658b9629b84","oracle":{"probs":[0.037242334336042404,0.9627577066421509],"logits_raw":[-4.303839683532715,3.5214898586273193],"argmax":1,"top2_margin":0.925515353679657,"near_tie":false},"graph":{"p":[0.03708898648619652,0.9629110097885132],"max_abs_dp":0.00015334784984588623,"argmax_equal":true}}]},{"id":"tv4x_paws_08","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":328,"_meta":{"row":2795,"text_sha256":"82dd4cf1090e4e6896653803a3291e25fe70e8edf60f86bcec183dc4eba36bbd","row_sha256":"04dc1933b0c811617487064c704c6d3487e7b9822fbe80fc85c4b9fed7f06ed7","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/2795","group_id":"paws/test/2795","variant":"clean"}},"state_tokens":21,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":73,"state_len":21,"decide":72,"opts":[60,71],"row_ids_sha256":"53f84d9df82bca6abb5bdaba9da65febe014e392cf39645bce3c8379545c0bca","oracle":{"probs":[0.04648711532354355,0.9535129070281982],"logits_raw":[-4.424586296081543,2.8440372943878174],"argmax":1,"top2_margin":0.9070258140563965,"near_tie":false},"graph":{"p":[0.046223487704992294,0.953776478767395],"max_abs_dp":0.00026362761855125427,"argmax_equal":true}}]},{"id":"tv4x_paws_09","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":329,"_meta":{"row":4273,"text_sha256":"0c10330e40010bb5442145d4e3b0042b75b8fcd7aa951e492ea08ddf187fd541","row_sha256":"0538754a83f520428a85681e700f292635f2560767b6f6f802ea2e86d87427e3","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/4273","group_id":"paws/test/4273","variant":"clean"}},"state_tokens":54,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":127,"state_len":54,"decide":126,"opts":[114,125],"row_ids_sha256":"24f70250a3e05a4beeb18a4b9e8cbebcff9c5728eb9ce4fefca7a87204d26060","oracle":{"probs":[0.16447566449642181,0.835524320602417],"logits_raw":[-3.9677836894989014,-0.05723816156387329],"argmax":1,"top2_margin":0.671048641204834,"near_tie":false},"graph":{"p":[0.16677325963974,0.8332267999649048],"max_abs_dp":0.0022975951433181763,"argmax_equal":true}}]},{"id":"tv4x_paws_10","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":330,"_meta":{"row":6236,"text_sha256":"4ae695c0138e1d65ee6f33aedf344c6527643060c363334abb66f32a95a60815","row_sha256":"0555f5b1b45405dbbd5811279db88ba1850d474ed716a30816a100ffe8934ac1","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/6236","group_id":"paws/test/6236","variant":"clean"}},"state_tokens":27,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":86,"state_len":27,"decide":85,"opts":[73,84],"row_ids_sha256":"e3e601e12e594b2c1733659ba591f2fbc0924c6de2bf3616ac74f38fd493386f","oracle":{"probs":[0.8644732236862183,0.13552677631378174],"logits_raw":[0.6127046942710876,-3.845588445663452],"argmax":0,"top2_margin":0.7289464473724365,"near_tie":false},"graph":{"p":[0.8627452850341797,0.13725470006465912],"max_abs_dp":0.0017279386520385742,"argmax_equal":true}}]},{"id":"tv4x_paws_11","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":331,"_meta":{"row":1125,"text_sha256":"e075a882bf75d143e262395005d4c9f23dd624839239c751861f485f5e914734","row_sha256":"05692822d545561130a006ccb671d0266b60879047c2459fa8308d90320ae5e3","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/1125","group_id":"paws/test/1125","variant":"clean"}},"state_tokens":32,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":99,"state_len":32,"decide":98,"opts":[86,97],"row_ids_sha256":"a78883cf156deb2066bd05f58c846ca71e61a1b372f0bf879cb35484eb9a55e1","oracle":{"probs":[0.05510639399290085,0.9448936581611633],"logits_raw":[-4.226660251617432,2.6108686923980713],"argmax":1,"top2_margin":0.8897872567176819,"near_tie":false},"graph":{"p":[0.055087074637413025,0.9449129104614258],"max_abs_dp":1.9319355487823486e-05,"argmax_equal":true}}]},{"id":"tv4x_paws_12","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":332,"_meta":{"row":3102,"text_sha256":"ba89f073ecf1fdcdbb239b309dc4b29f23f1df50db37a615d9947e9d4b749c59","row_sha256":"0601c0d70fe89e8132cc5430dbcf48c4ed76173df68d9e67c2e89318fbfcbaa9","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/3102","group_id":"paws/test/3102","variant":"clean"}},"state_tokens":30,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":93,"state_len":30,"decide":92,"opts":[80,91],"row_ids_sha256":"bdbb609139881f3e13d51be4cf5ce7a34e190befb69bee85434a5e95a4da2a31","oracle":{"probs":[0.23798313736915588,0.7620168924331665],"logits_raw":[-6.556375026702881,-3.75628924369812],"argmax":1,"top2_margin":0.524033784866333,"near_tie":false},"graph":{"p":[0.2405313402414322,0.759468674659729],"max_abs_dp":0.0025482177734375,"argmax_equal":true}}]},{"id":"tv4x_paws_13","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":333,"_meta":{"row":4325,"text_sha256":"b173fc9a6d018360204b3b606b878cedac8f25f2f97ec1465ccece0d58c7d695","row_sha256":"060adb7606bfe00f2da3af0b2abc1d9a594120a92cafadf8af5e9d433172999f","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/4325","group_id":"paws/test/4325","variant":"clean"}},"state_tokens":43,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":107,"state_len":43,"decide":106,"opts":[94,105],"row_ids_sha256":"808c3e38b8b2e0a9f51e462d4711b8de7284bda7769464dfd179ff1d4f3663da","oracle":{"probs":[0.09571240842342377,0.9042876362800598],"logits_raw":[-3.2117066383361816,2.1917996406555176],"argmax":1,"top2_margin":0.8085752129554749,"near_tie":false},"graph":{"p":[0.09497963637113571,0.9050202965736389],"max_abs_dp":0.0007327720522880554,"argmax_equal":true}}]},{"id":"tv4x_paws_14","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":334,"_meta":{"row":2474,"text_sha256":"287b64cf4368977a32078e00b9bb44428641276bf8831998e7e86c28197669d4","row_sha256":"068cbf67767ac9c3764014e96b7ea646e8582833120700f3166b352a64ecdb33","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/2474","group_id":"paws/test/2474","variant":"clean"}},"state_tokens":26,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":77,"state_len":26,"decide":76,"opts":[64,75],"row_ids_sha256":"fe7becbb16bed42bed34af28ee182d8b05720cea59639ed33765eb2fd62fbedc","oracle":{"probs":[0.3754120171070099,0.6245880126953125],"logits_raw":[-2.9523839950561523,-1.7275406122207642],"argmax":1,"top2_margin":0.2491759955883026,"near_tie":false},"graph":{"p":[0.3774430453777313,0.6225569844245911],"max_abs_dp":0.0020310282707214355,"argmax_equal":true}}]},{"id":"tv4x_paws_15","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":335,"_meta":{"row":1182,"text_sha256":"7d25b7f25bb15a92566f269a9c0f8bea5b36a4573027eabbd305eecc4868609c","row_sha256":"06ffc8dedb991a70b4010f2dec9af450598bf560ff1b3543204df8e9808ab72c","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/1182","group_id":"paws/test/1182","variant":"clean"}},"state_tokens":26,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":74,"state_len":26,"decide":73,"opts":[61,72],"row_ids_sha256":"fb9ce86a903cc1cfb7b0ef56e5e4426207bbc6e3c769ae3afd4838ac5752dd4f","oracle":{"probs":[0.16473692655563354,0.8352630734443665],"logits_raw":[-4.325653076171875,-0.41967907547950745],"argmax":1,"top2_margin":0.6705261468887329,"near_tie":false},"graph":{"p":[0.1665254533290863,0.8334745764732361],"max_abs_dp":0.0017885267734527588,"argmax_equal":true}}]},{"id":"tv4x_paws_16","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":336,"_meta":{"row":504,"text_sha256":"2e39c8cfc1bdc208756dc06f91477606f02eb689ed79e78bd27567fa00bcddfe","row_sha256":"0740e6c8ba70a47cd9f0000926f5ce2fcb2609119181c3f2ba3feb6f8326634f","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/504","group_id":"paws/test/504","variant":"clean"}},"state_tokens":23,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":68,"state_len":23,"decide":67,"opts":[55,66],"row_ids_sha256":"e1854ea9f2a2c065f68432d9249a363e98f00778d7de17bcf695a3bc03f34fb9","oracle":{"probs":[0.2923157513141632,0.7076842188835144],"logits_raw":[-6.403163909912109,-4.275822639465332],"argmax":1,"top2_margin":0.4153684675693512,"near_tie":false},"graph":{"p":[0.29218044877052307,0.7078194618225098],"max_abs_dp":0.00013530254364013672,"argmax_equal":true}}]},{"id":"tv4x_paws_17","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":337,"_meta":{"row":5828,"text_sha256":"655f415c50be03e8af2444637e0f8301b59dd81bbae377ca3cc73f28cbe07113","row_sha256":"0797b909b6f352589b88f6698648caafc779462616bbc174a27232600691c57d","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/5828","group_id":"paws/test/5828","variant":"clean"}},"state_tokens":22,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":78,"state_len":22,"decide":77,"opts":[65,76],"row_ids_sha256":"90bee2aa990262c7887b522cc80ae18934a8a2e81c46c9c2eb2ce410c44f5ed0","oracle":{"probs":[0.09704651683568954,0.9029535055160522],"logits_raw":[-3.7071964740753174,1.6594517230987549],"argmax":1,"top2_margin":0.8059070110321045,"near_tie":false},"graph":{"p":[0.09655562788248062,0.9034443497657776],"max_abs_dp":0.0004908889532089233,"argmax_equal":true}}]},{"id":"tv4x_paws_18","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":338,"_meta":{"row":6633,"text_sha256":"2bfe00279009ab77bf1263ca9e7a96fe542673b040ec466fd96bcfb6a15a46f9","row_sha256":"07edfe0d91a3190629f417abff85b410c7785b83cfe54a509fe650bdead2d7a4","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/6633","group_id":"paws/test/6633","variant":"clean"}},"state_tokens":27,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":83,"state_len":27,"decide":82,"opts":[70,81],"row_ids_sha256":"de8cfeb2c1f6fc38873039b45b786dd6aa8da9bc1058c1bce25ff8c2f83c946d","oracle":{"probs":[0.9904965162277222,0.009503461420536041],"logits_raw":[4.861339569091797,-6.318492889404297],"argmax":0,"top2_margin":0.9809930324554443,"near_tie":false},"graph":{"p":[0.9904175400733948,0.00958241056650877],"max_abs_dp":7.897615432739258e-05,"argmax_equal":true}}]},{"id":"tv4x_paws_19","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":339,"_meta":{"row":6102,"text_sha256":"30cf30c9dcfb4e1863f825ceda91572b0b98548220a3bd0be3ecaa52c096961f","row_sha256":"088b142ff5d38a084ee446facb1d90b967dedd418715c321fe367ebc78938ae1","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/6102","group_id":"paws/test/6102","variant":"clean"}},"state_tokens":21,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":75,"state_len":21,"decide":74,"opts":[62,73],"row_ids_sha256":"960e29e6891a53109014d3bfa159c913635e49a4c25520e24b9ed992d2a2c0c9","oracle":{"probs":[0.09070250391960144,0.909297525882721],"logits_raw":[-4.521389961242676,1.024765133857727],"argmax":1,"top2_margin":0.8185950517654419,"near_tie":false},"graph":{"p":[0.0905931144952774,0.909406840801239],"max_abs_dp":0.00010938942432403564,"argmax_equal":true}}]},{"id":"tv4x_sciq_00","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":400,"_meta":{"row":936,"text_sha256":"09202918cfc420e946051466763f6037d2b455dbf90d9b8fb9d79040e1047556","row_sha256":"00dd55f0b43115d6d413815d432e3b5f7dd9e15a3d341973d40cfc7c3b5defe4","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/936","group_id":"sciq/test/936","variant":"clean"}},"state_tokens":115,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":149,"state_len":115,"decide":148,"opts":[128,134,140,147],"row_ids_sha256":"4a73d937e08036d8d2c8466ab286a75640edec06134a8800c25c8d066625daea","oracle":{"probs":[0.0014410375151783228,0.985696017742157,0.008780783973634243,0.004082161001861095],"logits_raw":[-10.81425952911377,4.892398357391357,-6.466039657592773,-8.308927536010742],"argmax":1,"top2_margin":0.9769152402877808,"near_tie":false},"graph":{"p":[0.0014627029886469245,0.9855884909629822,0.00876365602016449,0.004185116849839687],"max_abs_dp":0.00010752677917480469,"argmax_equal":true}}]},{"id":"tv4x_sciq_01","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":401,"_meta":{"row":925,"text_sha256":"3ca591694cd9b7c4e5d45687af0e57894ee1793a91164d5985918a7a91ef5c02","row_sha256":"013ea20efdab1c499fa2c6dfa4fdf58df6cdc0e11ffbbbd41f9cdfefbed6f00c","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/925","group_id":"sciq/test/925","variant":"clean"}},"state_tokens":19,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":57,"state_len":19,"decide":56,"opts":[34,41,48,55],"row_ids_sha256":"ede8bf3ce598589c64f7de512d1bae6946cf0a423e339548aadc907abfe06c14","oracle":{"probs":[0.08146927505731583,0.07226943969726562,0.06310390681028366,0.7831574082374573],"logits_raw":[-3.9264492988586426,-4.214753150939941,-4.541058540344238,1.5187013149261475],"argmax":3,"top2_margin":0.7016881108283997,"near_tie":false},"graph":{"p":[0.08147134631872177,0.07281661033630371,0.06381434202194214,0.7818976640701294],"max_abs_dp":0.0012597441673278809,"argmax_equal":true}}]},{"id":"tv4x_sciq_02","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":402,"_meta":{"row":269,"text_sha256":"7f83e7a09b5eb8a44209e77843fa327982ef4d485e7cb8ae2a3dde88eb18439e","row_sha256":"01db09ca82e091471fdca3a997ad6c4a6abc4979cffd033573cc6eaa9522e1cb","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/269","group_id":"sciq/test/269","variant":"clean"}},"state_tokens":69,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":102,"state_len":69,"decide":101,"opts":[82,88,94,100],"row_ids_sha256":"bf30e49ca7972565e42c6d37b9eeff94a2e5dbcfec070d20bdcf17f312d60c0c","oracle":{"probs":[0.9914463758468628,0.004417371470481157,0.0014107113238424063,0.0027255492750555277],"logits_raw":[4.9497270584106445,-8.075713157653809,-10.822101593017578,-9.237534523010254],"argmax":0,"top2_margin":0.9870290160179138,"near_tie":false},"graph":{"p":[0.9914290308952332,0.0043981559574604034,0.0014176726108416915,0.00275512901134789],"max_abs_dp":2.9579736292362213e-05,"argmax_equal":true}}]},{"id":"tv4x_sciq_03","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":403,"_meta":{"row":258,"text_sha256":"9a0acb41813d28adadd7e5f2bc3f5f7d0019a90c6bdbed335de9f56425fdd42f","row_sha256":"0271ef05064f4cf27418547ced77b40684b58177241e24f80d14c36a269019a2","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/258","group_id":"sciq/test/258","variant":"clean"}},"state_tokens":67,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":97,"state_len":67,"decide":96,"opts":[79,84,89,95],"row_ids_sha256":"fff19876eb59c8da5716e064e47645c6450cefee2eca96d6ddc6db4cbe94b720","oracle":{"probs":[0.0017702925251796842,0.00338764488697052,0.025027349591255188,0.969814658164978],"logits_raw":[-10.10202407836914,-8.540520668029785,-3.7288196086883545,5.070431709289551],"argmax":3,"top2_margin":0.944787323474884,"near_tie":false},"graph":{"p":[0.0017902436666190624,0.0034152998123317957,0.02532445639371872,0.9694700241088867],"max_abs_dp":0.0003446340560913086,"argmax_equal":true}}]},{"id":"tv4x_sciq_04","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":404,"_meta":{"row":46,"text_sha256":"406cd4f25d6709cacecb1f41876271eaf073801ecce8249da6e328a8abaffebb","row_sha256":"02874e7f33f2dadb8cf77157ad30509158c7a3e5bbfb249dcd369fb8b4a947c0","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/46","group_id":"sciq/test/46","variant":"clean"}},"state_tokens":81,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":117,"state_len":81,"decide":116,"opts":[94,101,108,115],"row_ids_sha256":"45ad8c06e3277c36e9fdbafce99c850b56e9688eaca00497dd8c6a80a6b5572b","oracle":{"probs":[0.003914231900125742,0.8329983353614807,0.15500779449939728,0.008079595863819122],"logits_raw":[-10.321102142333984,2.5763180255889893,-1.469590187072754,-8.577383041381836],"argmax":1,"top2_margin":0.6779905557632446,"near_tie":false},"graph":{"p":[0.003899239469319582,0.8363878726959229,0.1516347974538803,0.008078102022409439],"max_abs_dp":0.0033895373344421387,"argmax_equal":true}}]},{"id":"tv4x_sciq_05","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":405,"_meta":{"row":670,"text_sha256":"d37ce9e469bb0c63715ef88cbbb7be115861e039ac45b1f9c9d3119c2e93ab5e","row_sha256":"0318d75b01d26524998217711113c87ac19d228d04d99c31afe8234882e94a0f","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/670","group_id":"sciq/test/670","variant":"clean"}},"state_tokens":39,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":70,"state_len":39,"decide":69,"opts":[51,56,61,68],"row_ids_sha256":"fb5e9c27a8f4a19a99030c6744e4e1893d25bfa42249495cd6f2b421ea6ef513","oracle":{"probs":[0.9623622894287109,0.015047289431095123,0.008398368023335934,0.014192021451890469],"logits_raw":[3.3993630409240723,-6.605457305908203,-8.008570671081543,-6.746253967285156],"argmax":0,"top2_margin":0.947314977645874,"near_tie":false},"graph":{"p":[0.9621914029121399,0.015196726657450199,0.00841755885630846,0.014194278977811337],"max_abs_dp":0.00017088651657104492,"argmax_equal":true}}]},{"id":"tv4x_sciq_06","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":406,"_meta":{"row":731,"text_sha256":"b705e7cbdf2e88ef234f0b163cfbcde607edf28974e4cc5bbb98544c6458e46f","row_sha256":"034547b8fa6238bee490397732c2ca0f662ccd5594e22b38eda01360c9a0a26a","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/731","group_id":"sciq/test/731","variant":"clean"}},"state_tokens":155,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":184,"state_len":155,"decide":183,"opts":[167,172,177,182],"row_ids_sha256":"b9e39b8957581761f50d67285e79abce3c30534780c509fb6463a4d2da029e14","oracle":{"probs":[0.001418677973560989,0.988223671913147,0.005501842591911554,0.0048558684065938],"logits_raw":[-11.108559608459473,4.641885757446289,-7.847500801086426,-8.148004531860352],"argmax":1,"top2_margin":0.9827218055725098,"near_tie":false},"graph":{"p":[0.0014358570333570242,0.9881517291069031,0.00551580497995019,0.004896589554846287],"max_abs_dp":7.194280624389648e-05,"argmax_equal":true}}]},{"id":"tv4x_sciq_07","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":407,"_meta":{"row":955,"text_sha256":"b3994ef4d3e0fd3fbe40ada60b67c11b6d9689a06c5679c6ed227899a93fe2e9","row_sha256":"035d44aeb1e246ed7b45f8091a77dc4cbd122786a6365d556fe8dd25d1d56633","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/955","group_id":"sciq/test/955","variant":"clean"}},"state_tokens":53,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":82,"state_len":53,"decide":81,"opts":[65,70,75,80],"row_ids_sha256":"afb07faa86f952e4da34008240a2f2680fc2c9760636687cdb8bb8cec9a60528","oracle":{"probs":[0.0009928246727213264,0.9895554780960083,0.0026075614150613546,0.006844221148639917],"logits_raw":[-10.743444442749023,5.86902379989624,-8.420123100280762,-6.098311424255371],"argmax":1,"top2_margin":0.9827112555503845,"near_tie":false},"graph":{"p":[0.0010039448970928788,0.9895407557487488,0.0026167670730501413,0.006838530767709017],"max_abs_dp":1.4722347259521484e-05,"argmax_equal":true}}]},{"id":"tv4x_sciq_08","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":408,"_meta":{"row":490,"text_sha256":"03ef14027d83f2bbb741b29c375fbaddfdb53ba7221da59eda4d7907699b2e1c","row_sha256":"039ab8cd042a271800e6c44a7f9f3f6e1c3be9bea2568c5024bed37cecc4ad22","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/490","group_id":"sciq/test/490","variant":"clean"}},"state_tokens":129,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":162,"state_len":129,"decide":161,"opts":[142,148,154,160],"row_ids_sha256":"e79aa9be40985f98906be06dbdc3b0c9c4f523a7a4605998782523bafa7fdbcf","oracle":{"probs":[0.9348140358924866,0.03997715562582016,0.009875726886093616,0.015333177521824837],"logits_raw":[4.160177230834961,-3.4237871170043945,-6.787994384765625,-5.72947883605957],"argmax":0,"top2_margin":0.8948369026184082,"near_tie":false},"graph":{"p":[0.9347981214523315,0.03997315466403961,0.009890502318739891,0.015338211320340633],"max_abs_dp":1.5914440155029297e-05,"argmax_equal":true}}]},{"id":"tv4x_sciq_09","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":409,"_meta":{"row":633,"text_sha256":"cf1e953f57b50c302376674da8534dc4ba1e6bf640905c7ddb206b9f1a988a35","row_sha256":"03a0482f6be75a66d516a3de837ca4ffc6f9346cac36bc8de3951eb67f6dcbd9","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/633","group_id":"sciq/test/633","variant":"clean"}},"state_tokens":115,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":144,"state_len":115,"decide":143,"opts":[127,132,137,142],"row_ids_sha256":"637f7c3f287ada1565de98e166d2edd0ca42c58dda6dd79fdf8014cca0f1a1fe","oracle":{"probs":[0.0036738442722707987,0.005542146507650614,0.004636489320546389,0.9861475825309753],"logits_raw":[-8.371583938598633,-7.382352352142334,-7.811650276184082,5.084413051605225],"argmax":3,"top2_margin":0.98060542345047,"near_tie":false},"graph":{"p":[0.003758783219382167,0.0055820331908762455,0.0047773076221346855,0.9858819246292114],"max_abs_dp":0.000265657901763916,"argmax_equal":true}}]},{"id":"tv4x_sciq_10","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":410,"_meta":{"row":889,"text_sha256":"5f0f535f37629cada78055161524ded18f6d0fdba7163ccf28665d36bcb3c7fe","row_sha256":"03cc76d7b491ab3112d651f3a926efd30577affd480652d2a9e17b243dec867c","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/889","group_id":"sciq/test/889","variant":"clean"}},"state_tokens":48,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":85,"state_len":48,"decide":84,"opts":[63,70,76,83],"row_ids_sha256":"b96ff052b7820fae2de1aaee163a3bc1250276be8bfe2bdf3e144647beeebbca","oracle":{"probs":[0.017873836681246758,0.005290279630571604,0.0003751253825612366,0.9764606952667236],"logits_raw":[-3.804596424102783,-6.733882904052734,-13.101171493530273,5.821038246154785],"argmax":3,"top2_margin":0.9585868716239929,"near_tie":false},"graph":{"p":[0.018043771386146545,0.005330669693648815,0.0003817045653704554,0.9762439131736755],"max_abs_dp":0.0002167820930480957,"argmax_equal":true}}]},{"id":"tv4x_sciq_11","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":411,"_meta":{"row":291,"text_sha256":"5395c80c7508db20b33d4d1a9f78cc0acbba008ed35cbc07343ca98a1b5e9fba","row_sha256":"03e1c68453f4cc08d4794024d3a5cf1e411f357cb693b539383f020c45149fc1","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/291","group_id":"sciq/test/291","variant":"clean"}},"state_tokens":330,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":366,"state_len":330,"decide":365,"opts":[344,351,358,364],"row_ids_sha256":"1bd791bcd6ea00938fcbdd82b1ba0d9295193ca4e39a737327496170850c2f1a","oracle":{"probs":[0.010405686683952808,0.004287616349756718,0.9843416810035706,0.0009649478597566485],"logits_raw":[-6.590062618255615,-8.723318099975586,4.3565521240234375,-12.31173038482666],"argmax":2,"top2_margin":0.9739360213279724,"near_tie":false},"graph":{"p":[0.010503776371479034,0.004333636723458767,0.9841893315315247,0.0009732014150358737],"max_abs_dp":0.00015234947204589844,"argmax_equal":true}}]},{"id":"tv4x_sciq_12","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":412,"_meta":{"row":558,"text_sha256":"11a31672ec8b03161c35a52dd5d804506fa719826358f64986d4429ffc53c0d0","row_sha256":"04439e11e08fb71e5618a97d90d879814f5cdc5e8dc249a630a21a3c8bec4e74","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/558","group_id":"sciq/test/558","variant":"clean"}},"state_tokens":163,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":192,"state_len":163,"decide":191,"opts":[175,180,185,190],"row_ids_sha256":"16056450eab7cdaccca8b442d3100c472c6e6b2881b2a5b548a811e18931c03e","oracle":{"probs":[0.001838281867094338,0.006847818847745657,0.9531726241111755,0.03814134746789932],"logits_raw":[-11.268963813781738,-8.10477066040039,3.7711691856384277,-3.9726955890655518],"argmax":2,"top2_margin":0.9150312542915344,"near_tie":false},"graph":{"p":[0.001874421490356326,0.006920339539647102,0.9527617692947388,0.038443565368652344],"max_abs_dp":0.0004108548164367676,"argmax_equal":true}}]},{"id":"tv4x_sciq_13","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":413,"_meta":{"row":712,"text_sha256":"15b7801931da9a1ca23589aee1a4ad7ad7d2cc2ae19f929a946248eb30517c84","row_sha256":"049cd85994b3f74ed4a086eefd408676a46826bdf9136728dcf1c06db493b008","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/712","group_id":"sciq/test/712","variant":"clean"}},"state_tokens":113,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":147,"state_len":113,"decide":146,"opts":[126,132,138,145],"row_ids_sha256":"67976a639b2576198480c81e49db27965fdf3c69ad9b132621911c535efba52f","oracle":{"probs":[0.9761357307434082,0.009856581687927246,0.006720255129039288,0.00728733791038394],"logits_raw":[4.455778121948242,-6.601134300231934,-7.522682189941406,-7.327763557434082],"argmax":0,"top2_margin":0.966279149055481,"near_tie":false},"graph":{"p":[0.9758961200714111,0.009927949868142605,0.006807286292314529,0.007368593011051416],"max_abs_dp":0.0002396106719970703,"argmax_equal":true}}]},{"id":"tv4x_sciq_14","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":414,"_meta":{"row":651,"text_sha256":"e44331cc4691a83e686fddd0238b1d13f60e6ffd856c7d256cd9a19667f9772a","row_sha256":"0604cb4d9a31f55964a2068967ac26bf90af1f8c9dad1047550013cf2c2ec7c8","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/651","group_id":"sciq/test/651","variant":"clean"}},"state_tokens":91,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":127,"state_len":91,"decide":126,"opts":[105,111,117,125],"row_ids_sha256":"a2cd35e1d889b1bb81453149856bdf62286463db7488616e43d264e5c90e80b2","oracle":{"probs":[0.0032891249284148216,0.0011733061401173472,0.0006244955002330244,0.9949129819869995],"logits_raw":[-8.172441482543945,-10.652587890625,-12.169931411743164,5.57099723815918],"argmax":3,"top2_margin":0.9916238784790039,"near_tie":false},"graph":{"p":[0.003326787380501628,0.0011990399798378348,0.0006359914550557733,0.9948381781578064],"max_abs_dp":7.480382919311523e-05,"argmax_equal":true}}]},{"id":"tv4x_sciq_15","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":415,"_meta":{"row":524,"text_sha256":"5e99a00019fbcd39a4382e01e593dbba1dbd5c883a8d986bbe1c0cfbb08d6646","row_sha256":"0650d359b3b1912cd9d46e148ec5b9a5fc892b52edeaffcd00390d6910b026f8","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/524","group_id":"sciq/test/524","variant":"clean"}},"state_tokens":239,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":270,"state_len":239,"decide":269,"opts":[251,257,263,268],"row_ids_sha256":"96f1ba094c14cd02c57d60512926ea4765944bd5edacf493ce89c22e0f26ef71","oracle":{"probs":[0.0014088174793869257,0.977857232093811,0.012921107932925224,0.007812806405127048],"logits_raw":[-10.648189544677734,5.093664169311523,-5.31611442565918,-6.52659273147583],"argmax":1,"top2_margin":0.9649361371994019,"near_tie":false},"graph":{"p":[0.0014321650378406048,0.9777892231941223,0.012922046706080437,0.007856584154069424],"max_abs_dp":6.80088996887207e-05,"argmax_equal":true}}]},{"id":"tv4x_sciq_16","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":416,"_meta":{"row":512,"text_sha256":"e4a0b75c4e5ecd5e9f29c31646003078448ef767e1d5c44d62b98f5cc1fbc6f6","row_sha256":"073a82faba43b6089d6d78bb990ae8b523fc90830aa80d3121fcde4c0486d880","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/512","group_id":"sciq/test/512","variant":"clean"}},"state_tokens":306,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":335,"state_len":306,"decide":334,"opts":[318,323,328,333],"row_ids_sha256":"0a96d869bc60a5b43e7d28746c797eee60e5d5ba68964ad7f9a72ab1723ecba2","oracle":{"probs":[0.2438400685787201,0.02756941318511963,0.0742371454834938,0.6543534398078918],"logits_raw":[-4.858704566955566,-10.103425979614258,-7.720094680786133,-2.4836084842681885],"argmax":3,"top2_margin":0.41051337122917175,"near_tie":false},"graph":{"p":[0.24599505960941315,0.0277009978890419,0.07408005744218826,0.6522238254547119],"max_abs_dp":0.002154991030693054,"argmax_equal":true}}]},{"id":"tv4x_sciq_17","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":417,"_meta":{"row":246,"text_sha256":"87779c312fe24a99cf39b66c82215a33948d2f96384705dffd8ff7062b58b9c9","row_sha256":"076a1b72202038496a7147eb57a5d3e7c492ada07fd06e79bf6f6074b25ca10f","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/246","group_id":"sciq/test/246","variant":"clean"}},"state_tokens":136,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":169,"state_len":136,"decide":168,"opts":[149,155,161,167],"row_ids_sha256":"981153fabba7c53346838b5d6c9a67879ddfab0d1eb38a78a235d01e4781198e","oracle":{"probs":[0.9768888354301453,0.009465008042752743,0.0071713803336024284,0.006474799942225218],"logits_raw":[3.814058780670166,-7.342244625091553,-8.009931564331055,-8.255783081054688],"argmax":0,"top2_margin":0.9674238562583923,"near_tie":false},"graph":{"p":[0.9766273498535156,0.009568939916789532,0.00724616926163435,0.006557531654834747],"max_abs_dp":0.00026148557662963867,"argmax_equal":true}}]},{"id":"tv4x_sciq_18","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":418,"_meta":{"row":844,"text_sha256":"a890e73029e550d303219f02577f9b6d5c521a3c4fdae7ee9843e487a9957dc9","row_sha256":"080069712d10fb8edd1f56f02ab5421d4d3f6e4530c42440f335904163cf9fdc","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/844","group_id":"sciq/test/844","variant":"clean"}},"state_tokens":34,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":65,"state_len":34,"decide":64,"opts":[46,53,58,63],"row_ids_sha256":"75f37c3d3119568c3b00b9fcb6ee7501179b06db66146cf3788d211775b26048","oracle":{"probs":[0.9780268669128418,0.011148149147629738,0.004114539362490177,0.006710366811603308],"logits_raw":[4.93001651763916,-5.835285186767578,-8.233508110046387,-7.056644439697266],"argmax":0,"top2_margin":0.9668787121772766,"near_tie":false},"graph":{"p":[0.9777951240539551,0.01123824343085289,0.004172022920101881,0.0067946938797831535],"max_abs_dp":0.00023174285888671875,"argmax_equal":true}}]},{"id":"tv4x_sciq_19","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":419,"_meta":{"row":810,"text_sha256":"1a0b83d783e4e41ca3fbc13d015560c96917c1a361cdf7a3b8465840181b9cce","row_sha256":"08154610d80750177a3d6188d374164cf122f86b567cd3721c8d6d025747f98b","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/810","group_id":"sciq/test/810","variant":"clean"}},"state_tokens":42,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":74,"state_len":42,"decide":73,"opts":[55,61,66,72],"row_ids_sha256":"ff8b45fb3ea8d840d0cd155964e8bde408a6041014989d47989d71a0dbad95a2","oracle":{"probs":[0.992010235786438,0.0015547259245067835,0.002144850790500641,0.004290225449949503],"logits_raw":[5.412062644958496,-10.127252578735352,-9.353055953979492,-7.68501615524292],"argmax":0,"top2_margin":0.9877200126647949,"near_tie":false},"graph":{"p":[0.9919373393058777,0.0015711341984570026,0.0021649471018463373,0.004326484631747007],"max_abs_dp":7.289648056030273e-05,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_00","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":588,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0000","pair_id":"v3-legacy-transfer-authorization-0000","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0000","id":"contrastive/authorization/v3-legacy-transfer-authorization-0000/a","text_sha256":"c98a9666364a0d6e1490f365bb02551f8c08384c385be2cc2415203c56933e7f","row_sha256":"dbc61f5880a17d29f23b679f391879417818ec7732c8a1550582e5b340b416cb","group_id":"v3-legacy-transfer-authorization-0000","variant":"clean"}},"state_tokens":65,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":78,"state_len":65,"decide":77,"opts":[73,76],"row_ids_sha256":"b2afeb627363ab5feadffe4c77814698b458d3c8f9da961ccc1b6e836d233281","oracle":{"probs":[0.006451320368796587,0.9935486912727356],"logits_raw":[-7.135417938232422,4.983851432800293],"argmax":1,"top2_margin":0.9870973825454712,"near_tie":false},"graph":{"p":[0.006519875954836607,0.9934800863265991],"max_abs_dp":6.860494613647461e-05,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_01","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":589,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0000","pair_id":"v3-legacy-transfer-authorization-0000","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0000","id":"contrastive/authorization/v3-legacy-transfer-authorization-0000/b","text_sha256":"e0b3e4ca4b46805650635de7381e7384fb45e77fc493303af3c59a1d5a2a351e","row_sha256":"476a73b46b9a452d10e7b216a9ea835497cf33d02a904b49f458d15435084670","group_id":"v3-legacy-transfer-authorization-0000","variant":"clean"}},"state_tokens":65,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":78,"state_len":65,"decide":77,"opts":[73,76],"row_ids_sha256":"526fa127ea666a6a6a9480eec96bcdc5b6235d2198657aec1da605dbe5d7507c","oracle":{"probs":[0.9943602681159973,0.00563979335129261],"logits_raw":[6.351830959320068,-6.092865943908691],"argmax":0,"top2_margin":0.9887204766273499,"near_tie":false},"graph":{"p":[0.9943249821662903,0.005675031803548336],"max_abs_dp":3.528594970703125e-05,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_02","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":590,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0001","pair_id":"v3-legacy-transfer-authorization-0001","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0001","id":"contrastive/authorization/v3-legacy-transfer-authorization-0001/a","text_sha256":"271eabac73e111da0e164e7e6e20159d34a804e80a51e6b8f89070689358ca68","row_sha256":"565d874a5f4062e94be13e8decce2b6ba07075ebfca990e7be6f60b5fc0906c8","group_id":"v3-legacy-transfer-authorization-0001","variant":"clean"}},"state_tokens":65,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":78,"state_len":65,"decide":77,"opts":[73,76],"row_ids_sha256":"4d3d46dee8ccbce46c06f0df3fcf23d8a5eafed79059d8b7d1fa582d8a5e4483","oracle":{"probs":[0.009908882901072502,0.99009108543396],"logits_raw":[-6.877449035644531,4.200883865356445],"argmax":1,"top2_margin":0.9801822304725647,"near_tie":false},"graph":{"p":[0.01014657225459814,0.9898534417152405],"max_abs_dp":0.00023768935352563858,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_03","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":591,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0001","pair_id":"v3-legacy-transfer-authorization-0001","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0001","id":"contrastive/authorization/v3-legacy-transfer-authorization-0001/b","text_sha256":"bec64bdb07d392ed48cb61c7bdc756a20a398395ab441ea59edd55447bed4a25","row_sha256":"de49096ff4ea76276f724473c358e2550e9cd571bad2c3c1863bcb021fb5202d","group_id":"v3-legacy-transfer-authorization-0001","variant":"clean"}},"state_tokens":64,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":77,"state_len":64,"decide":76,"opts":[72,75],"row_ids_sha256":"c587757bd302931a6f1ceb62b308e1e5cded8ecb145e7f87b490254c0dcf9dea","oracle":{"probs":[0.9921290874481201,0.007870982401072979],"logits_raw":[6.235422134399414,-5.401848793029785],"argmax":0,"top2_margin":0.9842581152915955,"near_tie":false},"graph":{"p":[0.9920793175697327,0.00792067963629961],"max_abs_dp":4.976987838745117e-05,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_04","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":592,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0002","pair_id":"v3-legacy-transfer-authorization-0002","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0002","id":"contrastive/authorization/v3-legacy-transfer-authorization-0002/a","text_sha256":"574d7caa6a760ed64982106052c743f6376d2aa7b24326ae56be46a1a402c30c","row_sha256":"974650c9a1402f048b27152f2e4982bd2c77b7ab994b97abad81a5e5e76e0168","group_id":"v3-legacy-transfer-authorization-0002","variant":"clean"}},"state_tokens":65,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":78,"state_len":65,"decide":77,"opts":[73,76],"row_ids_sha256":"401b391d606450c52b67a99697c1775f62dbb3a79c127164106ec6b6fe6b2570","oracle":{"probs":[0.007283738348633051,0.9927162528038025],"logits_raw":[-7.069869041442871,4.7553863525390625],"argmax":1,"top2_margin":0.985432505607605,"near_tie":false},"graph":{"p":[0.007399584166705608,0.9926003813743591],"max_abs_dp":0.00011587142944335938,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_05","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":593,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0002","pair_id":"v3-legacy-transfer-authorization-0002","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0002","id":"contrastive/authorization/v3-legacy-transfer-authorization-0002/b","text_sha256":"9033761e35fe5f8356f0d88559c800d1e36c84d38b260064d379cdacfdf1d902","row_sha256":"6266e4098a0564730723f608013b7cb52ebcaf6ad67720d848f44be3ab07fd81","group_id":"v3-legacy-transfer-authorization-0002","variant":"clean"}},"state_tokens":64,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":77,"state_len":64,"decide":76,"opts":[72,75],"row_ids_sha256":"2217b634cc6eb093b50d4b011977aacafebc8f7b470f250e7f65609082d4a221","oracle":{"probs":[0.9932976365089417,0.006702393293380737],"logits_raw":[6.236875057220459,-5.789924144744873],"argmax":0,"top2_margin":0.9865952730178833,"near_tie":false},"graph":{"p":[0.9932665824890137,0.00673337746411562],"max_abs_dp":3.1054019927978516e-05,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_06","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":594,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0003","pair_id":"v3-legacy-transfer-authorization-0003","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0003","id":"contrastive/authorization/v3-legacy-transfer-authorization-0003/a","text_sha256":"90f7b4f1e4633c41259e676aa81856a60d04ca903b186a957046bf6eece8e6f1","row_sha256":"aae0096dd83398cf2389cfcf820f81ccb7f05f65c0c3ac9adafe36d78cf3fd13","group_id":"v3-legacy-transfer-authorization-0003","variant":"clean"}},"state_tokens":62,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":75,"state_len":62,"decide":74,"opts":[70,73],"row_ids_sha256":"aea43c59375fa82ecd715358faa19fd7567f0e4eac5fc82719cd46c2b3963ac1","oracle":{"probs":[0.005004961974918842,0.9949949979782104],"logits_raw":[-7.363907814025879,5.3696489334106445],"argmax":1,"top2_margin":0.9899900555610657,"near_tie":false},"graph":{"p":[0.005119183100759983,0.9948807954788208],"max_abs_dp":0.00011422112584114075,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_07","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":595,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0003","pair_id":"v3-legacy-transfer-authorization-0003","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0003","id":"contrastive/authorization/v3-legacy-transfer-authorization-0003/b","text_sha256":"5cded64bb426ba6d4aa67e77405a9cd3d239a415437de0128e4e160e04867553","row_sha256":"4096ac0b5a6e3b84e8a9c9709ef3b22c0ea3b58e2d06e2305aa8dc3d10a7f691","group_id":"v3-legacy-transfer-authorization-0003","variant":"clean"}},"state_tokens":63,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":76,"state_len":63,"decide":75,"opts":[71,74],"row_ids_sha256":"897c3d661abba207e6e68f0e3eaf15a109a540d84440a4d51bff36564e052b6e","oracle":{"probs":[0.9971908926963806,0.0028091592248529196],"logits_raw":[6.945874214172363,-7.182588577270508],"argmax":0,"top2_margin":0.9943817257881165,"near_tie":false},"graph":{"p":[0.9971320629119873,0.0028678756207227707],"max_abs_dp":5.882978439331055e-05,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_08","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":596,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0004","pair_id":"v3-legacy-transfer-authorization-0004","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0004","id":"contrastive/authorization/v3-legacy-transfer-authorization-0004/a","text_sha256":"1c5cf01b96e9cf10fdf56ca75275cbff95d73ca7746f12d5241b23324d66f448","row_sha256":"b25cf494e3fe870d48e931c703bd8a43f1b8a9ea466dbd8a72cd9eb12030e1e0","group_id":"v3-legacy-transfer-authorization-0004","variant":"clean"}},"state_tokens":65,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":78,"state_len":65,"decide":77,"opts":[73,76],"row_ids_sha256":"4d298bd1b6fcff301b16fecd74346dc09e070229e597ae4a1052ebec8faaf06a","oracle":{"probs":[0.007033551577478647,0.9929664134979248],"logits_raw":[-7.067343711853027,4.842615604400635],"argmax":1,"top2_margin":0.9859328866004944,"near_tie":false},"graph":{"p":[0.007115194108337164,0.9928848147392273],"max_abs_dp":8.164253085851669e-05,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_09","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":597,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0004","pair_id":"v3-legacy-transfer-authorization-0004","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0004","id":"contrastive/authorization/v3-legacy-transfer-authorization-0004/b","text_sha256":"07c4c83c3738eb881c7eb630ff30ea7dd93c45dc7cdb5e1b07a56338ffb0245d","row_sha256":"45faa77f513bfd475dfe075cb98619a9b39d1d239f69d2aa3b0d381039b5d8a3","group_id":"v3-legacy-transfer-authorization-0004","variant":"clean"}},"state_tokens":65,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":78,"state_len":65,"decide":77,"opts":[73,76],"row_ids_sha256":"484527c90323d8ec19b301f336f9bc34ddfd877811ae42f8649c24bdc3cbd9fb","oracle":{"probs":[0.9934715032577515,0.006528508849442005],"logits_raw":[6.449974060058594,-5.640491485595703],"argmax":0,"top2_margin":0.9869430065155029,"near_tie":false},"graph":{"p":[0.9934403300285339,0.0065597426146268845],"max_abs_dp":3.12337651848793e-05,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_10","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":598,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0005","pair_id":"v3-legacy-transfer-authorization-0005","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0005","id":"contrastive/authorization/v3-legacy-transfer-authorization-0005/a","text_sha256":"4d3d3769d3d2191c191065c7403378cb6c30fb24ea807233305da13d7ef2e853","row_sha256":"6ef7d1426dae17368d2ce35151484d4ef411084a27d5a2cace6305e37bb68a8f","group_id":"v3-legacy-transfer-authorization-0005","variant":"clean"}},"state_tokens":64,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":77,"state_len":64,"decide":76,"opts":[72,75],"row_ids_sha256":"5164822edd8a3a3be5e3419b5b0509d78027bb8aeb2076a27ff66cc15beadd1c","oracle":{"probs":[0.009034690447151661,0.9909653663635254],"logits_raw":[-7.19028377532959,4.112395286560059],"argmax":1,"top2_margin":0.981930673122406,"near_tie":false},"graph":{"p":[0.009234748780727386,0.9907652139663696],"max_abs_dp":0.00020015239715576172,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_11","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":599,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0005","pair_id":"v3-legacy-transfer-authorization-0005","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0005","id":"contrastive/authorization/v3-legacy-transfer-authorization-0005/b","text_sha256":"f3f9a528d7d2183125eedc8f5661ca883e50fca6ae9979fc0cc8c92cd23c3e64","row_sha256":"b5b1ee21464a6bac5e38fd5379565b1d88802db539d6432fd3f3092be34192fd","group_id":"v3-legacy-transfer-authorization-0005","variant":"clean"}},"state_tokens":64,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":77,"state_len":64,"decide":76,"opts":[72,75],"row_ids_sha256":"2e8169227683fc94779956b46d30f9e0825c421f543971e65e00ca6d33df21c5","oracle":{"probs":[0.9936502575874329,0.006349707022309303],"logits_raw":[6.316184043884277,-5.841529369354248],"argmax":0,"top2_margin":0.9873005747795105,"near_tie":false},"graph":{"p":[0.9936311841011047,0.006368815898895264],"max_abs_dp":1.9108876585960388e-05,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_12","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":600,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0006","pair_id":"v3-legacy-transfer-authorization-0006","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0006","id":"contrastive/authorization/v3-legacy-transfer-authorization-0006/a","text_sha256":"ecd3c93d9b939d0f85444e52bd842e2d36cf0837bed11485152f5a31dd1bce58","row_sha256":"bfe78e8f9977da7242027380b90f05b448f3161150eb95d7d3ca8c91ad1d0624","group_id":"v3-legacy-transfer-authorization-0006","variant":"clean"}},"state_tokens":65,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":78,"state_len":65,"decide":77,"opts":[73,76],"row_ids_sha256":"cb9d6e21dd33edb4a0ef96cbbf6186860bd2ce78f2719e13f668a9cf5d52d5bc","oracle":{"probs":[0.005866703577339649,0.9941332936286926],"logits_raw":[-7.220009803771973,5.129230499267578],"argmax":1,"top2_margin":0.9882665872573853,"near_tie":false},"graph":{"p":[0.005965705029666424,0.9940343499183655],"max_abs_dp":9.90014523267746e-05,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_13","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":601,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0006","pair_id":"v3-legacy-transfer-authorization-0006","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0006","id":"contrastive/authorization/v3-legacy-transfer-authorization-0006/b","text_sha256":"3b017d703a12e51ab0f0b2103aa946faa9462f88375d06d8c2eb06914fcac35a","row_sha256":"8aad8b63e2ff385800a647aa1b2a64aecac781b33dd92b88670e516762c8eee0","group_id":"v3-legacy-transfer-authorization-0006","variant":"clean"}},"state_tokens":65,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":78,"state_len":65,"decide":77,"opts":[73,76],"row_ids_sha256":"c49322302671f0bc7c93cabf04e19bbd61e21cc25c8ae6fd1a2c771d3d0293c7","oracle":{"probs":[0.9958826303482056,0.004117405507713556],"logits_raw":[6.183998107910156,-7.02138090133667],"argmax":0,"top2_margin":0.9917652010917664,"near_tie":false},"graph":{"p":[0.9958102703094482,0.004189741797745228],"max_abs_dp":7.236003875732422e-05,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_14","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":602,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0007","pair_id":"v3-legacy-transfer-authorization-0007","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0007","id":"contrastive/authorization/v3-legacy-transfer-authorization-0007/a","text_sha256":"c3d41884d8a336d1d9179194fbc14c16f913d4d7f1e1629c373e5265f4cfdbff","row_sha256":"b6f039dfc5143e1695e27fb8d47a1bc920fb9fe7224d3c213e509c40580b4808","group_id":"v3-legacy-transfer-authorization-0007","variant":"clean"}},"state_tokens":63,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":76,"state_len":63,"decide":75,"opts":[71,74],"row_ids_sha256":"ece20c21454a2ebddf35c2427d2e4f1f8fcbc5c7ed80542381d313a1e7494eb3","oracle":{"probs":[0.005445572547614574,0.9945544004440308],"logits_raw":[-7.265783309936523,5.263701915740967],"argmax":1,"top2_margin":0.9891088008880615,"near_tie":false},"graph":{"p":[0.005523640662431717,0.9944763779640198],"max_abs_dp":7.806811481714249e-05,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_15","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":603,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0007","pair_id":"v3-legacy-transfer-authorization-0007","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0007","id":"contrastive/authorization/v3-legacy-transfer-authorization-0007/b","text_sha256":"7d7fd5548cc0cc301887f2df78a4fced5caebe785afa79c7b1333a56e9845122","row_sha256":"bf9d9a17775ba2e6a31271b4b2ddca06519e04612fbb7ad76cb367f825565e3e","group_id":"v3-legacy-transfer-authorization-0007","variant":"clean"}},"state_tokens":64,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":77,"state_len":64,"decide":76,"opts":[72,75],"row_ids_sha256":"46da0f8e67b7d4278e33fa062d29edbdfe3ea7906095cd026c0b1c679665d0b9","oracle":{"probs":[0.9955726861953735,0.004427345935255289],"logits_raw":[6.593457221984863,-6.436549663543701],"argmax":0,"top2_margin":0.9911453127861023,"near_tie":false},"graph":{"p":[0.9955099821090698,0.004490029998123646],"max_abs_dp":6.270408630371094e-05,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_16","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":604,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0008","pair_id":"v3-legacy-transfer-authorization-0008","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0008","id":"contrastive/authorization/v3-legacy-transfer-authorization-0008/a","text_sha256":"94da7e72d1aebd9b7460a8a4084f27285ee39303051ee737bcffede7c105293b","row_sha256":"892add71eeedd8d58d01a96db49223c9c311f66851dd689df41de05fb4e14957","group_id":"v3-legacy-transfer-authorization-0008","variant":"clean"}},"state_tokens":63,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":76,"state_len":63,"decide":75,"opts":[71,74],"row_ids_sha256":"7fc60fc6a1804e022bfde075f00fbea2fdf099dc310e28e5793bdad0fbe8b2e5","oracle":{"probs":[0.005723676178604364,0.9942763447761536],"logits_raw":[-7.363703727722168,5.045267105102539],"argmax":1,"top2_margin":0.9885526895523071,"near_tie":false},"graph":{"p":[0.005819388199597597,0.9941806197166443],"max_abs_dp":9.572505950927734e-05,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_17","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":605,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0008","pair_id":"v3-legacy-transfer-authorization-0008","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0008","id":"contrastive/authorization/v3-legacy-transfer-authorization-0008/b","text_sha256":"aa7487423f79b550238ef577886098b9268f0c4e25725b7ba1ca9383c024ce67","row_sha256":"ab93cf1d8d870794dc335cb5fdc49119cd9784ac56e3e4101439a7577b089dc9","group_id":"v3-legacy-transfer-authorization-0008","variant":"clean"}},"state_tokens":64,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":77,"state_len":64,"decide":76,"opts":[72,75],"row_ids_sha256":"af02d2bfb34258c442854a7d7097af3f1c3d3e50ba5f2571aed6f6bab1802567","oracle":{"probs":[0.9934685230255127,0.0065314569510519505],"logits_raw":[6.597652912139893,-5.4917192459106445],"argmax":0,"top2_margin":0.9869370460510254,"near_tie":false},"graph":{"p":[0.9934523105621338,0.006547721102833748],"max_abs_dp":1.626415178179741e-05,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_18","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":606,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0009","pair_id":"v3-legacy-transfer-authorization-0009","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0009","id":"contrastive/authorization/v3-legacy-transfer-authorization-0009/a","text_sha256":"7ce0efd4db2ef8224c841560d6b7d461fb8bbd6a20e06af9a420fa48e65ed42c","row_sha256":"a617c422ea7347aa0c746192a302f038077b92ae2aec541e76b8e901b118d571","group_id":"v3-legacy-transfer-authorization-0009","variant":"clean"}},"state_tokens":62,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":75,"state_len":62,"decide":74,"opts":[70,73],"row_ids_sha256":"fe357f17ef9f48efd3ccf11491e9fc8553c3a1f9147852fad9a3aed3b1919f4f","oracle":{"probs":[0.004943734034895897,0.9950562715530396],"logits_raw":[-7.468446731567383,5.29487419128418],"argmax":1,"top2_margin":0.9901125431060791,"near_tie":false},"graph":{"p":[0.005043978337198496,0.9949560761451721],"max_abs_dp":0.00010024430230259895,"argmax_equal":true}}]},{"id":"tv4x_legacy_holdout_19","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":607,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0009","pair_id":"v3-legacy-transfer-authorization-0009","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0009","id":"contrastive/authorization/v3-legacy-transfer-authorization-0009/b","text_sha256":"84bf86bf9cfd6d4232bcc52fb50594e132f5bad9cd156434686c4c046766b872","row_sha256":"731cfc2a35ba9bbaba7b0af0f72e1e13cd0cfb19a13ab5d0d896cfb5228a13ee","group_id":"v3-legacy-transfer-authorization-0009","variant":"clean"}},"state_tokens":63,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":76,"state_len":63,"decide":75,"opts":[71,74],"row_ids_sha256":"6da19dd4d7e6fec25ab94d73532b749fd8b7b540782d4bfbd7b119d3026a3c7d","oracle":{"probs":[0.9966118931770325,0.0033881019335240126],"logits_raw":[6.636788368225098,-7.039419174194336],"argmax":0,"top2_margin":0.9932237863540649,"near_tie":false},"graph":{"p":[0.9965757727622986,0.003424233291298151],"max_abs_dp":3.613135777413845e-05,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_00","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":668,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/0/relevant/a","group_id":"composition/v3-composition-transfer/held_and_or/0","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/0/relevant","sibling":"a","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"9d21c218a7b00c718c076542156adb5f2d344bf3f0fef09b372948148fb73f65","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"match","fields":["entry signer","entry designated approver"],"threshold":57},{"kind":"range","fields":["account value"],"threshold":14},{"kind":"le","fields":["shipment value"],"threshold":24}],"facts":{"entry signer":"Tomas","entry designated approver":"Kofi","account value":15,"shipment value":23,"routing reference":610},"order":["routing reference","shipment value","account value","entry designated approver","entry signer"],"deciding_field":"account value","label":true}}},"state_tokens":116,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":145,"state_len":116,"decide":144,"opts":[134,143],"row_ids_sha256":"9eee66df1602fd5e20e9dafcafb89be45bcc7d8b0d0b698aaa933d5ebc5722fb","oracle":{"probs":[0.012084481306374073,0.9879155158996582],"logits_raw":[-6.101183891296387,4.4942779541015625],"argmax":1,"top2_margin":0.9758310317993164,"near_tie":false},"graph":{"p":[0.012238842435181141,0.9877611398696899],"max_abs_dp":0.00015437602996826172,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_01","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":669,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/0/relevant/b","group_id":"composition/v3-composition-transfer/held_and_or/0","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/0/relevant","sibling":"b","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"6e759020afb08858bcc3422ac266ad43f50323c3f1268a2ce736ecc0a2677fc3","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"match","fields":["entry signer","entry designated approver"],"threshold":57},{"kind":"range","fields":["account value"],"threshold":14},{"kind":"le","fields":["shipment value"],"threshold":24}],"facts":{"entry signer":"Tomas","entry designated approver":"Kofi","account value":13,"shipment value":23,"routing reference":610},"order":["routing reference","shipment value","account value","entry designated approver","entry signer"],"deciding_field":"account value","label":false}}},"state_tokens":116,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":145,"state_len":116,"decide":144,"opts":[134,143],"row_ids_sha256":"c2ce3d03f44bc3481d4f1974dc1619e5a2a4a4a9ce25157d06ac961653a75f97","oracle":{"probs":[0.9782886505126953,0.02171137183904648],"logits_raw":[3.849426507949829,-5.312736511230469],"argmax":0,"top2_margin":0.9565773010253906,"near_tie":false},"graph":{"p":[0.9781193733215332,0.021880635991692543],"max_abs_dp":0.00016927719116210938,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_02","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":670,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/0/irrelevant/a","group_id":"composition/v3-composition-transfer/held_and_or/0","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/0/irrelevant","sibling":"a","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"9d21c218a7b00c718c076542156adb5f2d344bf3f0fef09b372948148fb73f65","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"match","fields":["entry signer","entry designated approver"],"threshold":57},{"kind":"range","fields":["account value"],"threshold":14},{"kind":"le","fields":["shipment value"],"threshold":24}],"facts":{"entry signer":"Tomas","entry designated approver":"Kofi","account value":15,"shipment value":23,"routing reference":610},"order":["routing reference","shipment value","account value","entry designated approver","entry signer"],"deciding_field":"account value","label":true}}},"state_tokens":116,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":145,"state_len":116,"decide":144,"opts":[134,143],"row_ids_sha256":"9eee66df1602fd5e20e9dafcafb89be45bcc7d8b0d0b698aaa933d5ebc5722fb","oracle":{"probs":[0.012084481306374073,0.9879155158996582],"logits_raw":[-6.101183891296387,4.4942779541015625],"argmax":1,"top2_margin":0.9758310317993164,"near_tie":false},"graph":{"p":[0.012238842435181141,0.9877611398696899],"max_abs_dp":0.00015437602996826172,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_03","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":671,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/0/irrelevant/b","group_id":"composition/v3-composition-transfer/held_and_or/0","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/0/irrelevant","sibling":"b","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"c37513c5cf15a96d63b6bcf1f7bc6c02f8ab02a7445170a07528081709f07aab","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"match","fields":["entry signer","entry designated approver"],"threshold":57},{"kind":"range","fields":["account value"],"threshold":14},{"kind":"le","fields":["shipment value"],"threshold":24}],"facts":{"entry signer":"Tomas","entry designated approver":"Kofi","account value":15,"shipment value":23,"routing reference":185},"order":["routing reference","shipment value","account value","entry designated approver","entry signer"],"deciding_field":"account value","label":true}}},"state_tokens":116,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":145,"state_len":116,"decide":144,"opts":[134,143],"row_ids_sha256":"53fbfcb855bd554e4714b868b25fd2e4ab35bac6a11980bc2a7df18f9490420d","oracle":{"probs":[0.012034783139824867,0.9879652261734009],"logits_raw":[-6.108939170837402,4.4965596199035645],"argmax":1,"top2_margin":0.9759304523468018,"near_tie":false},"graph":{"p":[0.012175633572041988,0.9878243207931519],"max_abs_dp":0.00014090538024902344,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_04","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":672,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/1/relevant/a","group_id":"composition/v3-composition-transfer/held_and_or/1","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/1/relevant","sibling":"a","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"e308fdfe8a88e0200c7281e2d1b6312ec2e22874a2e1165eb1088ebdf98d776a","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["review value"],"threshold":7},{"kind":"le","fields":["account value"],"threshold":47},{"kind":"match","fields":["request signer","request designated approver"],"threshold":47}],"facts":{"review value":7,"account value":46,"request signer":"Mira","request designated approver":"Sana","routing reference":131},"order":["request designated approver","account value","review value","routing reference","request signer"],"deciding_field":"request signer","label":false}}},"state_tokens":107,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":136,"state_len":107,"decide":135,"opts":[123,134],"row_ids_sha256":"2be904190095bc64a12aeb9413ab0a7d3f09c84308b300e3e5569dcbe84f3eb8","oracle":{"probs":[0.008753994479775429,0.9912460446357727],"logits_raw":[-7.012086868286133,4.367212772369385],"argmax":1,"top2_margin":0.9824920296669006,"near_tie":false},"graph":{"p":[0.00890008732676506,0.9910998940467834],"max_abs_dp":0.0001461505889892578,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_05","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":673,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/1/relevant/b","group_id":"composition/v3-composition-transfer/held_and_or/1","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/1/relevant","sibling":"b","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"c237482c05993ce384d2fe35c55853b22c11d677cecc9916466eff5498247811","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["review value"],"threshold":7},{"kind":"le","fields":["account value"],"threshold":47},{"kind":"match","fields":["request signer","request designated approver"],"threshold":47}],"facts":{"review value":7,"account value":46,"request signer":"Sana","request designated approver":"Sana","routing reference":131},"order":["request designated approver","account value","review value","routing reference","request signer"],"deciding_field":"request signer","label":true}}},"state_tokens":108,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":137,"state_len":108,"decide":136,"opts":[124,135],"row_ids_sha256":"75de31e073dd90e3769e51e19d6446926d89760a1d0cebeab150dda67170307b","oracle":{"probs":[0.9872406125068665,0.012759440578520298],"logits_raw":[3.6772356033325195,-6.785815715789795],"argmax":0,"top2_margin":0.9744811654090881,"near_tie":false},"graph":{"p":[0.987064003944397,0.012935995124280453],"max_abs_dp":0.00017660856246948242,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_06","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":674,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/1/irrelevant/a","group_id":"composition/v3-composition-transfer/held_and_or/1","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/1/irrelevant","sibling":"a","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"e308fdfe8a88e0200c7281e2d1b6312ec2e22874a2e1165eb1088ebdf98d776a","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["review value"],"threshold":7},{"kind":"le","fields":["account value"],"threshold":47},{"kind":"match","fields":["request signer","request designated approver"],"threshold":47}],"facts":{"review value":7,"account value":46,"request signer":"Mira","request designated approver":"Sana","routing reference":131},"order":["request designated approver","account value","review value","routing reference","request signer"],"deciding_field":"request signer","label":false}}},"state_tokens":107,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":136,"state_len":107,"decide":135,"opts":[123,134],"row_ids_sha256":"2be904190095bc64a12aeb9413ab0a7d3f09c84308b300e3e5569dcbe84f3eb8","oracle":{"probs":[0.008753994479775429,0.9912460446357727],"logits_raw":[-7.012086868286133,4.367212772369385],"argmax":1,"top2_margin":0.9824920296669006,"near_tie":false},"graph":{"p":[0.00890008732676506,0.9910998940467834],"max_abs_dp":0.0001461505889892578,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_07","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":675,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/1/irrelevant/b","group_id":"composition/v3-composition-transfer/held_and_or/1","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/1/irrelevant","sibling":"b","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"79886061d045f1d5332c4dd9643caeda04a0aeba9939bcfb40315e3c3e76fc89","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["review value"],"threshold":7},{"kind":"le","fields":["account value"],"threshold":47},{"kind":"match","fields":["request signer","request designated approver"],"threshold":47}],"facts":{"review value":7,"account value":46,"request signer":"Mira","request designated approver":"Sana","routing reference":980},"order":["request designated approver","account value","review value","routing reference","request signer"],"deciding_field":"request signer","label":false}}},"state_tokens":107,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":136,"state_len":107,"decide":135,"opts":[123,134],"row_ids_sha256":"93c83fac4bf8c7563806ca5543e3ee181e890c005de2b6810399f138433cd561","oracle":{"probs":[0.008712965995073318,0.9912870526313782],"logits_raw":[-7.028288841247559,4.3624138832092285],"argmax":1,"top2_margin":0.9825741052627563,"near_tie":false},"graph":{"p":[0.008853686973452568,0.9911462664604187],"max_abs_dp":0.00014078617095947266,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_08","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":676,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/2/relevant/a","group_id":"composition/v3-composition-transfer/held_and_or/2","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/2/relevant","sibling":"a","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"88789ea4fc9d940247d5027641a0a57a4f4e858005a4ab6b49b6be705372a2df","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"elapsed","fields":["case start date","case end date"],"threshold":59},{"kind":"flag","fields":["shipment verified"],"threshold":44},{"kind":"match","fields":["member signer","member designated approver"],"threshold":44}],"facts":{"case start date":"2027-08-27","case end date":"2027-10-24","shipment verified":false,"member signer":"Noah","member designated approver":"Noah","routing reference":280},"order":["member designated approver","member signer","shipment verified","case end date","routing reference","case start date"],"deciding_field":"member signer","label":true}}},"state_tokens":137,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":166,"state_len":137,"decide":165,"opts":[155,164],"row_ids_sha256":"6381e746afccc8b02747587f67bed60fb743f5d2dd358509b770bfd0e5a787cc","oracle":{"probs":[0.41759800910949707,0.5824019312858582],"logits_raw":[-1.670501947402954,-0.8701499700546265],"argmax":1,"top2_margin":0.16480392217636108,"near_tie":false},"graph":{"p":[0.4230703115463257,0.5769296288490295],"max_abs_dp":0.005472302436828613,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_09","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":677,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/2/relevant/b","group_id":"composition/v3-composition-transfer/held_and_or/2","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/2/relevant","sibling":"b","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"e37ef88d426d56212bb754e5a65f0aeafa211c282d1577867b00a2248ccd5770","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"elapsed","fields":["case start date","case end date"],"threshold":59},{"kind":"flag","fields":["shipment verified"],"threshold":44},{"kind":"match","fields":["member signer","member designated approver"],"threshold":44}],"facts":{"case start date":"2027-08-27","case end date":"2027-10-24","shipment verified":false,"member signer":"Aiko","member designated approver":"Noah","routing reference":280},"order":["member designated approver","member signer","shipment verified","case end date","routing reference","case start date"],"deciding_field":"member signer","label":false}}},"state_tokens":138,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":167,"state_len":138,"decide":166,"opts":[156,165],"row_ids_sha256":"55d19c19295f07135a2bc1c85427393a74cf5d333a5fc297049b98958870afbd","oracle":{"probs":[0.9802345633506775,0.019765453413128853],"logits_raw":[3.574431896209717,-5.818441390991211],"argmax":0,"top2_margin":0.960469126701355,"near_tie":false},"graph":{"p":[0.9798396229743958,0.020160337910056114],"max_abs_dp":0.0003949403762817383,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_10","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":678,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/2/irrelevant/a","group_id":"composition/v3-composition-transfer/held_and_or/2","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/2/irrelevant","sibling":"a","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"88789ea4fc9d940247d5027641a0a57a4f4e858005a4ab6b49b6be705372a2df","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"elapsed","fields":["case start date","case end date"],"threshold":59},{"kind":"flag","fields":["shipment verified"],"threshold":44},{"kind":"match","fields":["member signer","member designated approver"],"threshold":44}],"facts":{"case start date":"2027-08-27","case end date":"2027-10-24","shipment verified":false,"member signer":"Noah","member designated approver":"Noah","routing reference":280},"order":["member designated approver","member signer","shipment verified","case end date","routing reference","case start date"],"deciding_field":"member signer","label":true}}},"state_tokens":137,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":166,"state_len":137,"decide":165,"opts":[155,164],"row_ids_sha256":"6381e746afccc8b02747587f67bed60fb743f5d2dd358509b770bfd0e5a787cc","oracle":{"probs":[0.41759800910949707,0.5824019312858582],"logits_raw":[-1.670501947402954,-0.8701499700546265],"argmax":1,"top2_margin":0.16480392217636108,"near_tie":false},"graph":{"p":[0.4230703115463257,0.5769296288490295],"max_abs_dp":0.005472302436828613,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_11","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":679,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/2/irrelevant/b","group_id":"composition/v3-composition-transfer/held_and_or/2","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/2/irrelevant","sibling":"b","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"594d32ac9bd72de204403039f35eb2150741195ef36ea7e9c04551a52103eb42","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"elapsed","fields":["case start date","case end date"],"threshold":59},{"kind":"flag","fields":["shipment verified"],"threshold":44},{"kind":"match","fields":["member signer","member designated approver"],"threshold":44}],"facts":{"case start date":"2027-08-27","case end date":"2027-10-24","shipment verified":false,"member signer":"Noah","member designated approver":"Noah","routing reference":534},"order":["member designated approver","member signer","shipment verified","case end date","routing reference","case start date"],"deciding_field":"member signer","label":true}}},"state_tokens":137,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":166,"state_len":137,"decide":165,"opts":[155,164],"row_ids_sha256":"cca1c033448ad5ddd947fc4be4612505bf90ff1992eb4f246795fefe1d7aa9ad","oracle":{"probs":[0.4109298586845398,0.5890701413154602],"logits_raw":[-1.6219537258148193,-0.7554805278778076],"argmax":1,"top2_margin":0.1781402826309204,"near_tie":false},"graph":{"p":[0.4170590937137604,0.5829408764839172],"max_abs_dp":0.006129264831542969,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_12","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":680,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/3/relevant/a","group_id":"composition/v3-composition-transfer/held_and_or/3","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/3/relevant","sibling":"a","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"44333ca47dd477e539907f962281754aeba97abb57ff3fe0fc64b652bf341ec9","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["package value"],"threshold":51},{"kind":"flag","fields":["member verified"],"threshold":24},{"kind":"gt","fields":["account value"],"threshold":49}],"facts":{"package value":61,"member verified":true,"account value":48,"routing reference":251},"order":["account value","member verified","routing reference","package value"],"deciding_field":"account value","label":false}}},"state_tokens":93,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":122,"state_len":93,"decide":121,"opts":[111,120],"row_ids_sha256":"9d92bcac458d60bde2915f3e8c2c2b98c508b99452bb55574f81c006c03f64d6","oracle":{"probs":[0.9877034425735474,0.012296569533646107],"logits_raw":[4.170540809631348,-6.382543563842773],"argmax":0,"top2_margin":0.9754068851470947,"near_tie":false},"graph":{"p":[0.9875471591949463,0.012452778406441212],"max_abs_dp":0.00015628337860107422,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_13","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":681,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/3/relevant/b","group_id":"composition/v3-composition-transfer/held_and_or/3","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/3/relevant","sibling":"b","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"38b842d897276b85c54a2398f0269e9bbd01e488a08934084d0f273db092d1d9","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["package value"],"threshold":51},{"kind":"flag","fields":["member verified"],"threshold":24},{"kind":"gt","fields":["account value"],"threshold":49}],"facts":{"package value":61,"member verified":true,"account value":50,"routing reference":251},"order":["account value","member verified","routing reference","package value"],"deciding_field":"account value","label":true}}},"state_tokens":93,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":122,"state_len":93,"decide":121,"opts":[111,120],"row_ids_sha256":"c96ef0b6019fb9120f4d010b8db56bfd04d1bee57aad309259d520d050916bcc","oracle":{"probs":[0.20091518759727478,0.7990848422050476],"logits_raw":[-3.0211386680603027,0.3006158471107483],"argmax":1,"top2_margin":0.5981696844100952,"near_tie":false},"graph":{"p":[0.20037148892879486,0.799628496170044],"max_abs_dp":0.0005436986684799194,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_14","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":682,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/3/irrelevant/a","group_id":"composition/v3-composition-transfer/held_and_or/3","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/3/irrelevant","sibling":"a","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"44333ca47dd477e539907f962281754aeba97abb57ff3fe0fc64b652bf341ec9","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["package value"],"threshold":51},{"kind":"flag","fields":["member verified"],"threshold":24},{"kind":"gt","fields":["account value"],"threshold":49}],"facts":{"package value":61,"member verified":true,"account value":48,"routing reference":251},"order":["account value","member verified","routing reference","package value"],"deciding_field":"account value","label":false}}},"state_tokens":93,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":122,"state_len":93,"decide":121,"opts":[111,120],"row_ids_sha256":"9d92bcac458d60bde2915f3e8c2c2b98c508b99452bb55574f81c006c03f64d6","oracle":{"probs":[0.9877034425735474,0.012296569533646107],"logits_raw":[4.170540809631348,-6.382543563842773],"argmax":0,"top2_margin":0.9754068851470947,"near_tie":false},"graph":{"p":[0.9875471591949463,0.012452778406441212],"max_abs_dp":0.00015628337860107422,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_15","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":683,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/3/irrelevant/b","group_id":"composition/v3-composition-transfer/held_and_or/3","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/3/irrelevant","sibling":"b","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"5ae23575dd80b3268d4923affc02a219801d66f11840e53c897244e64e0ce3e8","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["package value"],"threshold":51},{"kind":"flag","fields":["member verified"],"threshold":24},{"kind":"gt","fields":["account value"],"threshold":49}],"facts":{"package value":61,"member verified":true,"account value":48,"routing reference":573},"order":["account value","member verified","routing reference","package value"],"deciding_field":"account value","label":false}}},"state_tokens":93,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":122,"state_len":93,"decide":121,"opts":[111,120],"row_ids_sha256":"f85fd3f67a8b9a588894b9bc168fcca99fe7ad90c341d68ec57cf1bfd6160506","oracle":{"probs":[0.9877790808677673,0.012220962904393673],"logits_raw":[4.175275802612305,-6.392832279205322],"argmax":0,"top2_margin":0.9755581021308899,"near_tie":false},"graph":{"p":[0.9876281023025513,0.012371846474707127],"max_abs_dp":0.00015097856521606445,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_16","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":684,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/4/relevant/a","group_id":"composition/v3-composition-transfer/held_and_or/4","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/4/relevant","sibling":"a","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"ee1eab34afdaea0093568ecfa1a1025cb93f17c4c1cc85043e7a61da75b32c39","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"match","fields":["package signer","package designated approver"],"threshold":36},{"kind":"le","fields":["entry value"],"threshold":57},{"kind":"match","fields":["account signer","account designated approver"],"threshold":38}],"facts":{"package signer":"Noah","package designated approver":"Ravi","entry value":56,"account signer":"Sana","account designated approver":"Ravi","routing reference":624},"order":["account signer","package designated approver","routing reference","account designated approver","package signer","entry value"],"deciding_field":"account designated approver","label":false}}},"state_tokens":120,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":149,"state_len":120,"decide":148,"opts":[136,147],"row_ids_sha256":"9f6dc3d48da8e1a7e1d1e8206b52874d328e305943f775ef3150c534221f9b4a","oracle":{"probs":[0.12423297017812729,0.8757670521736145],"logits_raw":[-4.070282459259033,0.6285924315452576],"argmax":1,"top2_margin":0.751534104347229,"near_tie":false},"graph":{"p":[0.12714532017707825,0.8728546500205994],"max_abs_dp":0.0029124021530151367,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_17","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":685,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/4/relevant/b","group_id":"composition/v3-composition-transfer/held_and_or/4","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/4/relevant","sibling":"b","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"03cfd4b4074dbbf8e3c8420fe91a93db29eda4e7a97252e1f6ce79ded694bcc6","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"match","fields":["package signer","package designated approver"],"threshold":36},{"kind":"le","fields":["entry value"],"threshold":57},{"kind":"match","fields":["account signer","account designated approver"],"threshold":38}],"facts":{"package signer":"Noah","package designated approver":"Ravi","entry value":56,"account signer":"Sana","account designated approver":"Sana","routing reference":624},"order":["account signer","package designated approver","routing reference","account designated approver","package signer","entry value"],"deciding_field":"account designated approver","label":true}}},"state_tokens":120,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":149,"state_len":120,"decide":148,"opts":[136,147],"row_ids_sha256":"e8354787c8d7bb8dad7f363a25c82b18c7faa00ef11d95369af98e4ecd2ec008","oracle":{"probs":[0.9889533519744873,0.0110466368496418],"logits_raw":[4.428966522216797,-6.38507604598999],"argmax":0,"top2_margin":0.9779067039489746,"near_tie":false},"graph":{"p":[0.9887895584106445,0.01121042761951685],"max_abs_dp":0.00016379356384277344,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_18","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":686,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/4/irrelevant/a","group_id":"composition/v3-composition-transfer/held_and_or/4","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/4/irrelevant","sibling":"a","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"ee1eab34afdaea0093568ecfa1a1025cb93f17c4c1cc85043e7a61da75b32c39","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"match","fields":["package signer","package designated approver"],"threshold":36},{"kind":"le","fields":["entry value"],"threshold":57},{"kind":"match","fields":["account signer","account designated approver"],"threshold":38}],"facts":{"package signer":"Noah","package designated approver":"Ravi","entry value":56,"account signer":"Sana","account designated approver":"Ravi","routing reference":624},"order":["account signer","package designated approver","routing reference","account designated approver","package signer","entry value"],"deciding_field":"account designated approver","label":false}}},"state_tokens":120,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":149,"state_len":120,"decide":148,"opts":[136,147],"row_ids_sha256":"9f6dc3d48da8e1a7e1d1e8206b52874d328e305943f775ef3150c534221f9b4a","oracle":{"probs":[0.12423297017812729,0.8757670521736145],"logits_raw":[-4.070282459259033,0.6285924315452576],"argmax":1,"top2_margin":0.751534104347229,"near_tie":false},"graph":{"p":[0.12714532017707825,0.8728546500205994],"max_abs_dp":0.0029124021530151367,"argmax_equal":true}}]},{"id":"tv4x_composition_holdout_19","source":"transfer_v4_dev_sources","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":687,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/4/irrelevant/b","group_id":"composition/v3-composition-transfer/held_and_or/4","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/4/irrelevant","sibling":"b","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"b725e3446fd24db24714ad2c277ca4e2a5943831b9bd3769d41d4bf3506d02e5","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"match","fields":["package signer","package designated approver"],"threshold":36},{"kind":"le","fields":["entry value"],"threshold":57},{"kind":"match","fields":["account signer","account designated approver"],"threshold":38}],"facts":{"package signer":"Noah","package designated approver":"Ravi","entry value":56,"account signer":"Sana","account designated approver":"Ravi","routing reference":326},"order":["account signer","package designated approver","routing reference","account designated approver","package signer","entry value"],"deciding_field":"account designated approver","label":false}}},"state_tokens":120,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":149,"state_len":120,"decide":148,"opts":[136,147],"row_ids_sha256":"51e06b6fa3300bb4335d3dfcfb9b819a1c497b0b610994ecc8e45c3f3ff91e73","oracle":{"probs":[0.13048654794692993,0.8695133924484253],"logits_raw":[-4.005674839019775,0.5577926635742188],"argmax":1,"top2_margin":0.7390268445014954,"near_tie":false},"graph":{"p":[0.13305813074111938,0.8669419288635254],"max_abs_dp":0.002571582794189453,"argmax_equal":true}}]},{"id":"tv4s_00","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":628,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0000","pair_id":"v3-legacy-transfer-deadline-0000","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0000","id":"contrastive/deadline/v3-legacy-transfer-deadline-0000/a","text_sha256":"60fc44aa009aad0ef852c67705132af2432c7a855f61b8d2010877ae53164c03","row_sha256":"79838f04b3dcbb877bab439bac73134a4715c193971c8795e28819b67e54bf2a","group_id":"v3-legacy-transfer-deadline-0000","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"score","n_options":3,"row_len":99,"state_len":78,"decide":98,"opts":[88,93,97],"row_ids_sha256":"91f7d8b38bd0480095914733ab35c2f2d415f529f05b53d1e6c94d71c3594b76","oracle":{"probs":[0.9447954297065735,0.03060240112245083,0.0246022567152977],"logits_raw":[2.1891849040985107,-6.0633015632629395,-6.588398456573486],"argmax":0,"top2_margin":0.9141930341720581,"near_tie":false},"graph":{"p":[0.9437302947044373,0.031057806685566902,0.025211889296770096],"max_abs_dp":0.0010651350021362305,"argmax_equal":true}}]},{"id":"tv4s_01","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":629,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0000","pair_id":"v3-legacy-transfer-deadline-0000","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0000","id":"contrastive/deadline/v3-legacy-transfer-deadline-0000/b","text_sha256":"7ee8365ace01dd340f7eb13fde0813efb80021c429d3f53798935c0f7df199e1","row_sha256":"a37b38bd0560d4d21da5a97bf5a3ef47bf2325e3ca6e90ec5ceef9267d082c3b","group_id":"v3-legacy-transfer-deadline-0000","variant":"clean"}},"state_tokens":77,"questions":[{"q":0,"type":"score","n_options":3,"row_len":98,"state_len":77,"decide":97,"opts":[87,92,96],"row_ids_sha256":"9e176f468b63b8f124908f53830f2f0bf89e142cf31d111a2718f70da2ec0388","oracle":{"probs":[0.09261045604944229,0.8792119026184082,0.02817772887647152],"logits_raw":[-5.188676357269287,0.22643718123435974,-8.051563262939453],"argmax":1,"top2_margin":0.7866014242172241,"near_tie":false},"graph":{"p":[0.09041955322027206,0.8811270594596863,0.028453316539525986],"max_abs_dp":0.002190902829170227,"argmax_equal":true}}]},{"id":"tv4s_02","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":630,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0001","pair_id":"v3-legacy-transfer-deadline-0001","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0001","id":"contrastive/deadline/v3-legacy-transfer-deadline-0001/a","text_sha256":"6a4c4a36c00d6c4858efd120cca9eff44df4258bc9ddc093bc4938bd94e57846","row_sha256":"594793cc8703daca99c86621a9931dee60eca526b6f1755f89051791e3bb755b","group_id":"v3-legacy-transfer-deadline-0001","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"score","n_options":3,"row_len":99,"state_len":78,"decide":98,"opts":[88,93,97],"row_ids_sha256":"9e552a3656b09fcb857849e51ec987e1e41f41ef6b2ea3aea1e6b0693e5742c6","oracle":{"probs":[0.39471790194511414,0.5253556966781616,0.07992632687091827],"logits_raw":[-3.294379472732544,-2.6064796447753906,-7.13700008392334],"argmax":1,"top2_margin":0.13063779473304749,"near_tie":false},"graph":{"p":[0.39340829849243164,0.5258045196533203,0.08078719675540924],"max_abs_dp":0.0013096034526824951,"argmax_equal":true}}]},{"id":"tv4s_03","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":631,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0001","pair_id":"v3-legacy-transfer-deadline-0001","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0001","id":"contrastive/deadline/v3-legacy-transfer-deadline-0001/b","text_sha256":"c94f79861d12ea95869612f0e571189d58acb302119b71d00adc9b792403a66c","row_sha256":"ebddf4f71912e0a1d93a48984634f94afb16c89aaba947ec2be47b74d9597066","group_id":"v3-legacy-transfer-deadline-0001","variant":"clean"}},"state_tokens":79,"questions":[{"q":0,"type":"score","n_options":3,"row_len":100,"state_len":79,"decide":99,"opts":[89,94,98],"row_ids_sha256":"80a7d295b3129540e8eeabaf8540b14eaeb170df89989fdb9f3f648e038172fc","oracle":{"probs":[0.03675976023077965,0.9524546265602112,0.010785609483718872],"logits_raw":[-6.640575408935547,1.1902482509613037,-9.590851783752441],"argmax":1,"top2_margin":0.9156948924064636,"near_tie":false},"graph":{"p":[0.03777331858873367,0.9512805938720703,0.010946103371679783],"max_abs_dp":0.0011740326881408691,"argmax_equal":true}}]},{"id":"tv4s_04","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":632,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0002","pair_id":"v3-legacy-transfer-deadline-0002","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0002","id":"contrastive/deadline/v3-legacy-transfer-deadline-0002/a","text_sha256":"e7abe027950fcd48a41c4e72bc5a7336fae563c9b8541f26ff733a2905f665e1","row_sha256":"f19aef83ea00725d0d9920c4462b46b40681657ff506ed348d8520315b004b45","group_id":"v3-legacy-transfer-deadline-0002","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"score","n_options":3,"row_len":99,"state_len":78,"decide":98,"opts":[88,93,97],"row_ids_sha256":"31306fa196ad0cb3ec6e65cacab4db3b27b9628a83bc116d981f5551c0eb9282","oracle":{"probs":[0.8276988863945007,0.08645104616880417,0.08585003763437271],"logits_raw":[0.5485004782676697,-4.886937141418457,-4.903722763061523],"argmax":0,"top2_margin":0.741247832775116,"near_tie":false},"graph":{"p":[0.8218695521354675,0.08915319293737411,0.08897726982831955],"max_abs_dp":0.005829334259033203,"argmax_equal":true}}]},{"id":"tv4s_05","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":633,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0002","pair_id":"v3-legacy-transfer-deadline-0002","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0002","id":"contrastive/deadline/v3-legacy-transfer-deadline-0002/b","text_sha256":"536f00f1b4a91c96dd327d22f1d409c2ecdeb065649d011c91aa09211404b75e","row_sha256":"75b49419e6b53a9d9d04a9898c415b1793bf01d18472b2a01de7a8660f55f604","group_id":"v3-legacy-transfer-deadline-0002","variant":"clean"}},"state_tokens":77,"questions":[{"q":0,"type":"score","n_options":3,"row_len":98,"state_len":77,"decide":97,"opts":[87,92,96],"row_ids_sha256":"484462d20cd6721ae9b0d2c3b1b953b6300713f185ac4089c3691c3e0d421e2a","oracle":{"probs":[0.06186022609472275,0.9207215309143066,0.01741824857890606],"logits_raw":[-5.841314792633057,0.6556941866874695,-8.890644073486328],"argmax":1,"top2_margin":0.8588613271713257,"near_tie":false},"graph":{"p":[0.059438616037368774,0.9232208728790283,0.01734049618244171],"max_abs_dp":0.0024993419647216797,"argmax_equal":true}}]},{"id":"tv4s_06","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":634,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0003","pair_id":"v3-legacy-transfer-deadline-0003","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0003","id":"contrastive/deadline/v3-legacy-transfer-deadline-0003/a","text_sha256":"518551635a353eb9b3666a2ad2e2a56c700352a0abe6b96235c7a9c300be1fcc","row_sha256":"85c34557c97a43a6b527f1eb5d81501fb3753466cf5d430d11814ce379883281","group_id":"v3-legacy-transfer-deadline-0003","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"score","n_options":3,"row_len":99,"state_len":78,"decide":98,"opts":[88,93,97],"row_ids_sha256":"affb2139b2585fb3dbe8ff545e8b13a7fb577dff39564976df60cc3b7ed024f6","oracle":{"probs":[0.33867859840393066,0.4852241277694702,0.17609724402427673],"logits_raw":[-2.9194631576538086,-2.0543456077575684,-4.493056774139404],"argmax":1,"top2_margin":0.14654552936553955,"near_tie":false},"graph":{"p":[0.33583250641822815,0.4851577877998352,0.17900969088077545],"max_abs_dp":0.0029124468564987183,"argmax_equal":true}}]},{"id":"tv4s_07","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":635,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0003","pair_id":"v3-legacy-transfer-deadline-0003","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0003","id":"contrastive/deadline/v3-legacy-transfer-deadline-0003/b","text_sha256":"bd59d3dc5d85b02f3cd8856ec2470b6dc904b52d2a2bb2ad74eb758e3dbde460","row_sha256":"919f3f660c4a5cf98310c25f8d46c27e2fb11b2a330231aa79c04f4bcbbc757f","group_id":"v3-legacy-transfer-deadline-0003","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"score","n_options":3,"row_len":99,"state_len":78,"decide":98,"opts":[88,93,97],"row_ids_sha256":"eb931343e293ba6c57d6d7679cbbd33aae8216c3fb9e11a3df2f1b419c7c5417","oracle":{"probs":[0.008263319730758667,0.775822639465332,0.21591408550739288],"logits_raw":[-11.379766464233398,-0.45125243067741394,-3.5286946296691895],"argmax":1,"top2_margin":0.5599085688591003,"near_tie":false},"graph":{"p":[0.008447147905826569,0.7715010643005371,0.22005176544189453],"max_abs_dp":0.004321575164794922,"argmax_equal":true}}]},{"id":"tv4s_08","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":636,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0004","pair_id":"v3-legacy-transfer-deadline-0004","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0004","id":"contrastive/deadline/v3-legacy-transfer-deadline-0004/a","text_sha256":"3f3a8ad387844c8104129d70bdaf40a72b9270d90ca1064fd031494b11205761","row_sha256":"8bf056570b0810ee9edbbe9a8634216c9d234bd8e4693d56c0d1cf1b9c54550b","group_id":"v3-legacy-transfer-deadline-0004","variant":"clean"}},"state_tokens":77,"questions":[{"q":0,"type":"score","n_options":3,"row_len":98,"state_len":77,"decide":97,"opts":[87,92,96],"row_ids_sha256":"8c51a9eafdb3477f15f3dfef8f8df76f17586beb875158e53387cfc3b17914df","oracle":{"probs":[0.4107963442802429,0.5505679249763489,0.038635700941085815],"logits_raw":[-2.946282386779785,-2.241664171218872,-8.633994102478027],"argmax":1,"top2_margin":0.13977158069610596,"near_tie":false},"graph":{"p":[0.40943384170532227,0.5515697598457336,0.038996484130620956],"max_abs_dp":0.0013625025749206543,"argmax_equal":true}}]},{"id":"tv4s_09","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":637,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0004","pair_id":"v3-legacy-transfer-deadline-0004","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0004","id":"contrastive/deadline/v3-legacy-transfer-deadline-0004/b","text_sha256":"ac0c886d2f3cc92a0141cf91dea9b3936564915a88134052cbf3a900a7b1962c","row_sha256":"c45c3a04933b7389e59f16530e4238b39366230f23edd10407bee44131367d83","group_id":"v3-legacy-transfer-deadline-0004","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"score","n_options":3,"row_len":99,"state_len":78,"decide":98,"opts":[88,93,97],"row_ids_sha256":"7488df5d6aebc0dfe26b75db21e4d02ef3db1fee8b8a7f69bbdd13752608b8e7","oracle":{"probs":[0.09061268717050552,0.8927403688430786,0.01664700172841549],"logits_raw":[-4.442709922790527,1.0616142749786377,-8.519434928894043],"argmax":1,"top2_margin":0.8021276593208313,"near_tie":false},"graph":{"p":[0.09412481635808945,0.8890618681907654,0.01681334152817726],"max_abs_dp":0.0036785006523132324,"argmax_equal":true}}]},{"id":"tv4s_10","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":638,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0005","pair_id":"v3-legacy-transfer-deadline-0005","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0005","id":"contrastive/deadline/v3-legacy-transfer-deadline-0005/a","text_sha256":"1dcf029b2f29eff1d5db377ecc9f7eaa5780943529cb6c0b6d9181e65ade83e3","row_sha256":"7f2ca1f8d91f857466e604bb40d6996f6fcc5e0bd16d44ae4dfc04af92beaf5a","group_id":"v3-legacy-transfer-deadline-0005","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"score","n_options":3,"row_len":99,"state_len":78,"decide":98,"opts":[88,93,97],"row_ids_sha256":"6894a80fd195e2467eb322d77aa21b58a0f295dca5115b6f002c4847122f37fb","oracle":{"probs":[0.2025303840637207,0.5787302255630493,0.2187393456697464],"logits_raw":[-4.534704208374023,-2.0084803104400635,-4.349460124969482],"argmax":1,"top2_margin":0.3599908947944641,"near_tie":false},"graph":{"p":[0.20247286558151245,0.576193630695343,0.22133351862430573],"max_abs_dp":0.002594172954559326,"argmax_equal":true}}]},{"id":"tv4s_11","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":639,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0005","pair_id":"v3-legacy-transfer-deadline-0005","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0005","id":"contrastive/deadline/v3-legacy-transfer-deadline-0005/b","text_sha256":"e57e6c96bf0b18642eaa06dbe450c57cce84d436022bd5342a4316459ea74c30","row_sha256":"5790faa6466eab4bc42fe96342c98c67ad9bb1f7ce190a0e8b2bfe73411bb04d","group_id":"v3-legacy-transfer-deadline-0005","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"score","n_options":3,"row_len":99,"state_len":78,"decide":98,"opts":[88,93,97],"row_ids_sha256":"74b92f70bf27a24c4057e0b2cc80f2e7efee62ce2fee0278709fbf925a2eaf19","oracle":{"probs":[0.010673082433640957,0.49413636326789856,0.4951905906200409],"logits_raw":[-11.718484878540039,-2.4910755157470703,-2.485947847366333],"argmax":2,"top2_margin":0.001054227352142334,"near_tie":true},"graph":{"p":[0.01080369297415018,0.49389034509658813,0.4953059256076813],"max_abs_dp":0.0002460181713104248,"argmax_equal":true}}]},{"id":"tv4s_12","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":640,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0006","pair_id":"v3-legacy-transfer-deadline-0006","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0006","id":"contrastive/deadline/v3-legacy-transfer-deadline-0006/a","text_sha256":"d14936c609d1f5a21ffed3f0360912b15b4f17548707aa5563cd978a1c4aafb4","row_sha256":"dc10fe25f1999ebed57149c6c57398d043aa4429de09ed8ac2c67bebd111be68","group_id":"v3-legacy-transfer-deadline-0006","variant":"clean"}},"state_tokens":79,"questions":[{"q":0,"type":"score","n_options":3,"row_len":100,"state_len":79,"decide":99,"opts":[89,94,98],"row_ids_sha256":"f31884b7f6febd212b095580e9255a0626672295eb519293aa122e79a55bcef8","oracle":{"probs":[0.9516255855560303,0.031831104308366776,0.01654333434998989],"logits_raw":[3.4105911254882812,-4.764511585235596,-6.339176177978516],"argmax":0,"top2_margin":0.919794499874115,"near_tie":false},"graph":{"p":[0.9491759538650513,0.03358937427401543,0.01723468489944935],"max_abs_dp":0.002449631690979004,"argmax_equal":true}}]},{"id":"tv4s_13","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":641,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0006","pair_id":"v3-legacy-transfer-deadline-0006","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0006","id":"contrastive/deadline/v3-legacy-transfer-deadline-0006/b","text_sha256":"7b3c28f53dad88a83f61c0933cb56bfdfb3a40824db6e42baa3a7fae324798fe","row_sha256":"78a2493317239cef828d33d92f4b5a51068308871013b4dd794036288e4b0817","group_id":"v3-legacy-transfer-deadline-0006","variant":"clean"}},"state_tokens":79,"questions":[{"q":0,"type":"score","n_options":3,"row_len":100,"state_len":79,"decide":99,"opts":[89,94,98],"row_ids_sha256":"d90092ebd7c62691e2e47840b2f73e4ecce7790c1062a9ab20627a05ae9f7b0c","oracle":{"probs":[0.010938792489469051,0.9775881767272949,0.011472976766526699],"logits_raw":[-8.507104873657227,2.3027312755584717,-8.392387390136719],"argmax":1,"top2_margin":0.9661151766777039,"near_tie":false},"graph":{"p":[0.010899138636887074,0.9775107502937317,0.011590085923671722],"max_abs_dp":0.00011710915714502335,"argmax_equal":true}}]},{"id":"tv4s_14","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":642,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0007","pair_id":"v3-legacy-transfer-deadline-0007","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0007","id":"contrastive/deadline/v3-legacy-transfer-deadline-0007/a","text_sha256":"a8c60ecb11984dd3e71ff10ccfae662e1a0eb24e06718bc4d7e2b931cbc2079d","row_sha256":"4f170ff0f24194bfedc3d20274df3493d0e265d5405d6c6152e3aa9a3277a8b1","group_id":"v3-legacy-transfer-deadline-0007","variant":"clean"}},"state_tokens":77,"questions":[{"q":0,"type":"score","n_options":3,"row_len":98,"state_len":77,"decide":97,"opts":[87,92,96],"row_ids_sha256":"e3edfb2d1d19980ba81345bb4b9ad1ac5752bc70ef1b2bf8e124f27bc23310da","oracle":{"probs":[0.20544451475143433,0.7468587160110474,0.0476967953145504],"logits_raw":[-4.317201614379883,-1.2117128372192383,-7.830784797668457],"argmax":1,"top2_margin":0.541414201259613,"near_tie":false},"graph":{"p":[0.20439834892749786,0.7478423714637756,0.04775925725698471],"max_abs_dp":0.0010461658239364624,"argmax_equal":true}}]},{"id":"tv4s_15","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":643,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0007","pair_id":"v3-legacy-transfer-deadline-0007","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0007","id":"contrastive/deadline/v3-legacy-transfer-deadline-0007/b","text_sha256":"13aa619399e3927530b663fc9d26e409c88b44a8c07a94eae8e93c06921e01f3","row_sha256":"9ddea24990c6956702d0a1f6fbb209c8f023cb3ad53c10fc742cb657b7511dca","group_id":"v3-legacy-transfer-deadline-0007","variant":"clean"}},"state_tokens":77,"questions":[{"q":0,"type":"score","n_options":3,"row_len":98,"state_len":77,"decide":97,"opts":[87,92,96],"row_ids_sha256":"f1dd3d2804bb9dafffccdd81092536d4ada85bc5b6831c61532d388efab45e72","oracle":{"probs":[0.02748376876115799,0.9624660611152649,0.010050070472061634],"logits_raw":[-6.699054718017578,1.8566257953643799,-9.119579315185547],"argmax":1,"top2_margin":0.9349822998046875,"near_tie":false},"graph":{"p":[0.028349002823233604,0.9615017771720886,0.010149295441806316],"max_abs_dp":0.0009642839431762695,"argmax_equal":true}}]},{"id":"tv4s_16","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":644,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0008","pair_id":"v3-legacy-transfer-deadline-0008","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0008","id":"contrastive/deadline/v3-legacy-transfer-deadline-0008/a","text_sha256":"c5742086c74a5a53a5f11ed184c3d611193196f6b8cb4ac7c3b66e84329f065a","row_sha256":"94bceb24c84ef2def72f044a3a45915a324ddac91d3f53ae01d9ac46999cb78e","group_id":"v3-legacy-transfer-deadline-0008","variant":"clean"}},"state_tokens":79,"questions":[{"q":0,"type":"score","n_options":3,"row_len":100,"state_len":79,"decide":99,"opts":[89,94,98],"row_ids_sha256":"c5eea875535610e50cfcec2129f573b66b26042464be5e995f813749443eddc2","oracle":{"probs":[0.6356032490730286,0.33090922236442566,0.03348754718899727],"logits_raw":[-1.3908350467681885,-2.961337089538574,-8.47280502319336],"argmax":0,"top2_margin":0.3046940267086029,"near_tie":false},"graph":{"p":[0.6287334561347961,0.3368675112724304,0.03439910709857941],"max_abs_dp":0.006869792938232422,"argmax_equal":true}}]},{"id":"tv4s_17","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":645,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0008","pair_id":"v3-legacy-transfer-deadline-0008","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0008","id":"contrastive/deadline/v3-legacy-transfer-deadline-0008/b","text_sha256":"b3dd1c44cae9495cd28ff312f20d319e07fabb962bc61138dade8c7adf524e53","row_sha256":"d15eca907b2a8c547bb96b02c92209e884685393cc0772fb5f0b73f26d6295bb","group_id":"v3-legacy-transfer-deadline-0008","variant":"clean"}},"state_tokens":79,"questions":[{"q":0,"type":"score","n_options":3,"row_len":100,"state_len":79,"decide":99,"opts":[89,94,98],"row_ids_sha256":"e8a5b879fafb0664c5d800d920fdf3c3e880315a126c51b383fea8939f0766f3","oracle":{"probs":[0.019772840663790703,0.8321055769920349,0.14812146127223969],"logits_raw":[-10.237015724182129,-1.2392308712005615,-5.3918962478637695],"argmax":1,"top2_margin":0.683984100818634,"near_tie":false},"graph":{"p":[0.020363489165902138,0.8302093744277954,0.14942717552185059],"max_abs_dp":0.001896202564239502,"argmax_equal":true}}]},{"id":"tv4s_18","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":646,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0009","pair_id":"v3-legacy-transfer-deadline-0009","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0009","id":"contrastive/deadline/v3-legacy-transfer-deadline-0009/a","text_sha256":"089647dcec080abecf6cfadc5cd00e0e6c03fcb94c92fb12e512539825233c2e","row_sha256":"9b334f8aef425a2b55f28ce20e2ddda7eb742cdc6d8d03cb8c06e223e9ffdbf8","group_id":"v3-legacy-transfer-deadline-0009","variant":"clean"}},"state_tokens":77,"questions":[{"q":0,"type":"score","n_options":3,"row_len":98,"state_len":77,"decide":97,"opts":[87,92,96],"row_ids_sha256":"5017c61aaca7ed323784b1bd48e8e066a1f5bc2da7d7d9c3b9cb9f49c05f529f","oracle":{"probs":[0.9818958044052124,0.01054573804140091,0.007558515761047602],"logits_raw":[3.033348798751831,-7.875113010406494,-8.676440238952637],"argmax":0,"top2_margin":0.9713500738143921,"near_tie":false},"graph":{"p":[0.9813812971115112,0.010825988836586475,0.007792649324983358],"max_abs_dp":0.0005145072937011719,"argmax_equal":true}}]},{"id":"tv4s_19","source":"transfer_v4_dev_score","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":647,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0009","pair_id":"v3-legacy-transfer-deadline-0009","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0009","id":"contrastive/deadline/v3-legacy-transfer-deadline-0009/b","text_sha256":"f332427ac84752956e94a0a0ae03443cf06bab5a255a0a240071bbce45da78fe","row_sha256":"e13cca24963a4dba6e0ef1311192f27e8a1d9f016ab4a51493be86a18c00ecb6","group_id":"v3-legacy-transfer-deadline-0009","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"score","n_options":3,"row_len":99,"state_len":78,"decide":98,"opts":[88,93,97],"row_ids_sha256":"544463d92db0c0d38ae98661ea607fde888b4bd3e3378547061651bbcfe69cfc","oracle":{"probs":[0.012992898933589458,0.9811990261077881,0.005808086134493351],"logits_raw":[-7.316352844238281,3.0883021354675293,-9.253589630126953],"argmax":1,"top2_margin":0.9682061076164246,"near_tie":false},"graph":{"p":[0.012712083756923676,0.9814167022705078,0.005871281959116459],"max_abs_dp":0.00028081517666578293,"argmax_equal":true}}]},{"id":"semif_a3f18f3a63d45345942b","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The optician ordered replacement lenses. The workshop confirms they have not yet been fitted to the customer's glasses.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the replacement lenses have been fitted.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=2fdaa8a61e6e989dd866","provenance":{"family":"evidence_interpretation","variant":"original","group_id":"2fdaa8a61e6e989dd866","label":2,"option_order":["supported","insufficient","contradicted"]},"state_tokens":23,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","insufficient","contradicted"],"gold":"contradicted","row_len":68,"state_len":23,"decide":67,"opts":[44,55,66],"row_ids_sha256":"17bf5e66701025f931d34d89877a207173a171f964d532c65ba417482741011c","oracle":{"probs":[0.0011697952868416905,0.011719745583832264,0.9871104955673218],"logits_raw":[-10.918830871582031,-5.374218940734863,5.293021202087402],"argmax":2,"top2_margin":0.9753907322883606,"near_tie":false},"graph":{"p":[0.0012049248907715082,0.012159205973148346,0.9866358637809753],"max_abs_dp":0.00047463178634643555,"argmax_equal":true}}]},{"id":"semif_f40beba9088c8db8bbd6","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The optician ordered replacement lenses. The workshop confirms they have not yet been fitted to the customer's glasses.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: replacement lenses were ordered.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=2fdaa8a61e6e989dd866","provenance":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"2fdaa8a61e6e989dd866","label":0,"option_order":["supported","insufficient","contradicted"]},"state_tokens":23,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","insufficient","contradicted"],"gold":"supported","row_len":66,"state_len":23,"decide":65,"opts":[42,53,64],"row_ids_sha256":"9ac2d2f95bf70bcce2a379bc31d5374eb423fd974decadbe926b68a26acfc790","oracle":{"probs":[0.9756666421890259,0.02179759182035923,0.0025357501581311226],"logits_raw":[4.004541873931885,-5.141627311706543,-10.317787170410156],"argmax":0,"top2_margin":0.9538690447807312,"near_tie":false},"graph":{"p":[0.9756499528884888,0.02179216779768467,0.0025578925851732492],"max_abs_dp":2.2142427042126656e-05,"argmax_equal":true}}]},{"id":"semif_0b43ea8e24e74d621f1b","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The workshop confirms it fitted the replacement lenses to the customer's glasses.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the replacement lenses have been fitted.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=2fdaa8a61e6e989dd866","provenance":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"2fdaa8a61e6e989dd866","label":1,"option_order":["insufficient","supported","contradicted"]},"state_tokens":15,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","supported","contradicted"],"gold":"supported","row_len":60,"state_len":15,"decide":59,"opts":[38,47,58],"row_ids_sha256":"1c3456b4a3fefa381c1d7707c4aafd85e3ab9499c5f2684568a2d2cb95b868ca","oracle":{"probs":[0.015394570305943489,0.9835087656974792,0.0010967107955366373],"logits_raw":[-4.135310173034668,5.866908550262451,-10.491371154785156],"argmax":1,"top2_margin":0.9681141972541809,"near_tie":false},"graph":{"p":[0.015693234279751778,0.9831932783126831,0.0011134188389405608],"max_abs_dp":0.0003154873847961426,"argmax_equal":true}}]},{"id":"semif_942ec65dac0bfb9874f9","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The optician records a glasses prescription but no order or fitting status.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the replacement lenses have been fitted.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=2fdaa8a61e6e989dd866","provenance":{"family":"evidence_interpretation","variant":"missing","group_id":"2fdaa8a61e6e989dd866","label":0,"option_order":["insufficient","supported","contradicted"]},"state_tokens":15,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","supported","contradicted"],"gold":"insufficient","row_len":60,"state_len":15,"decide":59,"opts":[38,47,58],"row_ids_sha256":"b8a9418cbbf9ef22ca16e37f0968427ff76d0427c0eb5f0f943c731cfe4a452e","oracle":{"probs":[0.37458881735801697,0.004189872182905674,0.6212213635444641],"logits_raw":[-0.43576693534851074,-11.246530532836914,0.7813540697097778],"argmax":2,"top2_margin":0.24663254618644714,"near_tie":false},"graph":{"p":[0.38154545426368713,0.004316531587392092,0.6141380667686462],"max_abs_dp":0.007083296775817871,"argmax_equal":true}}]},{"id":"semif_e0c140e9222d1bce5647","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The guest writes, \"The listing says towels are supplied. I agree: fresh towels were provided during my stay.\"","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the guest endorses the listing's towel statement.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=original group=f4bea3d3bcc01056740d","provenance":{"family":"evidence_interpretation","variant":"original","group_id":"f4bea3d3bcc01056740d","label":0,"option_order":["supported","contradicted","insufficient"]},"state_tokens":24,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","contradicted","insufficient"],"gold":"supported","row_len":72,"state_len":24,"decide":71,"opts":[48,59,70],"row_ids_sha256":"4120aba13df875e74500d168b5830c75fecf5582cccbb440804c4ba174a7f0cc","oracle":{"probs":[0.993440568447113,0.0009784047724679112,0.005581025034189224],"logits_raw":[5.362653732299805,-11.294445037841797,-7.105020999908447],"argmax":0,"top2_margin":0.9878595471382141,"near_tie":false},"graph":{"p":[0.9933111667633057,0.0010010239202529192,0.005687831435352564],"max_abs_dp":0.00012940168380737305,"argmax_equal":true}}]},{"id":"semif_8e8c3804a3c15ebb31e3","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The guest writes, \"The listing says towels are supplied. I agree: fresh towels were provided during my stay.\"","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the guest disputes the towel statement.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=criterion_reversal group=f4bea3d3bcc01056740d","provenance":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"f4bea3d3bcc01056740d","label":2,"option_order":["insufficient","supported","contradicted"]},"state_tokens":24,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","supported","contradicted"],"gold":"contradicted","row_len":69,"state_len":24,"decide":68,"opts":[47,56,67],"row_ids_sha256":"23101a9e88ac575b49ec5c546eef9f5d54f6f1e417d659c605f758371dee413b","oracle":{"probs":[0.24986553192138672,0.720701277256012,0.029433198273181915],"logits_raw":[-4.074680328369141,-1.5259474515914917,-9.220739364624023],"argmax":1,"top2_margin":0.47083574533462524,"near_tie":false},"graph":{"p":[0.24901129305362701,0.7214826941490173,0.029505962505936623],"max_abs_dp":0.0008542388677597046,"argmax_equal":true}}]},{"id":"semif_56c419b12cd37cbe790c","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The guest writes, \"The listing says towels are supplied, but none were provided. I dispute that statement.\"","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the guest endorses the listing's towel statement.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=evidence_change group=f4bea3d3bcc01056740d","provenance":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"f4bea3d3bcc01056740d","label":0,"option_order":["contradicted","insufficient","supported"]},"state_tokens":23,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["contradicted","insufficient","supported"],"gold":"contradicted","row_len":71,"state_len":23,"decide":70,"opts":[49,60,69],"row_ids_sha256":"0e611a67204a2714e176ae0bd91ce8565f2c0907c339ff884b12b694fcf1b56b","oracle":{"probs":[0.9407789707183838,0.054370611906051636,0.004850431345403194],"logits_raw":[3.072266101837158,-3.7871041297912598,-9.601940155029297],"argmax":0,"top2_margin":0.8864083290100098,"near_tie":false},"graph":{"p":[0.9400686025619507,0.05491799861192703,0.005013421177864075],"max_abs_dp":0.0007103681564331055,"argmax_equal":true}}]},{"id":"semif_75b470586834df8c568d","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The guest forwards the listing but provides no comment about towels or the stay.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the guest endorses the listing's towel statement.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=f4bea3d3bcc01056740d","provenance":{"family":"evidence_interpretation","variant":"missing","group_id":"f4bea3d3bcc01056740d","label":2,"option_order":["supported","contradicted","insufficient"]},"state_tokens":16,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","contradicted","insufficient"],"gold":"insufficient","row_len":64,"state_len":16,"decide":63,"opts":[40,51,62],"row_ids_sha256":"671f131d1aeab3d348429df2c21df9683e8f24cd2f1feee78ae635090c2cc640","oracle":{"probs":[0.014437650330364704,0.07238748669624329,0.9131748676300049],"logits_raw":[-9.959651947021484,-6.08063268661499,0.018449127674102783],"argmax":2,"top2_margin":0.840787410736084,"near_tie":false},"graph":{"p":[0.014702285639941692,0.07413926720619202,0.9111583828926086],"max_abs_dp":0.0020164847373962402,"argmax_equal":true}}]},{"id":"semif_834f3b3e10cc61b33d25","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The technician says terminal Cedar is online. Terminal Maple is explicitly offline.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: terminal Maple is online.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=b51b300da2936decc6b7","provenance":{"family":"evidence_interpretation","variant":"original","group_id":"b51b300da2936decc6b7","label":2,"option_order":["supported","insufficient","contradicted"]},"state_tokens":15,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","insufficient","contradicted"],"gold":"contradicted","row_len":58,"state_len":15,"decide":57,"opts":[34,45,56],"row_ids_sha256":"05814f9790669896328de0c786a68af062dfeefe7b51635f3618a2e54ee923c8","oracle":{"probs":[0.0005243295454420149,0.001624581404030323,0.9978511333465576],"logits_raw":[-12.050225257873535,-9.329258918762207,6.118432998657227],"argmax":2,"top2_margin":0.9962265491485596,"near_tie":false},"graph":{"p":[0.0005374442553147674,0.001702558365650475,0.9977599382400513],"max_abs_dp":9.119510650634766e-05,"argmax_equal":true}}]},{"id":"semif_76c7a79b1ff2363a60c5","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The technician says terminal Cedar is online. Terminal Maple is explicitly offline.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: terminal Cedar is online.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=b51b300da2936decc6b7","provenance":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"b51b300da2936decc6b7","label":0,"option_order":["supported","insufficient","contradicted"]},"state_tokens":15,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","insufficient","contradicted"],"gold":"supported","row_len":58,"state_len":15,"decide":57,"opts":[34,45,56],"row_ids_sha256":"226a76eef107a0adbf69f33fed667871fada2a77dfcf68017296bf57c7e9a8dc","oracle":{"probs":[0.9929383397102356,0.006191228050738573,0.0008704558713361621],"logits_raw":[5.479414939880371,-6.737388610839844,-11.457748413085938],"argmax":0,"top2_margin":0.9867470860481262,"near_tie":false},"graph":{"p":[0.9928330183029175,0.00627864757552743,0.0008884199196472764],"max_abs_dp":0.00010532140731811523,"argmax_equal":true}}]},{"id":"semif_36c71bb649b7d5eccb1f","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The technician confirms terminal Maple is online and terminal Cedar is offline.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: terminal Maple is online.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=b51b300da2936decc6b7","provenance":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"b51b300da2936decc6b7","label":0,"option_order":["supported","insufficient","contradicted"]},"state_tokens":14,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","insufficient","contradicted"],"gold":"supported","row_len":57,"state_len":14,"decide":56,"opts":[33,44,55],"row_ids_sha256":"c4ea5bb865d7e4c095484e9e234737d2fcf479f3b8b7f30fdee63159c318b598","oracle":{"probs":[0.9980860948562622,0.001179049606435001,0.000734867702703923],"logits_raw":[6.0828962326049805,-10.136600494384766,-11.274117469787598],"argmax":0,"top2_margin":0.9969070553779602,"near_tie":false},"graph":{"p":[0.9980265498161316,0.0012209431733936071,0.0007525208638980985],"max_abs_dp":5.9545040130615234e-05,"argmax_equal":true}}]},{"id":"semif_eac5121a4a1bf391a08c","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The technician lists both terminals but omits their connection status.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: terminal Maple is online.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=b51b300da2936decc6b7","provenance":{"family":"evidence_interpretation","variant":"missing","group_id":"b51b300da2936decc6b7","label":1,"option_order":["contradicted","insufficient","supported"]},"state_tokens":13,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["contradicted","insufficient","supported"],"gold":"insufficient","row_len":56,"state_len":13,"decide":55,"opts":[34,45,54],"row_ids_sha256":"6d13bb8e9122f447f0d922914af4ac7e2bc0f2748a192e4c3bd8fc33f79d4676","oracle":{"probs":[0.0067293462343513966,0.9894513487815857,0.0038192339707165956],"logits_raw":[-9.186280250549316,2.8215279579162598,-10.549134254455566],"argmax":1,"top2_margin":0.9827219843864441,"near_tie":false},"graph":{"p":[0.0068871271796524525,0.9891926646232605,0.003920143935829401],"max_abs_dp":0.0002586841583251953,"argmax_equal":true}}]},{"id":"semif_44574c44e243ee1b2f52","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The catalogue originally dated the letter to June. Its author issued a correction stating July was correct and June was mistaken.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the corrected date is in July.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=original group=e4cfd694c80e08d3e6a5","provenance":{"family":"evidence_interpretation","variant":"original","group_id":"e4cfd694c80e08d3e6a5","label":2,"option_order":["insufficient","contradicted","supported"]},"state_tokens":24,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","contradicted","supported"],"gold":"supported","row_len":69,"state_len":24,"decide":68,"opts":[47,58,67],"row_ids_sha256":"90e0a1671778f120fab73bf183fbc4aac94bf2c35a12fb25c5891e09de0ed79b","oracle":{"probs":[0.015547748655080795,0.003323551267385483,0.9811286926269531],"logits_raw":[-4.190665245056152,-7.90291690826416,5.7819013595581055],"argmax":2,"top2_margin":0.965580940246582,"near_tie":false},"graph":{"p":[0.015737224370241165,0.0033339140936732292,0.9809288382530212],"max_abs_dp":0.00019985437393188477,"argmax_equal":true}}]},{"id":"semif_2608370d01783735c3ea","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The catalogue originally dated the letter to June. Its author issued a correction stating July was correct and June was mistaken.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the corrected date is in June.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=criterion_reversal group=e4cfd694c80e08d3e6a5","provenance":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"e4cfd694c80e08d3e6a5","label":2,"option_order":["supported","insufficient","contradicted"]},"state_tokens":24,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","insufficient","contradicted"],"gold":"contradicted","row_len":69,"state_len":24,"decide":68,"opts":[45,56,67],"row_ids_sha256":"82bc638397f08b41cd5841301878dce5ea53b3931b6306a9ed4ac69eb30cabbd","oracle":{"probs":[0.0025537502951920033,0.036050379276275635,0.9613959193229675],"logits_raw":[-11.11764907836914,-4.747982025146484,3.1522083282470703],"argmax":2,"top2_margin":0.9253455400466919,"near_tie":false},"graph":{"p":[0.002597929909825325,0.0366380400955677,0.9607641100883484],"max_abs_dp":0.0006318092346191406,"argmax_equal":true}}]},{"id":"semif_ec94ef909f7918695ab3","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The catalogue author corrects the date to June and expressly rejects the earlier July date.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the corrected date is in July.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=evidence_change group=e4cfd694c80e08d3e6a5","provenance":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"e4cfd694c80e08d3e6a5","label":2,"option_order":["supported","insufficient","contradicted"]},"state_tokens":18,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","insufficient","contradicted"],"gold":"contradicted","row_len":63,"state_len":18,"decide":62,"opts":[39,50,61],"row_ids_sha256":"c549fe6be81e9e5f01166710827f69a837146c12b0f35324e4c01f8ed9c7facb","oracle":{"probs":[0.0006749563035555184,0.009884506464004517,0.9894405007362366],"logits_raw":[-12.250988006591797,-5.792967796325684,5.28971004486084],"argmax":2,"top2_margin":0.9795559644699097,"near_tie":false},"graph":{"p":[0.0006974409334361553,0.010247329249978065,0.9890552759170532],"max_abs_dp":0.0003852248191833496,"argmax_equal":true}}]},{"id":"semif_03d919fd0e9b3a7cd1e6","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The catalogue mentions a date correction but gives no corrected date.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the corrected date is in July.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=e4cfd694c80e08d3e6a5","provenance":{"family":"evidence_interpretation","variant":"missing","group_id":"e4cfd694c80e08d3e6a5","label":0,"option_order":["insufficient","supported","contradicted"]},"state_tokens":13,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","supported","contradicted"],"gold":"insufficient","row_len":58,"state_len":13,"decide":57,"opts":[36,45,56],"row_ids_sha256":"b297e19094d11308fcb1a876be73dd9805664c95ad7862e74b6299afe1621c47","oracle":{"probs":[0.9864422082901001,0.00597176980227232,0.007586044259369373],"logits_raw":[0.691011905670166,-11.596833229064941,-11.02114486694336],"argmax":0,"top2_margin":0.9788561463356018,"near_tie":false},"graph":{"p":[0.9860509037971497,0.006124324630945921,0.007824835367500782],"max_abs_dp":0.00039130449295043945,"argmax_equal":true}}]},{"id":"semif_86a15618ba4cb90a307e","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The intern attached instructions for weighing the parcel and explicitly says no weighing has been performed.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the parcel has been weighed.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=20460d4ae0db46091e84","provenance":{"family":"evidence_interpretation","variant":"original","group_id":"20460d4ae0db46091e84","label":0,"option_order":["contradicted","insufficient","supported"]},"state_tokens":18,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["contradicted","insufficient","supported"],"gold":"contradicted","row_len":62,"state_len":18,"decide":61,"opts":[40,51,60],"row_ids_sha256":"3f1f6002be9c9228170ed309510d9bf7ac0a660244512aba8e3f9ffbb142285c","oracle":{"probs":[0.9390813708305359,0.05890589579939842,0.002012806013226509],"logits_raw":[3.062711238861084,-3.5995473861694336,-11.723361015319824],"argmax":0,"top2_margin":0.8801754713058472,"near_tie":false},"graph":{"p":[0.9387422204017639,0.05920019745826721,0.002057617763057351],"max_abs_dp":0.00033915042877197266,"argmax_equal":true}}]},{"id":"semif_3dd9c3ac98f8ddf7bdd3","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The intern attached instructions for weighing the parcel and explicitly says no weighing has been performed.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the document contains weighing instructions.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=20460d4ae0db46091e84","provenance":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"20460d4ae0db46091e84","label":1,"option_order":["contradicted","supported","insufficient"]},"state_tokens":18,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["contradicted","supported","insufficient"],"gold":"supported","row_len":62,"state_len":18,"decide":61,"opts":[40,49,60],"row_ids_sha256":"da21b7bb6beacfb9d6ce839bfb242640acb4aa402f2e4411c1cc3d002b65abdd","oracle":{"probs":[0.01573110744357109,0.8997976183891296,0.08447127044200897],"logits_raw":[-7.547163963317871,2.18898868560791,-3.5031442642211914],"argmax":1,"top2_margin":0.8153263330459595,"near_tie":false},"graph":{"p":[0.015810325741767883,0.9005342721939087,0.08365552127361298],"max_abs_dp":0.0008157491683959961,"argmax_equal":true}}]},{"id":"semif_36273dd4777876a26338","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The intern performed the weighing and recorded the parcel's measured weight.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the parcel has been weighed.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=20460d4ae0db46091e84","provenance":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"20460d4ae0db46091e84","label":1,"option_order":["contradicted","supported","insufficient"]},"state_tokens":14,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["contradicted","supported","insufficient"],"gold":"supported","row_len":58,"state_len":14,"decide":57,"opts":[36,45,56],"row_ids_sha256":"87134dc286b32c3adcfcc5bf69076b54b71208b24c97a36611723d16ffd85b50","oracle":{"probs":[0.0028426237404346466,0.9861313104629517,0.011025992222130299],"logits_raw":[-8.987785339355469,5.085350036621094,-5.726317405700684],"argmax":1,"top2_margin":0.975105345249176,"near_tie":false},"graph":{"p":[0.002901294967159629,0.9859926104545593,0.011106076650321484],"max_abs_dp":0.00013870000839233398,"argmax_equal":true}}]},{"id":"semif_07ee70cfe4b01330ce89","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The intern supplies a parcel identifier but no procedure or measurement information.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the parcel has been weighed.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=20460d4ae0db46091e84","provenance":{"family":"evidence_interpretation","variant":"missing","group_id":"20460d4ae0db46091e84","label":2,"option_order":["supported","contradicted","insufficient"]},"state_tokens":14,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","contradicted","insufficient"],"gold":"insufficient","row_len":58,"state_len":14,"decide":57,"opts":[34,45,56],"row_ids_sha256":"0b87c9f44988219e7077ce4358fa0aedba4d164deb366b4e6f843a8da3ce2029","oracle":{"probs":[0.008546719327569008,0.044188953936100006,0.9472643136978149],"logits_raw":[-10.231552124023438,-6.278587341308594,1.0962045192718506],"argmax":2,"top2_margin":0.9030753374099731,"near_tie":false},"graph":{"p":[0.00865109357982874,0.045777905732393265,0.9455710649490356],"max_abs_dp":0.0016932487487792969,"argmax_equal":true}}]},{"id":"semif_51ceab3ce622c7c759f1","source":"semif_authored144","request":{"model":"kev-0.8b","state":"During a puppet show, a character announces the discovery of a dragon egg. The production note states this is fictional and no real egg was discovered.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the egg discovery was fictional.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=original group=51b91ba31984c5939c1c","provenance":{"family":"evidence_interpretation","variant":"original","group_id":"51b91ba31984c5939c1c","label":2,"option_order":["contradicted","insufficient","supported"]},"state_tokens":30,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["contradicted","insufficient","supported"],"gold":"supported","row_len":74,"state_len":30,"decide":73,"opts":[52,63,72],"row_ids_sha256":"4475f9e08c59e7bfd2248f902d27567c2b94be1b662a948f48cb1957b3d8af32","oracle":{"probs":[0.005832259077578783,0.010638633742928505,0.9835290908813477],"logits_raw":[-6.411049842834473,-4.964804172515869,5.926555156707764],"argmax":2,"top2_margin":0.9728904366493225,"near_tie":false},"graph":{"p":[0.005888204090297222,0.010654658079147339,0.9834571480751038],"max_abs_dp":7.194280624389648e-05,"argmax_equal":true}}]},{"id":"semif_b8091be929b1e489fcc4","source":"semif_authored144","request":{"model":"kev-0.8b","state":"During a puppet show, a character announces the discovery of a dragon egg. The production note states this is fictional and no real egg was discovered.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: a real egg was discovered.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=criterion_reversal group=51b91ba31984c5939c1c","provenance":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"51b91ba31984c5939c1c","label":1,"option_order":["supported","contradicted","insufficient"]},"state_tokens":30,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","contradicted","insufficient"],"gold":"contradicted","row_len":74,"state_len":30,"decide":73,"opts":[50,61,72],"row_ids_sha256":"4f63c2cead007888adca0e1d0307df2de478c6839a18e0a133842e0b1a4d1b75","oracle":{"probs":[0.001539973309263587,0.982919454574585,0.015540611930191517],"logits_raw":[-10.355524063110352,5.1845808029174805,-4.793478012084961],"argmax":1,"top2_margin":0.9673788547515869,"near_tie":false},"graph":{"p":[0.0015720383962616324,0.9827153086662292,0.015712672844529152],"max_abs_dp":0.0002041459083557129,"argmax_equal":true}}]},{"id":"semif_c89b86e3f55123bfa506","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The note describes a real bird egg discovered outside a performance and expressly says the discovery was not fictional.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the egg discovery was fictional.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=evidence_change group=51b91ba31984c5939c1c","provenance":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"51b91ba31984c5939c1c","label":1,"option_order":["supported","contradicted","insufficient"]},"state_tokens":21,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","contradicted","insufficient"],"gold":"contradicted","row_len":65,"state_len":21,"decide":64,"opts":[41,52,63],"row_ids_sha256":"81db999f72a16149476b41b879dcec04d950f7a132eac2fe38ec73a0cb8b91ca","oracle":{"probs":[0.0003377518733032048,0.9933098554611206,0.0063524614088237286],"logits_raw":[-13.040689468383789,6.175195693969727,-5.980650901794434],"argmax":1,"top2_margin":0.9869573712348938,"near_tie":false},"graph":{"p":[0.0003449833020567894,0.9931782484054565,0.006476708687841892],"max_abs_dp":0.0001316070556640625,"argmax_equal":true}}]},{"id":"semif_3b0958cabc339005cf66","source":"semif_authored144","request":{"model":"kev-0.8b","state":"A note mentions an egg without describing a discovery or whether the account is fictional.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the egg discovery was fictional.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=51b91ba31984c5939c1c","provenance":{"family":"evidence_interpretation","variant":"missing","group_id":"51b91ba31984c5939c1c","label":0,"option_order":["insufficient","supported","contradicted"]},"state_tokens":17,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","supported","contradicted"],"gold":"insufficient","row_len":61,"state_len":17,"decide":60,"opts":[39,48,59],"row_ids_sha256":"0ef318968bd4ea124a90108b1902306cf951651c4096c773f635432d5e6bcc67","oracle":{"probs":[0.9483416080474854,0.020708296447992325,0.030950123444199562],"logits_raw":[0.2645770311355591,-8.936592102050781,-7.969738960266113],"argmax":0,"top2_margin":0.9173914790153503,"near_tie":false},"graph":{"p":[0.9479343891143799,0.02086205594241619,0.031203502789139748],"max_abs_dp":0.00040721893310546875,"argmax_equal":true}}]},{"id":"semif_f2b4ec4930322fe45118","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The architect permits reproducing a plan in an internal training booklet and expressly forbids selling copies.","questions":{"decision":{"type":"choice","instructions":"Sell copies only with the architect's sales permission. May copies be sold?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=770c1607f0a11f88f81e","provenance":{"family":"rule_application","variant":"original","group_id":"770c1607f0a11f88f81e","label":1,"option_order":["insufficient","prohibited","permitted"]},"state_tokens":20,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","prohibited","permitted"],"gold":"prohibited","row_len":76,"state_len":20,"decide":75,"opts":[52,63,74],"row_ids_sha256":"88b2ec7f10084d461aa5563b9ec3e8c495e6c00c41111e40e9260689508dc091","oracle":{"probs":[0.05820902809500694,0.9380443692207336,0.0037466599605977535],"logits_raw":[-2.2903285026550293,4.397905349731445,-8.890546798706055],"argmax":1,"top2_margin":0.8798353672027588,"near_tie":false},"graph":{"p":[0.061896298080682755,0.9341655969619751,0.003938065376132727],"max_abs_dp":0.003878772258758545,"argmax_equal":true}}]},{"id":"semif_61ac20b07a5189231e15","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The architect permits reproducing a plan in an internal training booklet and expressly forbids selling copies.","questions":{"decision":{"type":"choice","instructions":"Internal training reproduction is allowed when the architect permits that use. May the plan appear in the training booklet?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=770c1607f0a11f88f81e","provenance":{"family":"rule_application","variant":"criterion_reversal","group_id":"770c1607f0a11f88f81e","label":1,"option_order":["prohibited","permitted","insufficient"]},"state_tokens":20,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","permitted","insufficient"],"gold":"permitted","row_len":82,"state_len":20,"decide":81,"opts":[52,63,80],"row_ids_sha256":"f560524c4dbcaad3aba2e3bd16889fe177bb376545c5dc3bbd559e16118cf118","oracle":{"probs":[0.003846115432679653,0.9858646392822266,0.01028919592499733],"logits_raw":[-8.37699031829834,4.968058109283447,-6.009364128112793],"argmax":1,"top2_margin":0.9755754470825195,"near_tie":false},"graph":{"p":[0.003855111077427864,0.9857909679412842,0.0103539377450943],"max_abs_dp":7.367134094238281e-05,"argmax_equal":true}}]},{"id":"semif_4b3d794c004566610181","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The architect expressly permits selling copies of the plan.","questions":{"decision":{"type":"choice","instructions":"Sell copies only with the architect's sales permission. May copies be sold?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=770c1607f0a11f88f81e","provenance":{"family":"rule_application","variant":"evidence_change","group_id":"770c1607f0a11f88f81e","label":2,"option_order":["prohibited","insufficient","permitted"]},"state_tokens":11,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","insufficient","permitted"],"gold":"permitted","row_len":67,"state_len":11,"decide":66,"opts":[37,54,65],"row_ids_sha256":"d208ee9399b443fb5ccebb64b2078ee9bad643bcaf4e0f3e79d8be4528f2b3f1","oracle":{"probs":[0.003918743692338467,0.011709721758961678,0.9843714833259583],"logits_raw":[-8.078283309936523,-5.444504737854004,5.2181077003479],"argmax":2,"top2_margin":0.9726617336273193,"near_tie":false},"graph":{"p":[0.004019181709736586,0.011933539062738419,0.9840472340583801],"max_abs_dp":0.000324249267578125,"argmax_equal":true}}]},{"id":"semif_b7c4e62d07986286fce0","source":"semif_authored144","request":{"model":"kev-0.8b","state":"An architect's permission letter exists, but the uses it allows are omitted.","questions":{"decision":{"type":"choice","instructions":"Sell copies only with the architect's sales permission. May copies be sold?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=770c1607f0a11f88f81e","provenance":{"family":"rule_application","variant":"missing","group_id":"770c1607f0a11f88f81e","label":0,"option_order":["insufficient","permitted","prohibited"]},"state_tokens":16,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","permitted","prohibited"],"gold":"insufficient","row_len":72,"state_len":16,"decide":71,"opts":[48,59,70],"row_ids_sha256":"ba0065897718e5ed4a3628e067a80875f9355e1c74ec810c8b86c32b37b8de40","oracle":{"probs":[0.6423974633216858,0.11804641038179398,0.23955607414245605],"logits_raw":[-1.5473694801330566,-5.623529434204102,-3.9207444190979004],"argmax":0,"top2_margin":0.40284138917922974,"near_tie":false},"graph":{"p":[0.641628623008728,0.1176530048251152,0.24071842432022095],"max_abs_dp":0.0011623501777648926,"argmax_equal":true}}]},{"id":"semif_9473603b44af63beab79","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The guide completed first-aid training but not mountain-navigation training.","questions":{"decision":{"type":"choice","instructions":"The first-aid badge requires only first-aid training. May this guide receive it?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=original group=ffd081ada3bd9be9207a","provenance":{"family":"rule_application","variant":"original","group_id":"ffd081ada3bd9be9207a","label":0,"option_order":["permitted","insufficient","prohibited"]},"state_tokens":14,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["permitted","insufficient","prohibited"],"gold":"permitted","row_len":73,"state_len":14,"decide":72,"opts":[43,60,71],"row_ids_sha256":"1569962f1d8f138df667219dc160def789318ae9e137a5b599281e72f759c2ad","oracle":{"probs":[0.9359741806983948,0.05570608004927635,0.008319754153490067],"logits_raw":[2.355177879333496,-4.433488845825195,-9.008487701416016],"argmax":0,"top2_margin":0.8802680969238281,"near_tie":false},"graph":{"p":[0.9365100264549255,0.05512379854917526,0.008366109803318977],"max_abs_dp":0.0005822815001010895,"argmax_equal":true}}]},{"id":"semif_d41072dc35d8cd14c056","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The guide completed first-aid training but not mountain-navigation training.","questions":{"decision":{"type":"choice","instructions":"Under the alternative rule, the badge requires both first-aid and mountain-navigation training. May this guide receive it?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=criterion_reversal group=ffd081ada3bd9be9207a","provenance":{"family":"rule_application","variant":"criterion_reversal","group_id":"ffd081ada3bd9be9207a","label":1,"option_order":["permitted","prohibited","insufficient"]},"state_tokens":14,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["permitted","prohibited","insufficient"],"gold":"prohibited","row_len":78,"state_len":14,"decide":77,"opts":[48,59,76],"row_ids_sha256":"d9f2653af2247ace2439d19728ed2dc25e70cb61c32917a27bb2e42a0b35c78c","oracle":{"probs":[0.006637663580477238,0.9858901500701904,0.007472109515219927],"logits_raw":[-7.305585861206055,4.7265520095825195,-7.0206685066223145],"argmax":1,"top2_margin":0.9784180521965027,"near_tie":false},"graph":{"p":[0.00670287199318409,0.985903263092041,0.00739379832521081],"max_abs_dp":7.831119000911713e-05,"argmax_equal":true}}]},{"id":"semif_33d6e5da58f0fee2490d","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The guide completed mountain-navigation training but not first-aid training.","questions":{"decision":{"type":"choice","instructions":"The first-aid badge requires only first-aid training. May this guide receive it?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=evidence_change group=ffd081ada3bd9be9207a","provenance":{"family":"rule_application","variant":"evidence_change","group_id":"ffd081ada3bd9be9207a","label":0,"option_order":["prohibited","insufficient","permitted"]},"state_tokens":14,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","insufficient","permitted"],"gold":"prohibited","row_len":73,"state_len":14,"decide":72,"opts":[43,60,71],"row_ids_sha256":"f6991320da9615e554c783abe761f73f56224166f8ea67e9698937b2c1952abd","oracle":{"probs":[0.1812182068824768,0.47371405363082886,0.3450677692890167],"logits_raw":[-4.297391891479492,-1.985413670539856,-2.7478017807006836],"argmax":1,"top2_margin":0.12864628434181213,"near_tie":false},"graph":{"p":[0.18284161388874054,0.47493451833724976,0.34222379326820374],"max_abs_dp":0.0028439760208129883,"argmax_equal":true}}]},{"id":"semif_823bf736b92e9d8fa66d","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The guide registered for training, but completion information is not supplied.","questions":{"decision":{"type":"choice","instructions":"The first-aid badge requires only first-aid training. May this guide receive it?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=ffd081ada3bd9be9207a","provenance":{"family":"rule_application","variant":"missing","group_id":"ffd081ada3bd9be9207a","label":1,"option_order":["prohibited","insufficient","permitted"]},"state_tokens":14,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","insufficient","permitted"],"gold":"insufficient","row_len":73,"state_len":14,"decide":72,"opts":[43,60,71],"row_ids_sha256":"595720e71acdf858920973be9d5001d524278d448e42cf48d4896d49d06a14fa","oracle":{"probs":[0.27547454833984375,0.4465469717979431,0.2779785394668579],"logits_raw":[-3.404355049133301,-2.2421140670776367,-3.3825836181640625],"argmax":1,"top2_margin":0.1685684323310852,"near_tie":false},"graph":{"p":[0.27401086688041687,0.4477798342704773,0.27820929884910583],"max_abs_dp":0.0014636814594268799,"argmax_equal":true}}]},{"id":"semif_8a4c3de28be95c105ca1","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The designer approves making a sample badge but expressly withholds approval for mass production.","questions":{"decision":{"type":"choice","instructions":"Mass production is allowed only after mass-production approval. May it begin?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=2c3a0cb61f7bae78739e","provenance":{"family":"rule_application","variant":"original","group_id":"2c3a0cb61f7bae78739e","label":2,"option_order":["permitted","insufficient","prohibited"]},"state_tokens":17,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["permitted","insufficient","prohibited"],"gold":"prohibited","row_len":72,"state_len":17,"decide":71,"opts":[42,59,70],"row_ids_sha256":"f84ea863e9144c50ce1e380640098d8c95299261f4fc4a457ec55647377489b2","oracle":{"probs":[0.15828996896743774,0.1463388204574585,0.6953712105751038],"logits_raw":[-3.095747947692871,-3.284632682800293,0.46524739265441895],"argmax":2,"top2_margin":0.537081241607666,"near_tie":false},"graph":{"p":[0.15192189812660217,0.14003898203372955,0.7080391049385071],"max_abs_dp":0.01266789436340332,"argmax_equal":true}}]},{"id":"semif_ea7ec66a5e8eac886548","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The designer approves making a sample badge but expressly withholds approval for mass production.","questions":{"decision":{"type":"choice","instructions":"A sample may be made when sample production is approved. May the sample be made?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=2c3a0cb61f7bae78739e","provenance":{"family":"rule_application","variant":"criterion_reversal","group_id":"2c3a0cb61f7bae78739e","label":0,"option_order":["permitted","insufficient","prohibited"]},"state_tokens":17,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["permitted","insufficient","prohibited"],"gold":"permitted","row_len":75,"state_len":17,"decide":74,"opts":[45,62,73],"row_ids_sha256":"2c9aca54448803363d2c59a4f3e77e1fbc94681dca19ea9709d7f85f1bae0c8f","oracle":{"probs":[0.6645711064338684,0.3032211363315582,0.03220774605870247],"logits_raw":[-0.4980597496032715,-2.386037826538086,-7.7810163497924805],"argmax":0,"top2_margin":0.3613499701023102,"near_tie":false},"graph":{"p":[0.6667028069496155,0.30103054642677307,0.032266631722450256],"max_abs_dp":0.0021905899047851562,"argmax_equal":true}}]},{"id":"semif_25172cd6ddc49ab0ca70","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The designer explicitly approves mass production of the badge.","questions":{"decision":{"type":"choice","instructions":"Mass production is allowed only after mass-production approval. May it begin?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=2c3a0cb61f7bae78739e","provenance":{"family":"rule_application","variant":"evidence_change","group_id":"2c3a0cb61f7bae78739e","label":2,"option_order":["prohibited","insufficient","permitted"]},"state_tokens":11,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","insufficient","permitted"],"gold":"permitted","row_len":66,"state_len":11,"decide":65,"opts":[36,53,64],"row_ids_sha256":"9c559b7cdc2ffda081348c2fbf604bae35203020c98efabacb865997324cdf1d","oracle":{"probs":[0.010390074923634529,0.0187065489590168,0.9709033370018005],"logits_raw":[-7.096902370452881,-5.6820902824401855,3.820250988006592],"argmax":2,"top2_margin":0.9521967768669128,"near_tie":false},"graph":{"p":[0.010700552724301815,0.01896790787577629,0.9703314900398254],"max_abs_dp":0.0005718469619750977,"argmax_equal":true}}]},{"id":"semif_fb73a85d386cf59b0c51","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The designer issued a decision but its production scope is unavailable.","questions":{"decision":{"type":"choice","instructions":"Mass production is allowed only after mass-production approval. May it begin?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=2c3a0cb61f7bae78739e","provenance":{"family":"rule_application","variant":"missing","group_id":"2c3a0cb61f7bae78739e","label":0,"option_order":["insufficient","permitted","prohibited"]},"state_tokens":13,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","permitted","prohibited"],"gold":"insufficient","row_len":68,"state_len":13,"decide":67,"opts":[44,55,66],"row_ids_sha256":"45ccedb4b46b3d13a540182c024c64d3ca8ac735a1e653f6e1171d44e3d3e3be","oracle":{"probs":[0.6640077829360962,0.11264616250991821,0.22334615886211395],"logits_raw":[-1.3681122064590454,-5.636546611785889,-3.9896745681762695],"argmax":0,"top2_margin":0.44066160917282104,"near_tie":false},"graph":{"p":[0.6626162528991699,0.11274376511573792,0.22463996708393097],"max_abs_dp":0.0013915300369262695,"argmax_equal":true}}]},{"id":"semif_a02fd582d2881cc462b2","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The visitor's wristband covers the sculpture garden and expressly excludes the rooftop terrace.","questions":{"decision":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the sculpture garden?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=original group=42ed4cbfde7f2f64274b","provenance":{"family":"rule_application","variant":"original","group_id":"42ed4cbfde7f2f64274b","label":2,"option_order":["prohibited","insufficient","permitted"]},"state_tokens":17,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","insufficient","permitted"],"gold":"permitted","row_len":78,"state_len":17,"decide":77,"opts":[48,65,76],"row_ids_sha256":"d46d739213f63936988ecd765322e5eb0f8b5889ab37d50b455aa55a020c0d36","oracle":{"probs":[0.004480735864490271,0.009983424097299576,0.9855358600616455],"logits_raw":[-8.10856819152832,-6.180987358093262,4.868217468261719],"argmax":2,"top2_margin":0.9755524396896362,"near_tie":false},"graph":{"p":[0.0045144036412239075,0.00994602870196104,0.9855395555496216],"max_abs_dp":3.739539533853531e-05,"argmax_equal":true}}]},{"id":"semif_19f37d5a97034bd7d04f","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The visitor's wristband covers the sculpture garden and expressly excludes the rooftop terrace.","questions":{"decision":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the rooftop terrace?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=criterion_reversal group=42ed4cbfde7f2f64274b","provenance":{"family":"rule_application","variant":"criterion_reversal","group_id":"42ed4cbfde7f2f64274b","label":1,"option_order":["insufficient","prohibited","permitted"]},"state_tokens":17,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","prohibited","permitted"],"gold":"prohibited","row_len":78,"state_len":17,"decide":77,"opts":[54,65,76],"row_ids_sha256":"7ee5031220236c7f73f4228f2dce1fbd91e36a3591f9a6f56d27e24eae4dbe31","oracle":{"probs":[0.04313259199261665,0.9501571655273438,0.006710187066346407],"logits_raw":[-5.850513935089111,1.5898313522338867,-10.327335357666016],"argmax":1,"top2_margin":0.9070245623588562,"near_tie":false},"graph":{"p":[0.04485883563756943,0.9480558633804321,0.007085295394062996],"max_abs_dp":0.002101302146911621,"argmax_equal":true}}]},{"id":"semif_fda0ca94652c6d739ecf","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The wristband expressly excludes the sculpture garden and covers only the rooftop terrace.","questions":{"decision":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the sculpture garden?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=evidence_change group=42ed4cbfde7f2f64274b","provenance":{"family":"rule_application","variant":"evidence_change","group_id":"42ed4cbfde7f2f64274b","label":0,"option_order":["prohibited","permitted","insufficient"]},"state_tokens":16,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","permitted","insufficient"],"gold":"prohibited","row_len":77,"state_len":16,"decide":76,"opts":[47,58,75],"row_ids_sha256":"2319c321132dcc8f3ffbdd110ff905aa866758f0c1741cfb99742ff25ec4af6b","oracle":{"probs":[0.6104937791824341,0.04214628040790558,0.34735989570617676],"logits_raw":[-2.4210405349731445,-8.852705001831055,-3.7778282165527344],"argmax":0,"top2_margin":0.2631338834762573,"near_tie":false},"graph":{"p":[0.6104516983032227,0.04287881404161453,0.3466695249080658],"max_abs_dp":0.0007325336337089539,"argmax_equal":true}}]},{"id":"semif_d0e36c4cb10a29b3191c","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The visitor has a wristband but its covered areas are not supplied.","questions":{"decision":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the sculpture garden?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=42ed4cbfde7f2f64274b","provenance":{"family":"rule_application","variant":"missing","group_id":"42ed4cbfde7f2f64274b","label":0,"option_order":["insufficient","permitted","prohibited"]},"state_tokens":15,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","permitted","prohibited"],"gold":"insufficient","row_len":76,"state_len":15,"decide":75,"opts":[52,63,74],"row_ids_sha256":"821290498f7c312a4b814aa83454d65d914f6fa9422779a94c5a4b55c77e3867","oracle":{"probs":[0.7904524803161621,0.08996497094631195,0.11958259344100952],"logits_raw":[-1.2469587326049805,-6.475750923156738,-5.791020393371582],"argmax":0,"top2_margin":0.6708698868751526,"near_tie":false},"graph":{"p":[0.7911588549613953,0.09074243158102036,0.11809875816106796],"max_abs_dp":0.0014838352799415588,"argmax_equal":true}}]},{"id":"semif_a0a39ace7e5337d038a7","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The account owner permits changing the display name but expressly forbids changing the recovery email.","questions":{"decision":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the recovery email be changed?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=6dd193172b9240a04bf9","provenance":{"family":"rule_application","variant":"original","group_id":"6dd193172b9240a04bf9","label":0,"option_order":["prohibited","permitted","insufficient"]},"state_tokens":18,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","permitted","insufficient"],"gold":"prohibited","row_len":80,"state_len":18,"decide":79,"opts":[50,61,78],"row_ids_sha256":"e342fda6eaee30eb0362ab288861e786b222959655386c223c99faef9d3550d2","oracle":{"probs":[0.9940991401672363,0.0018019711133092642,0.004098864272236824],"logits_raw":[6.06341552734375,-9.125870704650879,-7.148510932922363],"argmax":0,"top2_margin":0.9900002479553223,"near_tie":false},"graph":{"p":[0.9940404295921326,0.001834048074670136,0.004125528037548065],"max_abs_dp":5.8710575103759766e-05,"argmax_equal":true}}]},{"id":"semif_eeae283841ce75dd25bf","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The account owner permits changing the display name but expressly forbids changing the recovery email.","questions":{"decision":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the display name be changed?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=6dd193172b9240a04bf9","provenance":{"family":"rule_application","variant":"criterion_reversal","group_id":"6dd193172b9240a04bf9","label":1,"option_order":["insufficient","permitted","prohibited"]},"state_tokens":18,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","permitted","prohibited"],"gold":"permitted","row_len":80,"state_len":18,"decide":79,"opts":[56,67,78],"row_ids_sha256":"7b91b697f5cb9dccba1f923bee934af2df2d0733491cd0679d1392ede339e8fc","oracle":{"probs":[0.007486369926482439,0.9897896647453308,0.0027239860501140356],"logits_raw":[-6.578224182128906,5.173906326293945,-9.010711669921875],"argmax":1,"top2_margin":0.9823033213615417,"near_tie":false},"graph":{"p":[0.00752001628279686,0.9897090792655945,0.0027709612622857094],"max_abs_dp":8.058547973632812e-05,"argmax_equal":true}}]},{"id":"semif_00bb4cc043f7d2b288d4","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The owner expressly permits changing the recovery email.","questions":{"decision":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the recovery email be changed?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=6dd193172b9240a04bf9","provenance":{"family":"rule_application","variant":"evidence_change","group_id":"6dd193172b9240a04bf9","label":2,"option_order":["insufficient","prohibited","permitted"]},"state_tokens":10,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","prohibited","permitted"],"gold":"permitted","row_len":72,"state_len":10,"decide":71,"opts":[48,59,70],"row_ids_sha256":"96027fe5e3a22147052109ae2d171f4d6f4fcc44dffa8408262502ea64cec892","oracle":{"probs":[0.007584902923554182,0.0019394466653466225,0.9904757142066956],"logits_raw":[-6.3804497718811035,-9.66171646118164,5.341887474060059],"argmax":2,"top2_margin":0.982890784740448,"near_tie":false},"graph":{"p":[0.0077123818919062614,0.001982057234272361,0.9903055429458618],"max_abs_dp":0.00017017126083374023,"argmax_equal":true}}]},{"id":"semif_1a81ead69e93e5d0c86a","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The owner requested an account update, but the fields to be changed are omitted.","questions":{"decision":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the recovery email be changed?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=6dd193172b9240a04bf9","provenance":{"family":"rule_application","variant":"missing","group_id":"6dd193172b9240a04bf9","label":2,"option_order":["permitted","prohibited","insufficient"]},"state_tokens":17,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["permitted","prohibited","insufficient"],"gold":"insufficient","row_len":79,"state_len":17,"decide":78,"opts":[49,60,77],"row_ids_sha256":"e263ee6278c63545194a14f874da0a1fbbae8f01e362a24a8eee59c27beef2fb","oracle":{"probs":[0.3203682005405426,0.1895190328359604,0.4901128113269806],"logits_raw":[-2.4291203022003174,-3.6922521591186523,-1.4061529636383057],"argmax":2,"top2_margin":0.169744610786438,"near_tie":false},"graph":{"p":[0.3185552656650543,0.1896914392709732,0.4917532503604889],"max_abs_dp":0.0018129348754882812,"argmax_equal":true}}]},{"id":"semif_4444dad25e438f128673","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The applicant lives in the valley but works outside it.","questions":{"decision":{"type":"choice","instructions":"A resident's pass requires living in the valley and has no workplace restriction. May it be issued?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=original group=3200eb252d6737e03027","provenance":{"family":"rule_application","variant":"original","group_id":"3200eb252d6737e03027","label":1,"option_order":["prohibited","permitted","insufficient"]},"state_tokens":12,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","permitted","insufficient"],"gold":"permitted","row_len":73,"state_len":12,"decide":72,"opts":[43,54,71],"row_ids_sha256":"1c50a555809bf1a4efb4c2b9b3c328b1cb5e1c53140ee3d1e4970e1982bd9818","oracle":{"probs":[0.1383441537618637,0.48804470896720886,0.3736112415790558],"logits_raw":[-4.8966827392578125,-1.8634655475616455,-2.506340980529785],"argmax":1,"top2_margin":0.11443346738815308,"near_tie":false},"graph":{"p":[0.1392083764076233,0.48831239342689514,0.37247925996780396],"max_abs_dp":0.001131981611251831,"argmax_equal":true}}]},{"id":"semif_3226202ba1ada2570e9f","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The applicant lives in the valley but works outside it.","questions":{"decision":{"type":"choice","instructions":"Under the alternative rule, a pass requires working in the valley; residence alone is insufficient. May it be issued?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=criterion_reversal group=3200eb252d6737e03027","provenance":{"family":"rule_application","variant":"criterion_reversal","group_id":"3200eb252d6737e03027","label":0,"option_order":["prohibited","permitted","insufficient"]},"state_tokens":12,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","permitted","insufficient"],"gold":"prohibited","row_len":76,"state_len":12,"decide":75,"opts":[46,57,74],"row_ids_sha256":"47c807e392e77ea8aba8c016ba07262ba9627135ec49fc28ae2dd0087710fc77","oracle":{"probs":[0.8890881538391113,0.021135009825229645,0.089776910841465],"logits_raw":[3.260497570037842,-5.736361980438232,-2.256258964538574],"argmax":0,"top2_margin":0.7993112206459045,"near_tie":false},"graph":{"p":[0.8909421563148499,0.02105826325714588,0.08799958229064941],"max_abs_dp":0.0018540024757385254,"argmax_equal":true}}]},{"id":"semif_3f783eec5c90812479bc","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The applicant lives outside the valley and works inside it.","questions":{"decision":{"type":"choice","instructions":"A resident's pass requires living in the valley and has no workplace restriction. May it be issued?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=evidence_change group=3200eb252d6737e03027","provenance":{"family":"rule_application","variant":"evidence_change","group_id":"3200eb252d6737e03027","label":1,"option_order":["permitted","prohibited","insufficient"]},"state_tokens":12,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["permitted","prohibited","insufficient"],"gold":"prohibited","row_len":73,"state_len":12,"decide":72,"opts":[43,54,71],"row_ids_sha256":"e7a27914e1511c7673c52134b96587ad97b5135a40595d31267c5f44adee57fc","oracle":{"probs":[0.05365244671702385,0.5985097885131836,0.34783777594566345],"logits_raw":[-6.092349052429199,-0.2891596853733063,-1.5949389934539795],"argmax":1,"top2_margin":0.25067201256752014,"near_tie":false},"graph":{"p":[0.05477968975901604,0.5961421132087708,0.34907811880111694],"max_abs_dp":0.002367675304412842,"argmax_equal":true}}]},{"id":"semif_ffee861474a9665a093a","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The applicant supplies a name but no home or work location.","questions":{"decision":{"type":"choice","instructions":"A resident's pass requires living in the valley and has no workplace restriction. May it be issued?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=3200eb252d6737e03027","provenance":{"family":"rule_application","variant":"missing","group_id":"3200eb252d6737e03027","label":2,"option_order":["prohibited","permitted","insufficient"]},"state_tokens":13,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","permitted","insufficient"],"gold":"insufficient","row_len":74,"state_len":13,"decide":73,"opts":[44,55,72],"row_ids_sha256":"e3d9dda626126421e54890ba4bd4b7d2149b4a6ec6672c8cf2afb01a088377b2","oracle":{"probs":[0.3995288610458374,0.19905975461006165,0.4014114439487457],"logits_raw":[-1.7608011960983276,-3.4370503425598145,-1.7494903802871704],"argmax":2,"top2_margin":0.0018825829029083252,"near_tie":true},"graph":{"p":[0.3973971903324127,0.1995469182729721,0.40305593609809875],"max_abs_dp":0.0021316707134246826,"argmax_equal":true}}]},{"id":"semif_8c992e17e4fb54b044f5","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The florist confirms that the wreath was delivered. Candidate B: The florist offers to deliver a wreath tomorrow.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting completed delivery.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=c0cbeb97315307af8111","provenance":{"family":"candidate_selection","variant":"original","group_id":"c0cbeb97315307af8111","label":1,"option_order":["B","A","insufficient"]},"state_tokens":29,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","A","insufficient"],"gold":"A","row_len":61,"state_len":29,"decide":60,"opts":[42,48,59],"row_ids_sha256":"35f3d573105e9ae5fa5c52ee66335b640a9fe4d8963184fec1e250c3011489da","oracle":{"probs":[0.0011179291177541018,0.983923614025116,0.014958481304347515],"logits_raw":[-10.183874130249023,6.129314422607422,-3.9430599212646484],"argmax":1,"top2_margin":0.9689651131629944,"near_tie":false},"graph":{"p":[0.0011347385589033365,0.9838619828224182,0.01500327605754137],"max_abs_dp":6.16312026977539e-05,"argmax_equal":true}}]},{"id":"semif_695dd61195ca025c4f4c","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The florist confirms that the wreath was delivered. Candidate B: The florist offers to deliver a wreath tomorrow.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate offering a future delivery.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=c0cbeb97315307af8111","provenance":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"c0cbeb97315307af8111","label":0,"option_order":["B","insufficient","A"]},"state_tokens":29,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","insufficient","A"],"gold":"B","row_len":62,"state_len":29,"decide":61,"opts":[43,54,60],"row_ids_sha256":"55e50864d5fb48f4c61780c39b77177c3735c0c3e80e0e202b801f96b5fe2c5f","oracle":{"probs":[0.9457037448883057,0.045305054634809494,0.008991256356239319],"logits_raw":[5.009383201599121,-2.3014254570007324,-6.192407608032227],"argmax":0,"top2_margin":0.9003986716270447,"near_tie":false},"graph":{"p":[0.9456384181976318,0.04528052359819412,0.009081054478883743],"max_abs_dp":8.979812264442444e-05,"argmax_equal":true}}]},{"id":"semif_e6155431ab7b0f8170d4","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: I can deliver a wreath tomorrow. Candidate B: The wreath was delivered.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting completed delivery.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=c0cbeb97315307af8111","provenance":{"family":"candidate_selection","variant":"evidence_change","group_id":"c0cbeb97315307af8111","label":1,"option_order":["insufficient","B","A"]},"state_tokens":21,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","B","A"],"gold":"B","row_len":53,"state_len":21,"decide":52,"opts":[39,45,51],"row_ids_sha256":"bf46db2beaf34ce2f3c68ac8b36b384787e731759bcb552e6a361207851a622e","oracle":{"probs":[0.02949335239827633,0.965252161026001,0.005254472140222788],"logits_raw":[-2.786595106124878,5.606247901916504,-6.9372358322143555],"argmax":1,"top2_margin":0.9357588291168213,"near_tie":false},"graph":{"p":[0.029879571869969368,0.9647921323776245,0.005328331608325243],"max_abs_dp":0.00046002864837646484,"argmax_equal":true}}]},{"id":"semif_b49ecda75086d8c68d2a","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The florist gives an address. Candidate B: The wreath contains lilies.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting completed delivery.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=c0cbeb97315307af8111","provenance":{"family":"candidate_selection","variant":"missing","group_id":"c0cbeb97315307af8111","label":2,"option_order":["B","A","insufficient"]},"state_tokens":21,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","A","insufficient"],"gold":"insufficient","row_len":53,"state_len":21,"decide":52,"opts":[34,40,51],"row_ids_sha256":"1b50c47fdf05da2906e4544f7712aa89e27c11a6866a189aa9c46db8a1a3b1ce","oracle":{"probs":[0.022015336900949478,0.6057172417640686,0.3722674548625946],"logits_raw":[-6.611644744873047,1.36362624168396,0.1923593282699585],"argmax":1,"top2_margin":0.233449786901474,"near_tie":false},"graph":{"p":[0.022279679775238037,0.605392336845398,0.3723280131816864],"max_abs_dp":0.0003249049186706543,"argmax_equal":true}}]},{"id":"semif_587ced261c6092cd1325","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The customer asks how an engraving would look but explicitly declines having it engraved yet. Candidate B: The customer instructs the shop to engrave the ring.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate authorizing engraving.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=original group=428176cd8ad2c7a74eea","provenance":{"family":"candidate_selection","variant":"original","group_id":"428176cd8ad2c7a74eea","label":1,"option_order":["insufficient","B","A"]},"state_tokens":36,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","B","A"],"gold":"B","row_len":69,"state_len":36,"decide":68,"opts":[55,61,67],"row_ids_sha256":"517e4459197f5f462cb334fc705feae582b963aaed88efdf0d89748b7628d6ce","oracle":{"probs":[0.046055130660533905,0.9483346343040466,0.005610225722193718],"logits_raw":[-3.3582229614257812,3.9197611808776855,-8.423555374145508],"argmax":1,"top2_margin":0.9022794961929321,"near_tie":false},"graph":{"p":[0.04668768495321274,0.9476630091667175,0.0056493449956178665],"max_abs_dp":0.0006716251373291016,"argmax_equal":true}}]},{"id":"semif_533d4423d311de82b27d","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The customer asks how an engraving would look but explicitly declines having it engraved yet. Candidate B: The customer instructs the shop to engrave the ring.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate asking only for information while withholding authorization.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=criterion_reversal group=428176cd8ad2c7a74eea","provenance":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"428176cd8ad2c7a74eea","label":1,"option_order":["insufficient","A","B"]},"state_tokens":36,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","A","B"],"gold":"A","row_len":72,"state_len":36,"decide":71,"opts":[58,64,70],"row_ids_sha256":"fb70171b1f55d54557471ffa45c114a937e16bd33c1990d0683f60056d2baee7","oracle":{"probs":[0.05380565673112869,0.9402286410331726,0.005965682677924633],"logits_raw":[-1.3969218730926514,5.486172199249268,-6.6886796951293945],"argmax":1,"top2_margin":0.8864229917526245,"near_tie":false},"graph":{"p":[0.0546233169734478,0.9392798542976379,0.006096850614994764],"max_abs_dp":0.000948786735534668,"argmax_equal":true}}]},{"id":"semif_2bf540d515a51755b25c","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: Please engrave the ring. Candidate B: Show me how it would look, but do not engrave it yet.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate authorizing engraving.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=evidence_change group=428176cd8ad2c7a74eea","provenance":{"family":"candidate_selection","variant":"evidence_change","group_id":"428176cd8ad2c7a74eea","label":2,"option_order":["insufficient","B","A"]},"state_tokens":28,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","B","A"],"gold":"A","row_len":61,"state_len":28,"decide":60,"opts":[47,53,59],"row_ids_sha256":"d67b800263196923732952d5ba23e68c73ba4b8a5d204cc9afe54a7799515970","oracle":{"probs":[0.04508393257856369,0.002212027320638299,0.9527040719985962],"logits_raw":[-1.869105577468872,-9.12242317199707,5.471219539642334],"argmax":2,"top2_margin":0.9076201319694519,"near_tie":false},"graph":{"p":[0.04525205120444298,0.0022130566649138927,0.9525349140167236],"max_abs_dp":0.0001691579818725586,"argmax_equal":true}}]},{"id":"semif_fdb9fc214aea2710763a","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The customer gives a name. Candidate B: The ring is silver.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate authorizing engraving.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=428176cd8ad2c7a74eea","provenance":{"family":"candidate_selection","variant":"missing","group_id":"428176cd8ad2c7a74eea","label":2,"option_order":["B","A","insufficient"]},"state_tokens":18,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","A","insufficient"],"gold":"insufficient","row_len":51,"state_len":18,"decide":50,"opts":[32,38,49],"row_ids_sha256":"a5887295c96bd1248785615ef738accd0efa7711e1d75a813d00f0f60283a06c","oracle":{"probs":[0.007924344390630722,0.927176296710968,0.06489942967891693],"logits_raw":[-6.965169906616211,4.492931365966797,-1.9054890871047974],"argmax":1,"top2_margin":0.8622768521308899,"near_tie":false},"graph":{"p":[0.008049456402659416,0.9267272353172302,0.06522324681282043],"max_abs_dp":0.00044906139373779297,"argmax_equal":true}}]},{"id":"semif_f6d68126b43f8c9ed183","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The note records that the completed insulation test passed. Candidate B: The note lists insulation-test steps but says no test was run.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an observed test result.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=389e65bf1ba709b84b12","provenance":{"family":"candidate_selection","variant":"original","group_id":"389e65bf1ba709b84b12","label":1,"option_order":["insufficient","A","B"]},"state_tokens":30,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","A","B"],"gold":"A","row_len":64,"state_len":30,"decide":63,"opts":[50,56,62],"row_ids_sha256":"4f2fb6c34bf42a46029577aee9b9efd1ba5ce4ade48a717e695de01550fe4de5","oracle":{"probs":[0.040509115904569626,0.9535921216011047,0.005898686125874519],"logits_raw":[-2.1315951347351074,5.468417167663574,-6.767565727233887],"argmax":1,"top2_margin":0.913083016872406,"near_tie":false},"graph":{"p":[0.04071710258722305,0.953326404094696,0.005956514272838831],"max_abs_dp":0.0002657175064086914,"argmax_equal":true}}]},{"id":"semif_44bae0c8a1615a096c43","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The note records that the completed insulation test passed. Candidate B: The note lists insulation-test steps but says no test was run.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate containing only an unexecuted test procedure.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=389e65bf1ba709b84b12","provenance":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"389e65bf1ba709b84b12","label":0,"option_order":["B","A","insufficient"]},"state_tokens":30,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","A","insufficient"],"gold":"B","row_len":67,"state_len":30,"decide":66,"opts":[48,54,65],"row_ids_sha256":"ec965eab5b7d33c08c1053225ecdee35a4f536e93aab0299d7a6061390653ddf","oracle":{"probs":[0.9387495517730713,0.008345278911292553,0.05290508270263672],"logits_raw":[5.182891368865967,-6.180527687072754,-1.7370266914367676],"argmax":0,"top2_margin":0.8858444690704346,"near_tie":false},"graph":{"p":[0.9376094341278076,0.008493851870298386,0.0538967065513134],"max_abs_dp":0.0011401176452636719,"argmax_equal":true}}]},{"id":"semif_2175379f4c69e6207626","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: Here are insulation-test steps; no test was run. Candidate B: The completed insulation test passed.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an observed test result.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=389e65bf1ba709b84b12","provenance":{"family":"candidate_selection","variant":"evidence_change","group_id":"389e65bf1ba709b84b12","label":1,"option_order":["insufficient","B","A"]},"state_tokens":24,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","B","A"],"gold":"B","row_len":58,"state_len":24,"decide":57,"opts":[44,50,56],"row_ids_sha256":"32132b5d370e0e414ec4efb4662d271677165dd2e5941964d962873cbe77395c","oracle":{"probs":[0.04833228886127472,0.9422966241836548,0.009371018968522549],"logits_raw":[-1.713789939880371,5.432708740234375,-5.660861492156982],"argmax":1,"top2_margin":0.8939643502235413,"near_tie":false},"graph":{"p":[0.04871567338705063,0.9417711496353149,0.009513179771602154],"max_abs_dp":0.0005254745483398438,"argmax_equal":true}}]},{"id":"semif_def9fe59d262357a673a","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The cable is red. Candidate B: The tester is in a drawer.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an observed test result.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=389e65bf1ba709b84b12","provenance":{"family":"candidate_selection","variant":"missing","group_id":"389e65bf1ba709b84b12","label":0,"option_order":["insufficient","A","B"]},"state_tokens":19,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","A","B"],"gold":"insufficient","row_len":53,"state_len":19,"decide":52,"opts":[39,45,51],"row_ids_sha256":"214f50d6d91625a3a7ab12973e51028b73752877f771607dd70256e0528f9153","oracle":{"probs":[0.2170504629611969,0.700346052646637,0.08260342478752136],"logits_raw":[-1.0876407623291016,1.7309136390686035,-3.412074565887451],"argmax":1,"top2_margin":0.48329558968544006,"near_tie":false},"graph":{"p":[0.21922583878040314,0.6973602175712585,0.08341395109891891],"max_abs_dp":0.002985835075378418,"argmax_equal":true}}]},{"id":"semif_712f6ffb763eef7c9a12","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The editor quotes a proposal for a paywall only to argue against it. Candidate B: The editor endorses introducing a paywall.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate supporting a paywall.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=original group=45f8ead2522a14384cdc","provenance":{"family":"candidate_selection","variant":"original","group_id":"45f8ead2522a14384cdc","label":0,"option_order":["B","A","insufficient"]},"state_tokens":31,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","A","insufficient"],"gold":"B","row_len":64,"state_len":31,"decide":63,"opts":[45,51,62],"row_ids_sha256":"5686dcb74403be3e1fe04266a1cda0b8bd00a07632f87c7fb30d08a68167cea5","oracle":{"probs":[0.9517671465873718,0.00764705752953887,0.04058578610420227],"logits_raw":[4.2721476554870605,-7.3346357345581055,-3.3187055587768555],"argmax":0,"top2_margin":0.9111813306808472,"near_tie":false},"graph":{"p":[0.9516978859901428,0.007647923659533262,0.04065413028001785],"max_abs_dp":6.92605972290039e-05,"argmax_equal":true}}]},{"id":"semif_e1d610bd14e3d16c09b9","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The editor quotes a proposal for a paywall only to argue against it. Candidate B: The editor endorses introducing a paywall.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate opposing a paywall.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=criterion_reversal group=45f8ead2522a14384cdc","provenance":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"45f8ead2522a14384cdc","label":0,"option_order":["A","insufficient","B"]},"state_tokens":31,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["A","insufficient","B"],"gold":"A","row_len":64,"state_len":31,"decide":63,"opts":[45,56,62],"row_ids_sha256":"7b95d60db279c5fb51998061efbcd608f57ead5edcf556e91a52132cdd0a3ce0","oracle":{"probs":[0.9558648467063904,0.037930525839328766,0.006204604171216488],"logits_raw":[4.863903999328613,-2.90008282661438,-7.25615119934082],"argmax":0,"top2_margin":0.9179342985153198,"near_tie":false},"graph":{"p":[0.9555055499076843,0.03824630752205849,0.006248193327337503],"max_abs_dp":0.0003592967987060547,"argmax_equal":true}}]},{"id":"semif_22ec57822db50ec121cf","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: I endorse introducing a paywall. Candidate B: I reject introducing a paywall.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate supporting a paywall.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=evidence_change group=45f8ead2522a14384cdc","provenance":{"family":"candidate_selection","variant":"evidence_change","group_id":"45f8ead2522a14384cdc","label":1,"option_order":["insufficient","A","B"]},"state_tokens":21,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","A","B"],"gold":"A","row_len":54,"state_len":21,"decide":53,"opts":[40,46,52],"row_ids_sha256":"78dfd25f5b5f0f350ffc497adbad29c1701786db88c591ca320f07d27c49d66d","oracle":{"probs":[0.015419436618685722,0.983207106590271,0.0013733897358179092],"logits_raw":[-4.24796724319458,5.749629974365234,-10.066631317138672],"argmax":1,"top2_margin":0.9677876830101013,"near_tie":false},"graph":{"p":[0.015706205740571022,0.9829012155532837,0.0013925248058512807],"max_abs_dp":0.0003058910369873047,"argmax_equal":true}}]},{"id":"semif_3e75f2b84e8e08e3d3f2","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The editor names the magazine. Candidate B: The website has a blue header.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate supporting a paywall.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=45f8ead2522a14384cdc","provenance":{"family":"candidate_selection","variant":"missing","group_id":"45f8ead2522a14384cdc","label":2,"option_order":["B","A","insufficient"]},"state_tokens":20,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","A","insufficient"],"gold":"insufficient","row_len":53,"state_len":20,"decide":52,"opts":[34,40,51],"row_ids_sha256":"0491d85cdf458158947c4d7a2e68a916b3564f222fdcf59a1f232e8f33f95c5b","oracle":{"probs":[0.11529788374900818,0.15538500249385834,0.7293170690536499],"logits_raw":[-4.601580619812012,-3.8836469650268555,-0.16340631246566772],"argmax":2,"top2_margin":0.5739320516586304,"near_tie":false},"graph":{"p":[0.11579664796590805,0.15571530163288116,0.728488028049469],"max_abs_dp":0.0008290410041809082,"argmax_equal":true}}]},{"id":"semif_ebd74aa05324a1556288","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The owner asks to reseal the existing window while keeping its frame. Candidate B: The owner asks to remove the window and fit a complete replacement.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate asking to repair the existing window.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=3ecec8e2c166e397ad07","provenance":{"family":"candidate_selection","variant":"original","group_id":"3ecec8e2c166e397ad07","label":1,"option_order":["insufficient","A","B"]},"state_tokens":34,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","A","B"],"gold":"A","row_len":69,"state_len":34,"decide":68,"opts":[55,61,67],"row_ids_sha256":"6df8cad64e170c828ec555bccb09d1a5e5ecdd3683d20fba2b11032839bc4d30","oracle":{"probs":[0.04075147211551666,0.9548779726028442,0.004370470065623522],"logits_raw":[-1.7843971252441406,5.804504871368408,-7.156195640563965],"argmax":1,"top2_margin":0.9141265153884888,"near_tie":false},"graph":{"p":[0.0411495678126812,0.9544561505317688,0.004394311923533678],"max_abs_dp":0.00042182207107543945,"argmax_equal":true}}]},{"id":"semif_d7e84e4245a29817da1f","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The owner asks to reseal the existing window while keeping its frame. Candidate B: The owner asks to remove the window and fit a complete replacement.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate asking for complete replacement.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=3ecec8e2c166e397ad07","provenance":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"3ecec8e2c166e397ad07","label":2,"option_order":["insufficient","A","B"]},"state_tokens":34,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","A","B"],"gold":"B","row_len":67,"state_len":34,"decide":66,"opts":[53,59,65],"row_ids_sha256":"098a35dcd75886b0af318df846aaf749cec3d5d7a2c88deeebfd81e41d330a17","oracle":{"probs":[0.022718368098139763,0.0016438306774944067,0.9756377935409546],"logits_raw":[-2.0069212913513184,-8.325555801391602,7.039628505706787],"argmax":2,"top2_margin":0.9529194235801697,"near_tie":false},"graph":{"p":[0.02284936048090458,0.0016506105894222856,0.9755000472068787],"max_abs_dp":0.00013774633407592773,"argmax_equal":true}}]},{"id":"semif_f41ae66b3c09630c7a42","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: Remove the window and install a replacement. Candidate B: Reseal the existing window and keep its frame.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate asking to repair the existing window.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=3ecec8e2c166e397ad07","provenance":{"family":"candidate_selection","variant":"evidence_change","group_id":"3ecec8e2c166e397ad07","label":2,"option_order":["insufficient","A","B"]},"state_tokens":26,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","A","B"],"gold":"B","row_len":61,"state_len":26,"decide":60,"opts":[47,53,59],"row_ids_sha256":"a51bae2374da901a57cf29e7910cc8c4212ce5ef33627965e82707c3f7eab93e","oracle":{"probs":[0.039210349321365356,0.004813856445252895,0.9559757709503174],"logits_raw":[-1.135624647140503,-6.182174205780029,6.548798561096191],"argmax":2,"top2_margin":0.9167654514312744,"near_tie":false},"graph":{"p":[0.03949188068509102,0.004848693031817675,0.9556593894958496],"max_abs_dp":0.00031638145446777344,"argmax_equal":true}}]},{"id":"semif_82c2cd97c160ad27951b","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The window faces a courtyard. Candidate B: The owner gives a postcode.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate asking to repair the existing window.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=3ecec8e2c166e397ad07","provenance":{"family":"candidate_selection","variant":"missing","group_id":"3ecec8e2c166e397ad07","label":0,"option_order":["insufficient","B","A"]},"state_tokens":19,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","B","A"],"gold":"insufficient","row_len":54,"state_len":19,"decide":53,"opts":[40,46,52],"row_ids_sha256":"0a75e012565ed1259b4e093093e02a6367445f9d8938d6645feaf2378cec9101","oracle":{"probs":[0.7681891918182373,0.03809200972318649,0.193718820810318],"logits_raw":[-0.6092422604560852,-7.83709192276001,-3.923884630203247],"argmax":0,"top2_margin":0.5744704008102417,"near_tie":false},"graph":{"p":[0.7672715783119202,0.03845253959298134,0.19427590072155],"max_abs_dp":0.0009176135063171387,"argmax_equal":true}}]},{"id":"semif_7ada97b6f99f4788e5c7","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The sponsor confirms receipt of a grant application but explicitly says no award decision has been made. Candidate B: The sponsor confirms that the grant was awarded.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an awarded grant.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=original group=a71a17a287f3d30db2aa","provenance":{"family":"candidate_selection","variant":"original","group_id":"a71a17a287f3d30db2aa","label":0,"option_order":["B","A","insufficient"]},"state_tokens":34,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","A","insufficient"],"gold":"B","row_len":67,"state_len":34,"decide":66,"opts":[48,54,65],"row_ids_sha256":"92ff89b91861dd9a34ba4e00d91a5095afd5440a78861ac7a9eb77d593361810","oracle":{"probs":[0.9861090779304504,0.003296883311122656,0.01059398427605629],"logits_raw":[5.449187755584717,-8.267196655273438,-5.458593368530273],"argmax":0,"top2_margin":0.9755150675773621,"near_tie":false},"graph":{"p":[0.9860830903053284,0.003330154111608863,0.010586691088974476],"max_abs_dp":3.327080048620701e-05,"argmax_equal":true}}]},{"id":"semif_4d7a003342db5b0a17e3","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The sponsor confirms receipt of a grant application but explicitly says no award decision has been made. Candidate B: The sponsor confirms that the grant was awarded.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting receipt with the award still undecided.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=criterion_reversal group=a71a17a287f3d30db2aa","provenance":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"a71a17a287f3d30db2aa","label":1,"option_order":["insufficient","A","B"]},"state_tokens":34,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","A","B"],"gold":"A","row_len":70,"state_len":34,"decide":69,"opts":[56,62,68],"row_ids_sha256":"228f9b69aee5150dd69d7420a36e13b43406ed95969ae6048024f8ba97f386e4","oracle":{"probs":[0.0333622582256794,0.9631707072257996,0.00346710835583508],"logits_raw":[-2.2164058685302734,5.874671936035156,-7.66395378112793],"argmax":1,"top2_margin":0.9298084378242493,"near_tie":false},"graph":{"p":[0.0333634689450264,0.9631698131561279,0.003466720227152109],"max_abs_dp":1.210719347000122e-06,"argmax_equal":true}}]},{"id":"semif_19d3d8d0bf773e88be7b","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The grant was awarded. Candidate B: The application was received; the award remains undecided.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an awarded grant.","criteria":{"A":"Candidate A","B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=evidence_change group=a71a17a287f3d30db2aa","provenance":{"family":"candidate_selection","variant":"evidence_change","group_id":"a71a17a287f3d30db2aa","label":0,"option_order":["A","B","insufficient"]},"state_tokens":22,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["A","B","insufficient"],"gold":"A","row_len":55,"state_len":22,"decide":54,"opts":[36,42,53],"row_ids_sha256":"04935c4d2da025ceea4842fa601a4f342360572bf05889ef760dc238ddf80b98","oracle":{"probs":[0.9811984300613403,0.0033262514043599367,0.015475344844162464],"logits_raw":[5.974879264831543,-7.708155155181885,-4.00908899307251],"argmax":0,"top2_margin":0.9657230973243713,"near_tie":false},"graph":{"p":[0.9810782074928284,0.0033524194732308388,0.015569301322102547],"max_abs_dp":0.00012022256851196289,"argmax_equal":true}}]},{"id":"semif_d6de731cca52359b1c87","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The sponsor gives its name. Candidate B: The application has a reference number.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an awarded grant.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=a71a17a287f3d30db2aa","provenance":{"family":"candidate_selection","variant":"missing","group_id":"a71a17a287f3d30db2aa","label":0,"option_order":["insufficient","A","B"]},"state_tokens":20,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","A","B"],"gold":"insufficient","row_len":53,"state_len":20,"decide":52,"opts":[39,45,51],"row_ids_sha256":"9aac4185bcd04ff6346dbf53716fe8f3a341e5414f0f73a175853e93da76a01c","oracle":{"probs":[0.45985567569732666,0.4122198522090912,0.12792447209358215],"logits_raw":[-0.6749356985092163,-0.9380514025688171,-3.753410816192627],"argmax":0,"top2_margin":0.047635823488235474,"near_tie":false},"graph":{"p":[0.4595704674720764,0.41206303238868713,0.12836644053459167],"max_abs_dp":0.0004419684410095215,"argmax_equal":true}}]},{"id":"semif_b45c63fb92729af27f05","source":"semif_authored144","request":{"model":"kev-0.8b","state":"On Monday the owner authorized scanning her notebook. On Tuesday she explicitly withdrew that authorization. The record says withdrawals cancel earlier permission, and no later permission was given.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: permission to scan the notebook is currently valid.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=986d7eaba0c7e9738e79","provenance":{"family":"evidence_interpretation","variant":"original","group_id":"986d7eaba0c7e9738e79","label":0,"option_order":["contradicted","supported","insufficient"]},"state_tokens":33,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["contradicted","supported","insufficient"],"gold":"contradicted","row_len":80,"state_len":33,"decide":79,"opts":[58,67,78],"row_ids_sha256":"efe05f322748cd2627e13b54aec5915dfc910cf2973d7911c1ee0b86c74b16ac","oracle":{"probs":[0.9448158740997314,0.009024588391184807,0.04615957662463188],"logits_raw":[2.8068113327026367,-8.383816719055176,-4.456778049468994],"argmax":0,"top2_margin":0.8986563086509705,"near_tie":false},"graph":{"p":[0.9445961713790894,0.00923102255910635,0.046172868460416794],"max_abs_dp":0.00021970272064208984,"argmax_equal":true}}]},{"id":"semif_46b7029b9a704138b77a","source":"semif_authored144","request":{"model":"kev-0.8b","state":"On Monday the owner authorized scanning her notebook. On Tuesday she explicitly withdrew that authorization. The record says withdrawals cancel earlier permission, and no later permission was given.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the owner gave permission on Monday.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=986d7eaba0c7e9738e79","provenance":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"986d7eaba0c7e9738e79","label":2,"option_order":["insufficient","contradicted","supported"]},"state_tokens":33,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","contradicted","supported"],"gold":"supported","row_len":78,"state_len":33,"decide":77,"opts":[56,67,76],"row_ids_sha256":"72ce358729318314ad03da1bde12a3a1877c9c942d48cf5829b30a58161d8a65","oracle":{"probs":[0.020303642377257347,0.00488915154710412,0.9748072028160095],"logits_raw":[-4.791959762573242,-8.21764850616455,4.5229172706604],"argmax":2,"top2_margin":0.9545035362243652,"near_tie":false},"graph":{"p":[0.02073846012353897,0.005035669542849064,0.9742258787155151],"max_abs_dp":0.0005813241004943848,"argmax_equal":true}}]},{"id":"semif_caf773b23934fa7362a1","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The owner authorized scanning the notebook on Monday. The complete permission record confirms that she has not withdrawn it and it remains valid.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: permission to scan the notebook is currently valid.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=986d7eaba0c7e9738e79","provenance":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"986d7eaba0c7e9738e79","label":0,"option_order":["supported","contradicted","insufficient"]},"state_tokens":26,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","contradicted","insufficient"],"gold":"supported","row_len":73,"state_len":26,"decide":72,"opts":[49,60,71],"row_ids_sha256":"aa61324b46f672b0f28d996bb615ecd5b3a643e7d7f86ba0fa5cdafedd5bfb04","oracle":{"probs":[0.9774745106697083,0.0037949716206640005,0.018730532377958298],"logits_raw":[4.782076358795166,-8.574617385864258,-4.73341178894043],"argmax":0,"top2_margin":0.9587439894676208,"near_tie":false},"graph":{"p":[0.9770721793174744,0.003879281459376216,0.01904866471886635],"max_abs_dp":0.0004023313522338867,"argmax_equal":true}}]},{"id":"semif_f46f392ef9e9e9df564b","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The owner authorized scanning on Monday. The supplied record omits all later permission changes and does not establish current validity.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: permission to scan the notebook is currently valid.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=986d7eaba0c7e9738e79","provenance":{"family":"evidence_interpretation","variant":"missing","group_id":"986d7eaba0c7e9738e79","label":1,"option_order":["supported","insufficient","contradicted"]},"state_tokens":24,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","insufficient","contradicted"],"gold":"insufficient","row_len":71,"state_len":24,"decide":70,"opts":[47,58,69],"row_ids_sha256":"db0a498931c9cd449c2aa004da2f5c9a2a60d1d87579a6f48de5b3ab94f47dcc","oracle":{"probs":[0.0073618353344500065,0.9841703176498413,0.008467898704111576],"logits_raw":[-10.207108497619629,1.5716842412948608,-9.87032699584961],"argmax":1,"top2_margin":0.9757024049758911,"near_tie":false},"graph":{"p":[0.007486408110707998,0.9839788675308228,0.008534652180969715],"max_abs_dp":0.0001914501190185547,"argmax_equal":true}}]},{"id":"semif_005dcf6d2c0f21d64796","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The homeowner first allowed a surveyor to enter the attic. Before the visit she explicitly withdrew that permission. No later decision exists.","questions":{"decision":{"type":"choice","instructions":"Use the homeowner's latest permission decision; a withdrawal cancels the earlier grant. May the surveyor enter the attic?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=ca2e0b043985fa49bcd2","provenance":{"family":"rule_application","variant":"original","group_id":"ca2e0b043985fa49bcd2","label":1,"option_order":["permitted","prohibited","insufficient"]},"state_tokens":27,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["permitted","prohibited","insufficient"],"gold":"prohibited","row_len":92,"state_len":27,"decide":91,"opts":[62,73,90],"row_ids_sha256":"9ce0ba40fc771339e8fe6e583a7bb2a3ec3c2f3c6d86e2bcc141fc9235cea91f","oracle":{"probs":[0.040446750819683075,0.8976015448570251,0.06195172667503357],"logits_raw":[-4.589365005493164,2.8687641620635986,-3.563499927520752],"argmax":1,"top2_margin":0.835649847984314,"near_tie":false},"graph":{"p":[0.04084242880344391,0.8970901370048523,0.06206744909286499],"max_abs_dp":0.0005114078521728516,"argmax_equal":true}}]},{"id":"semif_edb708bd0fe5fd67d6ee","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The homeowner first allowed a surveyor to enter the attic. Before the visit she explicitly withdrew that permission. No later decision exists.","questions":{"decision":{"type":"choice","instructions":"For this fictional protocol, the first recorded decision is binding and later withdrawals are ignored. May the surveyor enter the attic?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=ca2e0b043985fa49bcd2","provenance":{"family":"rule_application","variant":"criterion_reversal","group_id":"ca2e0b043985fa49bcd2","label":0,"option_order":["permitted","prohibited","insufficient"]},"state_tokens":27,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["permitted","prohibited","insufficient"],"gold":"permitted","row_len":93,"state_len":27,"decide":92,"opts":[63,74,91],"row_ids_sha256":"cc30a82c3369dcf1f86b4000713f5d98fb5d59fbccaa37017aa85c6eccd180b6","oracle":{"probs":[0.29649534821510315,0.22774609923362732,0.47575849294662476],"logits_raw":[-2.563941478729248,-3.198657751083374,-1.4261714220046997],"argmax":2,"top2_margin":0.1792631447315216,"near_tie":false},"graph":{"p":[0.29714393615722656,0.2283095121383667,0.47454655170440674],"max_abs_dp":0.0012119412422180176,"argmax_equal":true}}]},{"id":"semif_97a35499bd8717f3e786","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The homeowner first refused attic entry, then explicitly granted it before the visit. No later decision exists.","questions":{"decision":{"type":"choice","instructions":"Use the homeowner's latest permission decision; a withdrawal cancels the earlier grant. May the surveyor enter the attic?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=ca2e0b043985fa49bcd2","provenance":{"family":"rule_application","variant":"evidence_change","group_id":"ca2e0b043985fa49bcd2","label":2,"option_order":["prohibited","insufficient","permitted"]},"state_tokens":21,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","insufficient","permitted"],"gold":"permitted","row_len":86,"state_len":21,"decide":85,"opts":[56,73,84],"row_ids_sha256":"a9ca040eb43849864b90251c8391bda862949c23d290de4302fb73caf1b7cbe8","oracle":{"probs":[0.011749336495995522,0.014542786404490471,0.9737078547477722],"logits_raw":[-6.247186660766602,-5.73397970199585,4.3810930252075195],"argmax":2,"top2_margin":0.959165096282959,"near_tie":false},"graph":{"p":[0.011850500479340553,0.014685065485537052,0.9734644293785095],"max_abs_dp":0.0002434253692626953,"argmax_equal":true}}]},{"id":"semif_31eb4533d2c89959ec46","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The homeowner gave an early permission decision, but neither its content nor any later decisions are available.","questions":{"decision":{"type":"choice","instructions":"Use the homeowner's latest permission decision; a withdrawal cancels the earlier grant. May the surveyor enter the attic?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=ca2e0b043985fa49bcd2","provenance":{"family":"rule_application","variant":"missing","group_id":"ca2e0b043985fa49bcd2","label":0,"option_order":["insufficient","prohibited","permitted"]},"state_tokens":20,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","prohibited","permitted"],"gold":"insufficient","row_len":85,"state_len":20,"decide":84,"opts":[61,72,83],"row_ids_sha256":"80431c26592e81f0f6537fe5efdec3f9314103b55ed0e2b47fe3f630e7d256e5","oracle":{"probs":[0.9321109652519226,0.022204309701919556,0.04568484425544739],"logits_raw":[0.9328268766403198,-8.058979988098145,-6.323062896728516],"argmax":0,"top2_margin":0.8864260911941528,"near_tie":false},"graph":{"p":[0.9315767288208008,0.022137312218546867,0.046285927295684814],"max_abs_dp":0.0006010830402374268,"argmax_equal":true}}]},{"id":"semif_0aa24a6701ac9c56c4a0","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The speaker authorized streaming, then explicitly withdrew the authorization before the event. Candidate B: The speaker authorized streaming and the complete record confirms it remains in force.","questions":{"decision":{"type":"choice","instructions":"Select the candidate with currently valid streaming permission; a withdrawal cancels an earlier grant.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=original group=85051ba3c30277824ebe","provenance":{"family":"candidate_selection","variant":"original","group_id":"85051ba3c30277824ebe","label":1,"option_order":["insufficient","B","A"]},"state_tokens":35,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","B","A"],"gold":"B","row_len":77,"state_len":35,"decide":76,"opts":[63,69,75],"row_ids_sha256":"4a51457277d763de2c7809cf44ad3e992f8d993d075c897ca6f6df74cd945df0","oracle":{"probs":[0.03721749782562256,0.9515457153320312,0.011236771009862423],"logits_raw":[-3.4274516105651855,4.3712992668151855,-6.308907508850098],"argmax":1,"top2_margin":0.9143282175064087,"near_tie":false},"graph":{"p":[0.038086261600255966,0.9505219459533691,0.011391890235245228],"max_abs_dp":0.0010237693786621094,"argmax_equal":true}}]},{"id":"semif_796d5c0da6eaadae6a3f","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: The speaker authorized streaming, then explicitly withdrew the authorization before the event. Candidate B: The speaker authorized streaming and the complete record confirms it remains in force.","questions":{"decision":{"type":"choice","instructions":"Select the candidate documenting an authorization that was later withdrawn.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=criterion_reversal group=85051ba3c30277824ebe","provenance":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"85051ba3c30277824ebe","label":1,"option_order":["B","A","insufficient"]},"state_tokens":35,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","A","insufficient"],"gold":"A","row_len":71,"state_len":35,"decide":70,"opts":[52,58,69],"row_ids_sha256":"6b1157fcd9207f1607d464589bb17d4c3727d9d14a3142b920c9792cba767fab","oracle":{"probs":[0.0006631516735069454,0.9882605075836182,0.011076344177126884],"logits_raw":[-10.993181228637695,6.587099552154541,-4.218794345855713],"argmax":1,"top2_margin":0.9771841764450073,"near_tie":false},"graph":{"p":[0.0006753102643415332,0.9881728291511536,0.011151820421218872],"max_abs_dp":8.767843246459961e-05,"argmax_equal":true}}]},{"id":"semif_599b3092b6fd18312d23","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: Streaming was authorized and the complete record says permission remains in force. Candidate B: Streaming was authorized, then explicitly withdrawn.","questions":{"decision":{"type":"choice","instructions":"Select the candidate with currently valid streaming permission; a withdrawal cancels an earlier grant.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=evidence_change group=85051ba3c30277824ebe","provenance":{"family":"candidate_selection","variant":"evidence_change","group_id":"85051ba3c30277824ebe","label":2,"option_order":["B","insufficient","A"]},"state_tokens":28,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","insufficient","A"],"gold":"A","row_len":70,"state_len":28,"decide":69,"opts":[51,62,68],"row_ids_sha256":"292f0b6036f22fb0c49d452e432770fd267a20c3c9ea25ee293340b902183c02","oracle":{"probs":[0.0028383079916238785,0.013951881788671017,0.9832097291946411],"logits_raw":[-8.174848556518555,-4.343440532684326,5.894803047180176],"argmax":2,"top2_margin":0.9692578315734863,"near_tie":false},"graph":{"p":[0.002856932580471039,0.014048576354980469,0.9830945134162903],"max_abs_dp":0.00011521577835083008,"argmax_equal":true}}]},{"id":"semif_960a4bcd27363e6db4b9","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A: A permission form is mentioned but its terms and later changes are missing. Candidate B: The event name is supplied without permission history.","questions":{"decision":{"type":"choice","instructions":"Select the candidate with currently valid streaming permission; a withdrawal cancels an earlier grant.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=85051ba3c30277824ebe","provenance":{"family":"candidate_selection","variant":"missing","group_id":"85051ba3c30277824ebe","label":2,"option_order":["B","A","insufficient"]},"state_tokens":30,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","A","insufficient"],"gold":"insufficient","row_len":72,"state_len":30,"decide":71,"opts":[53,59,70],"row_ids_sha256":"70e48eff11d8cf11e89e440e96b455c6749f11f1b37156063cc0375dad62556e","oracle":{"probs":[0.11635752022266388,0.1398671418428421,0.7437753677368164],"logits_raw":[-3.307591438293457,-2.864816665649414,1.1558035612106323],"argmax":2,"top2_margin":0.6039082407951355,"near_tie":false},"graph":{"p":[0.11702697724103928,0.14108102023601532,0.7418920397758484],"max_abs_dp":0.0018833279609680176,"argmax_equal":true}}]},{"id":"semif_c6a411e7f2bbdf377173","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The director gave Lina authority to approve exhibit labels and explicitly forbade further delegation. Lina then told Mo to approve them. The rule states an attempted forbidden delegation grants no authority; Mo has no other grant.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: Mo has authority to approve exhibit labels.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=e7798dab5f4f121cab77","provenance":{"family":"evidence_interpretation","variant":"original","group_id":"e7798dab5f4f121cab77","label":0,"option_order":["contradicted","insufficient","supported"]},"state_tokens":44,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["contradicted","insufficient","supported"],"gold":"contradicted","row_len":90,"state_len":44,"decide":89,"opts":[68,79,88],"row_ids_sha256":"cf8a3a3a96ba7f3e922fed7a60f9da754b2dba774dd1eb584fb394a988c1cf51","oracle":{"probs":[0.8773191571235657,0.10780506581068039,0.014875800348818302],"logits_raw":[1.9971923828125,-3.0472025871276855,-7.81259822845459],"argmax":0,"top2_margin":0.7695140838623047,"near_tie":false},"graph":{"p":[0.8767099380493164,0.10826631635427475,0.015023669227957726],"max_abs_dp":0.0006092190742492676,"argmax_equal":true}}]},{"id":"semif_e332aa927d0c2ed044aa","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The director gave Lina authority to approve exhibit labels and explicitly forbade further delegation. Lina then told Mo to approve them. The rule states an attempted forbidden delegation grants no authority; Mo has no other grant.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: Lina has authority to approve exhibit labels.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=e7798dab5f4f121cab77","provenance":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"e7798dab5f4f121cab77","label":2,"option_order":["contradicted","insufficient","supported"]},"state_tokens":44,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["contradicted","insufficient","supported"],"gold":"supported","row_len":91,"state_len":44,"decide":90,"opts":[69,80,89],"row_ids_sha256":"a44b1f97cc3132edc699ea5d2fc8c1c7ce9f2e44dd2fd7994a889c27fb022d82","oracle":{"probs":[0.023517033085227013,0.07549149543046951,0.9009914398193359],"logits_raw":[-7.003950119018555,-4.197786331176758,1.7679563760757446],"argmax":2,"top2_margin":0.825499951839447,"near_tie":false},"graph":{"p":[0.023835672065615654,0.0759940817952156,0.9001702070236206],"max_abs_dp":0.000821232795715332,"argmax_equal":true}}]},{"id":"semif_a0128e551d095f3449f8","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The director gives Mo direct authority to approve exhibit labels; the record confirms this grant is valid.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: Mo has authority to approve exhibit labels.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=e7798dab5f4f121cab77","provenance":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"e7798dab5f4f121cab77","label":0,"option_order":["supported","insufficient","contradicted"]},"state_tokens":20,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","insufficient","contradicted"],"gold":"supported","row_len":66,"state_len":20,"decide":65,"opts":[42,53,64],"row_ids_sha256":"36eab7e30545eeac5b9e5b4beb40f6daa9ba858e57ea85e9c999f71193c3ac38","oracle":{"probs":[0.9748297333717346,0.023023555055260658,0.002146761631593108],"logits_raw":[3.9498538970947266,-5.062595367431641,-10.771087646484375],"argmax":0,"top2_margin":0.9518061876296997,"near_tie":false},"graph":{"p":[0.9745994210243225,0.0232272669672966,0.002173322020098567],"max_abs_dp":0.00023031234741210938,"argmax_equal":true}}]},{"id":"semif_46d6a0731bd288c370f0","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Mo is named in correspondence about labels, but no authority grants or delegation terms are supplied.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: Mo has authority to approve exhibit labels.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=e7798dab5f4f121cab77","provenance":{"family":"evidence_interpretation","variant":"missing","group_id":"e7798dab5f4f121cab77","label":0,"option_order":["insufficient","contradicted","supported"]},"state_tokens":19,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","contradicted","supported"],"gold":"insufficient","row_len":65,"state_len":19,"decide":64,"opts":[43,54,63],"row_ids_sha256":"d406e1f4c4ab00d156baec8357b5c43c6afb8f5c7865d408255c3a5542e246fc","oracle":{"probs":[0.8540967106819153,0.13801006972789764,0.007893184199929237],"logits_raw":[1.0997811555862427,-3.285768985748291,-10.170265197753906],"argmax":0,"top2_margin":0.7160866260528564,"near_tie":false},"graph":{"p":[0.8593226671218872,0.13268406689167023,0.007993333972990513],"max_abs_dp":0.005326002836227417,"argmax_equal":true}}]},{"id":"semif_21ca7a8b970e36b8e20f","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The producer authorizes the stage manager to approve props. The stage manager passes the task to an assistant, who has no other approval authority.","questions":{"decision":{"type":"choice","instructions":"The stage manager may approve props but may not delegate that authority. Only authorized approvers may approve. May the assistant approve a prop?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=928ebc7817a8f1cac3e4","provenance":{"family":"rule_application","variant":"original","group_id":"928ebc7817a8f1cac3e4","label":0,"option_order":["prohibited","insufficient","permitted"]},"state_tokens":29,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","insufficient","permitted"],"gold":"prohibited","row_len":97,"state_len":29,"decide":96,"opts":[67,84,95],"row_ids_sha256":"82acce299a211d60a901ea19230fb829f5e91fd15c3ae0619a362fd7cb3f8f4d","oracle":{"probs":[0.5799847841262817,0.34583261609077454,0.07418261468410492],"logits_raw":[-0.8278971910476685,-2.0719380378723145,-5.775871753692627],"argmax":0,"top2_margin":0.2341521680355072,"near_tie":false},"graph":{"p":[0.584053635597229,0.34288910031318665,0.07305716723203659],"max_abs_dp":0.004068851470947266,"argmax_equal":true}}]},{"id":"semif_4aa83af95bb69e90bc2d","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The producer authorizes the stage manager to approve props. The stage manager passes the task to an assistant, who has no other approval authority.","questions":{"decision":{"type":"choice","instructions":"The stage manager may delegate prop approval to an assistant, and doing so grants the assistant approval authority. May this assistant approve?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=928ebc7817a8f1cac3e4","provenance":{"family":"rule_application","variant":"criterion_reversal","group_id":"928ebc7817a8f1cac3e4","label":0,"option_order":["permitted","prohibited","insufficient"]},"state_tokens":29,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["permitted","prohibited","insufficient"],"gold":"permitted","row_len":95,"state_len":29,"decide":94,"opts":[65,76,93],"row_ids_sha256":"ecccf6e2a904262081a3c0b7fc002c8c8a37bba342e892727b64848db2e61538","oracle":{"probs":[0.4745873212814331,0.11533106863498688,0.4100815951824188],"logits_raw":[-1.676340103149414,-5.080031394958496,-2.027838706970215],"argmax":0,"top2_margin":0.06450572609901428,"near_tie":false},"graph":{"p":[0.4750696122646332,0.1157684475183487,0.4091619551181793],"max_abs_dp":0.000919640064239502,"argmax_equal":true}}]},{"id":"semif_cdfa9eed51fafcf09b72","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The producer directly authorizes the assistant to approve props.","questions":{"decision":{"type":"choice","instructions":"The stage manager may approve props but may not delegate that authority. Only authorized approvers may approve. May the assistant approve a prop?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=928ebc7817a8f1cac3e4","provenance":{"family":"rule_application","variant":"evidence_change","group_id":"928ebc7817a8f1cac3e4","label":2,"option_order":["insufficient","prohibited","permitted"]},"state_tokens":12,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","prohibited","permitted"],"gold":"permitted","row_len":80,"state_len":12,"decide":79,"opts":[56,67,78],"row_ids_sha256":"13eec12cc81d5666a23b9a8ddb3a11545dea6a34bbfc30fcc7d2680f88e103ca","oracle":{"probs":[0.06770098209381104,0.019198626279830933,0.9131004214286804],"logits_raw":[-3.2429850101470947,-6.275238037109375,3.0169434547424316],"argmax":2,"top2_margin":0.8453994393348694,"near_tie":false},"graph":{"p":[0.06914174556732178,0.019548075273633003,0.9113101363182068],"max_abs_dp":0.0017902851104736328,"argmax_equal":true}}]},{"id":"semif_d51b05bfb9d6d7f09331","source":"semif_authored144","request":{"model":"kev-0.8b","state":"An assistant works on the production, but no approval grants or delegation history are provided.","questions":{"decision":{"type":"choice","instructions":"The stage manager may approve props but may not delegate that authority. Only authorized approvers may approve. May the assistant approve a prop?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=928ebc7817a8f1cac3e4","provenance":{"family":"rule_application","variant":"missing","group_id":"928ebc7817a8f1cac3e4","label":1,"option_order":["prohibited","insufficient","permitted"]},"state_tokens":18,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","insufficient","permitted"],"gold":"insufficient","row_len":86,"state_len":18,"decide":85,"opts":[56,73,84],"row_ids_sha256":"04983a6ef787f23ae58909e60a5f062e8a1c4bf75314020e02e97988c219f161","oracle":{"probs":[0.3538706302642822,0.5917116403579712,0.05441771447658539],"logits_raw":[-2.256734848022461,-1.0198134183883667,-6.761441707611084],"argmax":1,"top2_margin":0.23784101009368896,"near_tie":false},"graph":{"p":[0.35379931330680847,0.5913194417953491,0.0548812635242939],"max_abs_dp":0.00046354904770851135,"argmax_equal":true}}]},{"id":"semif_fdd902cc6e65fad7cb4f","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The rule permits direct delegation by the director but forbids any further delegation. Candidate A: I received prop approval authority directly from the director. Candidate B: I received it only from A, and have no other grant.","questions":{"decision":{"type":"choice","instructions":"Select the candidate holding valid prop approval authority under the stated delegation rule.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=53b0770ec36835b81d3a","provenance":{"family":"candidate_selection","variant":"original","group_id":"53b0770ec36835b81d3a","label":1,"option_order":["B","A","insufficient"]},"state_tokens":45,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","A","insufficient"],"gold":"A","row_len":84,"state_len":45,"decide":83,"opts":[65,71,82],"row_ids_sha256":"ae63b178afb5afc8053733290af93d355c5849bea87bdcc29c81b2d2106aeb50","oracle":{"probs":[0.08008323609828949,0.8453215956687927,0.07459510117769241],"logits_raw":[-3.480044364929199,2.1901748180389404,-3.6508538722991943],"argmax":1,"top2_margin":0.765238344669342,"near_tie":false},"graph":{"p":[0.07936764508485794,0.8455126881599426,0.07511970400810242],"max_abs_dp":0.0007155910134315491,"argmax_equal":true}}]},{"id":"semif_42384af9ae56beb37ff5","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The rule permits direct delegation by the director but forbids any further delegation. Candidate A: I received prop approval authority directly from the director. Candidate B: I received it only from A, and have no other grant.","questions":{"decision":{"type":"choice","instructions":"Select the candidate whose claimed authority comes only through a prohibited second delegation.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=53b0770ec36835b81d3a","provenance":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"53b0770ec36835b81d3a","label":2,"option_order":["insufficient","A","B"]},"state_tokens":45,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","A","B"],"gold":"B","row_len":84,"state_len":45,"decide":83,"opts":[70,76,82],"row_ids_sha256":"6348c32d778a80c356e9f5c6dbba9e02b43413124f0110d4296f367ae91468fd","oracle":{"probs":[0.1357346475124359,0.027728796005249023,0.8365365862846375],"logits_raw":[-1.6776618957519531,-5.4990234375,2.697904586791992],"argmax":2,"top2_margin":0.7008019685745239,"near_tie":false},"graph":{"p":[0.13252530992031097,0.026640400290489197,0.8408343195915222],"max_abs_dp":0.004297733306884766,"argmax_equal":true}}]},{"id":"semif_cdf7f428857e1b841cb3","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The rule permits direct delegation by the director but forbids further delegation. Candidate A: My only claimed authority came from B. Candidate B: The director directly granted me prop approval authority.","questions":{"decision":{"type":"choice","instructions":"Select the candidate holding valid prop approval authority under the stated delegation rule.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=53b0770ec36835b81d3a","provenance":{"family":"candidate_selection","variant":"evidence_change","group_id":"53b0770ec36835b81d3a","label":2,"option_order":["A","insufficient","B"]},"state_tokens":38,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["A","insufficient","B"],"gold":"B","row_len":77,"state_len":38,"decide":76,"opts":[58,69,75],"row_ids_sha256":"2d6afc0a565a3b7d224782d77702d4525169361eec87af71f44737319b83a216","oracle":{"probs":[0.041775625199079514,0.15995649993419647,0.7982678413391113],"logits_raw":[-5.239965438842773,-2.009629487991333,1.8581974506378174],"argmax":2,"top2_margin":0.6383113265037537,"near_tie":false},"graph":{"p":[0.04137624427676201,0.15584513545036316,0.8027786016464233],"max_abs_dp":0.004510760307312012,"argmax_equal":true}}]},{"id":"semif_774f7f9245f83c796d44","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The director's delegation rule is supplied, but Candidate A and Candidate B provide only their names and no authority histories.","questions":{"decision":{"type":"choice","instructions":"Select the candidate holding valid prop approval authority under the stated delegation rule.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=53b0770ec36835b81d3a","provenance":{"family":"candidate_selection","variant":"missing","group_id":"53b0770ec36835b81d3a","label":2,"option_order":["B","A","insufficient"]},"state_tokens":24,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","A","insufficient"],"gold":"insufficient","row_len":63,"state_len":24,"decide":62,"opts":[44,50,61],"row_ids_sha256":"5de93924ff3903bd378397697a1c5ffbba3a87870fdea536c223378f3280cd60","oracle":{"probs":[0.02820395864546299,0.03614291548728943,0.9356530904769897],"logits_raw":[-5.489147186279297,-4.892402648925781,2.9363160133361816],"argmax":2,"top2_margin":0.8995101451873779,"near_tie":false},"graph":{"p":[0.028758833184838295,0.03683023154735565,0.9344109296798706],"max_abs_dp":0.0012421607971191406,"argmax_equal":true}}]},{"id":"semif_92ec2ffa3dd46ada1335","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The return policy accepts faulty lamps despite opened packaging, but explicitly excludes water damage even when the lamp is faulty. This opened lamp is faulty because water entered it. No other cause is present.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: this lamp qualifies for return under the stated policy.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=056cc5b726d6e8300a77","provenance":{"family":"evidence_interpretation","variant":"original","group_id":"056cc5b726d6e8300a77","label":0,"option_order":["contradicted","insufficient","supported"]},"state_tokens":39,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["contradicted","insufficient","supported"],"gold":"contradicted","row_len":87,"state_len":39,"decide":86,"opts":[65,76,85],"row_ids_sha256":"2c147a428d2fbba2b8dc14ca515b31c65e25f2f099e3042166d6c5a0aba8ff9a","oracle":{"probs":[0.985283613204956,0.012348263524472713,0.002368178451433778],"logits_raw":[5.183025360107422,-5.354063987731934,-9.327401161193848],"argmax":0,"top2_margin":0.9729353785514832,"near_tie":false},"graph":{"p":[0.984870195388794,0.012665124610066414,0.0024647268000990152],"max_abs_dp":0.0004134178161621094,"argmax_equal":true}}]},{"id":"semif_6149a17bc154f9c5b4a0","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The return policy accepts faulty lamps despite opened packaging, but explicitly excludes water damage even when the lamp is faulty. This opened lamp is faulty because water entered it. No other cause is present.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the lamp has a fault.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=056cc5b726d6e8300a77","provenance":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"056cc5b726d6e8300a77","label":0,"option_order":["supported","insufficient","contradicted"]},"state_tokens":39,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","insufficient","contradicted"],"gold":"supported","row_len":83,"state_len":39,"decide":82,"opts":[59,70,81],"row_ids_sha256":"74e5de1551119a51dcd0d1b71b461bec7e136fdc88241dc08173270d59a85457","oracle":{"probs":[0.9516547322273254,0.0400046706199646,0.008340579457581043],"logits_raw":[2.3377528190612793,-5.287515640258789,-9.059873580932617],"argmax":0,"top2_margin":0.9116500616073608,"near_tie":false},"graph":{"p":[0.9501552581787109,0.041142966598272324,0.008701751939952374],"max_abs_dp":0.001499474048614502,"argmax_equal":true}}]},{"id":"semif_be866e1dbec5d42d9423","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The return policy accepts faulty lamps despite opened packaging, but excludes water damage even when faulty. This opened lamp has a manufacturing wiring fault, with no water damage or other exclusion.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: this lamp qualifies for return under the stated policy.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=056cc5b726d6e8300a77","provenance":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"056cc5b726d6e8300a77","label":2,"option_order":["contradicted","insufficient","supported"]},"state_tokens":36,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["contradicted","insufficient","supported"],"gold":"supported","row_len":84,"state_len":36,"decide":83,"opts":[62,73,82],"row_ids_sha256":"87ae4283760fb61b79b6f2b61184d3c3d28270449cb047531d0c3cd793f159bc","oracle":{"probs":[0.014089873060584068,0.03271709755063057,0.9531930685043335],"logits_raw":[-6.88315486907959,-4.856198310852051,3.2568087577819824],"argmax":2,"top2_margin":0.920475959777832,"near_tie":false},"graph":{"p":[0.014448880217969418,0.033264029771089554,0.9522870182991028],"max_abs_dp":0.0009060502052307129,"argmax_equal":true}}]},{"id":"semif_97d2ed2591fc7638806c","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The return policy accepts faulty lamps despite opened packaging but excludes water damage even when faulty. The lamp is faulty, but the cause and presence of water damage are omitted.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: this lamp qualifies for return under the stated policy.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=056cc5b726d6e8300a77","provenance":{"family":"evidence_interpretation","variant":"missing","group_id":"056cc5b726d6e8300a77","label":0,"option_order":["insufficient","supported","contradicted"]},"state_tokens":34,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","supported","contradicted"],"gold":"insufficient","row_len":82,"state_len":34,"decide":81,"opts":[60,69,80],"row_ids_sha256":"7f68db97f8e9455d54b00963c63c784dcee0e183df2cb9bd1d89b5fec7adf0a7","oracle":{"probs":[0.8272495269775391,0.15039554238319397,0.02235495299100876],"logits_raw":[-1.162671685218811,-5.264595985412598,-9.851058959960938],"argmax":0,"top2_margin":0.6768540143966675,"near_tie":false},"graph":{"p":[0.8209860324859619,0.15648111701011658,0.022532783448696136],"max_abs_dp":0.0062634944915771484,"argmax_equal":true}}]},{"id":"semif_4533dd91267b4d815690","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The opened paint tin has a manufacturing defect and also contains added solvent. Both facts are confirmed.","questions":{"decision":{"type":"choice","instructions":"Normally reject opened paint. Accept it for a manufacturing defect, except that added solvent always makes it ineligible. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=e3ce8ec496dbc9974f94","provenance":{"family":"rule_application","variant":"original","group_id":"e3ce8ec496dbc9974f94","label":2,"option_order":["permitted","insufficient","prohibited"]},"state_tokens":20,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["permitted","insufficient","prohibited"],"gold":"prohibited","row_len":88,"state_len":20,"decide":87,"opts":[58,75,86],"row_ids_sha256":"ea89f75c9a458f8f673e611e19173e2b13c6f4f56eaef13e99bf0de39975c485","oracle":{"probs":[0.011629368178546429,0.011494539678096771,0.9768760800361633],"logits_raw":[-5.278443336486816,-5.306500434875488,5.382346153259277],"argmax":2,"top2_margin":0.9652467370033264,"near_tie":false},"graph":{"p":[0.011593603529036045,0.011522633954882622,0.9768837094306946],"max_abs_dp":3.5764649510383606e-05,"argmax_equal":true}}]},{"id":"semif_d880e65fef5e2993fdd0","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The opened paint tin has a manufacturing defect and also contains added solvent. Both facts are confirmed.","questions":{"decision":{"type":"choice","instructions":"Accept defective paint even if opened or mixed with solvent; neither condition overrides the defect exception. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=e3ce8ec496dbc9974f94","provenance":{"family":"rule_application","variant":"criterion_reversal","group_id":"e3ce8ec496dbc9974f94","label":0,"option_order":["permitted","insufficient","prohibited"]},"state_tokens":20,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["permitted","insufficient","prohibited"],"gold":"permitted","row_len":85,"state_len":20,"decide":84,"opts":[55,72,83],"row_ids_sha256":"4c38e3d632ea249b2707d7437b7f3e1c8c74f005d93716a725f147ed71ca348d","oracle":{"probs":[0.42083075642585754,0.2929432690143585,0.28622591495513916],"logits_raw":[-1.2043951749801636,-2.075990915298462,-2.131805419921875],"argmax":0,"top2_margin":0.12788748741149902,"near_tie":false},"graph":{"p":[0.4166029095649719,0.295366108417511,0.28803104162216187],"max_abs_dp":0.00422784686088562,"argmax_equal":true}}]},{"id":"semif_90494c76078ffc3c717a","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The opened tin has a manufacturing defect and the inspection confirms no solvent was added.","questions":{"decision":{"type":"choice","instructions":"Normally reject opened paint. Accept it for a manufacturing defect, except that added solvent always makes it ineligible. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=e3ce8ec496dbc9974f94","provenance":{"family":"rule_application","variant":"evidence_change","group_id":"e3ce8ec496dbc9974f94","label":0,"option_order":["permitted","insufficient","prohibited"]},"state_tokens":17,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["permitted","insufficient","prohibited"],"gold":"permitted","row_len":85,"state_len":17,"decide":84,"opts":[55,72,83],"row_ids_sha256":"729a0a3736e7b5badc786f550b8db45e18d0ba23080d6d0f3e0ff162f554203b","oracle":{"probs":[0.8819436430931091,0.09038404375314713,0.027672328054904938],"logits_raw":[1.9425396919250488,-3.5385875701904297,-6.386471748352051],"argmax":0,"top2_margin":0.7915595769882202,"near_tie":false},"graph":{"p":[0.8815334439277649,0.09043607115745544,0.028030432760715485],"max_abs_dp":0.0004101991653442383,"argmax_equal":true}}]},{"id":"semif_9ac5cc82c4126c91174a","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The opened tin is defective, but whether solvent was added is not known.","questions":{"decision":{"type":"choice","instructions":"Normally reject opened paint. Accept it for a manufacturing defect, except that added solvent always makes it ineligible. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=e3ce8ec496dbc9974f94","provenance":{"family":"rule_application","variant":"missing","group_id":"e3ce8ec496dbc9974f94","label":1,"option_order":["permitted","insufficient","prohibited"]},"state_tokens":16,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["permitted","insufficient","prohibited"],"gold":"insufficient","row_len":84,"state_len":16,"decide":83,"opts":[54,71,82],"row_ids_sha256":"463422fa5f3962cad72c85384c5c90010c92a34d3e518210a4ad10d055cf7ac9","oracle":{"probs":[0.12228090316057205,0.8576568365097046,0.020062316209077835],"logits_raw":[-3.126605987548828,1.5600980520248413,-7.475488185882568],"argmax":1,"top2_margin":0.7353759407997131,"near_tie":false},"graph":{"p":[0.12326367944478989,0.8564820885658264,0.02025429531931877],"max_abs_dp":0.0011747479438781738,"argmax_equal":true}}]},{"id":"semif_386e10b1c1ce906473a4","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Accept late submissions for documented outages, except that missing signatures always disqualify. Candidate A: My submission was late because of a documented outage and includes the required signature. Candidate B: Mine was late because of a documented outage but lacks the signature.","questions":{"decision":{"type":"choice","instructions":"Select the submission qualifying under the stated exception and exclusion.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=a5b234c2e4fb304f1071","provenance":{"family":"candidate_selection","variant":"original","group_id":"a5b234c2e4fb304f1071","label":1,"option_order":["B","A","insufficient"]},"state_tokens":52,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","A","insufficient"],"gold":"A","row_len":88,"state_len":52,"decide":87,"opts":[69,75,86],"row_ids_sha256":"42fc143a76a4cd231e12737c04987d5778a2bd796b599ba88377ddc1a72f529f","oracle":{"probs":[0.002506383927538991,0.9900159239768982,0.007477672770619392],"logits_raw":[-8.245689392089844,6.139795303344727,-5.615683078765869],"argmax":1,"top2_margin":0.9825382232666016,"near_tie":false},"graph":{"p":[0.0025260169059038162,0.9899561405181885,0.007517870049923658],"max_abs_dp":5.97834587097168e-05,"argmax_equal":true}}]},{"id":"semif_4da6aedc1139224eeff2","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Accept late submissions for documented outages, except that missing signatures always disqualify. Candidate A: My submission was late because of a documented outage and includes the required signature. Candidate B: Mine was late because of a documented outage but lacks the signature.","questions":{"decision":{"type":"choice","instructions":"Select the submission disqualified specifically by the signature exclusion despite the outage exception.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=a5b234c2e4fb304f1071","provenance":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"a5b234c2e4fb304f1071","label":1,"option_order":["insufficient","B","A"]},"state_tokens":52,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","B","A"],"gold":"B","row_len":91,"state_len":52,"decide":90,"opts":[77,83,89],"row_ids_sha256":"f0a1c869f6ff8d55b85c5bf3192835710e89e5cc0251b82014f4a7ec311b65b1","oracle":{"probs":[0.01981699839234352,0.9781287312507629,0.00205425382591784],"logits_raw":[-3.2290420532226562,6.152390003204346,-8.682661056518555],"argmax":1,"top2_margin":0.9583117365837097,"near_tie":false},"graph":{"p":[0.0198680367320776,0.978022575378418,0.002109447494149208],"max_abs_dp":0.0001061558723449707,"argmax_equal":true}}]},{"id":"semif_a22dab8ba073ff6cdb8d","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Accept late submissions for documented outages, except that missing signatures always disqualify. Candidate A: My late outage-related submission lacks the signature. Candidate B: My late outage-related submission has the signature. Both outages are documented.","questions":{"decision":{"type":"choice","instructions":"Select the submission qualifying under the stated exception and exclusion.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=a5b234c2e4fb304f1071","provenance":{"family":"candidate_selection","variant":"evidence_change","group_id":"a5b234c2e4fb304f1071","label":1,"option_order":["insufficient","B","A"]},"state_tokens":48,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","B","A"],"gold":"B","row_len":84,"state_len":48,"decide":83,"opts":[70,76,82],"row_ids_sha256":"1ed40c2721fdc732d6bddaf5f067d61d403eef8dcbe6450ee43d39b71eb2e3a7","oracle":{"probs":[0.21431751549243927,0.6948567628860474,0.09082573652267456],"logits_raw":[-4.20926570892334,-1.3791563510894775,-6.274898052215576],"argmax":1,"top2_margin":0.4805392622947693,"near_tie":false},"graph":{"p":[0.21730849146842957,0.6901697516441345,0.0925218015909195],"max_abs_dp":0.004687011241912842,"argmax_equal":true}}]},{"id":"semif_42c54fa2ad45743e24d8","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Accept late submissions for documented outages, except that missing signatures always disqualify. Candidate A and Candidate B report late submissions but provide neither outage documentation nor signature information.","questions":{"decision":{"type":"choice","instructions":"Select the submission qualifying under the stated exception and exclusion.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=a5b234c2e4fb304f1071","provenance":{"family":"candidate_selection","variant":"missing","group_id":"a5b234c2e4fb304f1071","label":1,"option_order":["A","insufficient","B"]},"state_tokens":35,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["A","insufficient","B"],"gold":"insufficient","row_len":71,"state_len":35,"decide":70,"opts":[52,63,69],"row_ids_sha256":"e4258e0adc2b968bcc7260a513160bc78fb2d036d3e72f641b18031872c0d1f1","oracle":{"probs":[0.02198265679180622,0.9308003187179565,0.04721706360578537],"logits_raw":[-6.823294162750244,2.1892662048339844,-4.983865261077881],"argmax":1,"top2_margin":0.8835832476615906,"near_tie":false},"graph":{"p":[0.022143887355923653,0.9303323030471802,0.04752378165721893],"max_abs_dp":0.0004680156707763672,"argmax_equal":true}}]},{"id":"semif_f3d13cf184f916677def","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The desk log says the crate is in storage. The warehouse inventory says it has left storage. Both reports are current. The supplied protocol says the warehouse inventory controls location decisions whenever these two sources conflict.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: under the supplied protocol, the crate is currently in storage.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=d142f7fd0efa0eddc788","provenance":{"family":"evidence_interpretation","variant":"original","group_id":"d142f7fd0efa0eddc788","label":0,"option_order":["contradicted","insufficient","supported"]},"state_tokens":41,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["contradicted","insufficient","supported"],"gold":"contradicted","row_len":91,"state_len":41,"decide":90,"opts":[69,80,89],"row_ids_sha256":"b0caeaff47af26f7f5781ecd5e65f230c3f4dbf1af5374c4a366aaacceef0458","oracle":{"probs":[0.960503339767456,0.03714030608534813,0.002356334589421749],"logits_raw":[4.317008972167969,-3.5092811584472656,-10.144193649291992],"argmax":0,"top2_margin":0.9233630299568176,"near_tie":false},"graph":{"p":[0.9598430395126343,0.0377342626452446,0.0024227167014032602],"max_abs_dp":0.0006603002548217773,"argmax_equal":true}}]},{"id":"semif_1105577da4c8dad4609d","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The desk log says the crate is in storage. The warehouse inventory says it has left storage. Both reports are current. The supplied protocol says the warehouse inventory controls location decisions whenever these two sources conflict.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the desk log reports the crate in storage.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=d142f7fd0efa0eddc788","provenance":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"d142f7fd0efa0eddc788","label":2,"option_order":["contradicted","insufficient","supported"]},"state_tokens":41,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["contradicted","insufficient","supported"],"gold":"supported","row_len":88,"state_len":41,"decide":87,"opts":[66,77,86],"row_ids_sha256":"53d83633f39f82457b3564b19642c7cc94fdd43982628d1b6f5168b37222a96c","oracle":{"probs":[0.22226007282733917,0.18289987742900848,0.5948400497436523],"logits_raw":[-3.967597484588623,-4.436558723449707,-1.5989750623703003],"argmax":2,"top2_margin":0.37257999181747437,"near_tie":false},"graph":{"p":[0.23063714802265167,0.18982911109924316,0.5795336961746216],"max_abs_dp":0.015306353569030762,"argmax_equal":true}}]},{"id":"semif_0ebd299739302d8b057b","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The desk log says the crate has left storage; the warehouse inventory says it is in storage. Both reports are current, and the inventory controls conflicts under the protocol.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: under the supplied protocol, the crate is currently in storage.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=d142f7fd0efa0eddc788","provenance":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"d142f7fd0efa0eddc788","label":2,"option_order":["contradicted","insufficient","supported"]},"state_tokens":34,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["contradicted","insufficient","supported"],"gold":"supported","row_len":84,"state_len":34,"decide":83,"opts":[62,73,82],"row_ids_sha256":"9a5363fa5f3f4952c9fcf54541b48f8ce3193a1f5947e7a911699afddce6bfb7","oracle":{"probs":[0.5471832156181335,0.3265804648399353,0.12623631954193115],"logits_raw":[-2.5929839611053467,-3.834764003753662,-6.121764183044434],"argmax":0,"top2_margin":0.22060275077819824,"near_tie":false},"graph":{"p":[0.5528401732444763,0.3236195743083954,0.1235402524471283],"max_abs_dp":0.0056569576263427734,"argmax_equal":true}}]},{"id":"semif_eafc22c8c40df3932a8e","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The protocol gives the inventory priority, but the current inventory and desk-log entries for the crate are omitted.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: under the supplied protocol, the crate is currently in storage.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=d142f7fd0efa0eddc788","provenance":{"family":"evidence_interpretation","variant":"missing","group_id":"d142f7fd0efa0eddc788","label":2,"option_order":["supported","contradicted","insufficient"]},"state_tokens":22,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","contradicted","insufficient"],"gold":"insufficient","row_len":72,"state_len":22,"decide":71,"opts":[48,59,70],"row_ids_sha256":"d0394fc0852c9c7eccdc5a9d0de1167fa2fa1d0bbffb95507e2a5d39d93a1880","oracle":{"probs":[0.025724269449710846,0.02742128260433674,0.9468544721603394],"logits_raw":[-8.352001190185547,-8.198291778564453,0.3235185444355011],"argmax":2,"top2_margin":0.9194331765174866,"near_tie":false},"graph":{"p":[0.02637346088886261,0.028013532981276512,0.9456130266189575],"max_abs_dp":0.001241445541381836,"argmax_equal":true}}]},{"id":"semif_0aa1447680e4f25862bb","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The desk clerk reports the equipment deposit as paid. The signed ledger reports it as unpaid. Both reports are current.","questions":{"decision":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; in conflicts, the signed ledger overrides the desk clerk. May it be released?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=3476d3f4f32290fe57c3","provenance":{"family":"rule_application","variant":"original","group_id":"3476d3f4f32290fe57c3","label":1,"option_order":["insufficient","prohibited","permitted"]},"state_tokens":24,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","prohibited","permitted"],"gold":"prohibited","row_len":91,"state_len":24,"decide":90,"opts":[67,78,89],"row_ids_sha256":"56dfbc2b58c21d4fb61185686ebf61b0d85fcaaac0adeca64ca2e5cb0fc6c978","oracle":{"probs":[0.3138221502304077,0.5418198704719543,0.14435794949531555],"logits_raw":[-2.6196415424346924,-1.305680274963379,-4.488012313842773],"argmax":1,"top2_margin":0.22799772024154663,"near_tie":false},"graph":{"p":[0.31331729888916016,0.5426204204559326,0.1440623253583908],"max_abs_dp":0.0008005499839782715,"argmax_equal":true}}]},{"id":"semif_980e76145c925747220c","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The desk clerk reports the equipment deposit as paid. The signed ledger reports it as unpaid. Both reports are current.","questions":{"decision":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; under this alternative protocol, the desk clerk overrides the ledger. May it be released?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=3476d3f4f32290fe57c3","provenance":{"family":"rule_application","variant":"criterion_reversal","group_id":"3476d3f4f32290fe57c3","label":2,"option_order":["insufficient","prohibited","permitted"]},"state_tokens":24,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","prohibited","permitted"],"gold":"permitted","row_len":92,"state_len":24,"decide":91,"opts":[68,79,90],"row_ids_sha256":"3425fdaec8f59b7f4d95ae87809e5b2e522dabcb7cc62b92be444274d0dadb7e","oracle":{"probs":[0.055183831602334976,0.022996563464403152,0.9218195676803589],"logits_raw":[-2.6522388458251953,-4.758315086364746,4.122426509857178],"argmax":2,"top2_margin":0.8666357398033142,"near_tie":false},"graph":{"p":[0.05536825954914093,0.02284281514585018,0.9217888712882996],"max_abs_dp":0.00018442794680595398,"argmax_equal":true}}]},{"id":"semif_4aee37a6a155747766aa","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The desk clerk reports the deposit unpaid, while the signed ledger reports it paid. Both reports are current.","questions":{"decision":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; in conflicts, the signed ledger overrides the desk clerk. May it be released?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=3476d3f4f32290fe57c3","provenance":{"family":"rule_application","variant":"evidence_change","group_id":"3476d3f4f32290fe57c3","label":0,"option_order":["permitted","prohibited","insufficient"]},"state_tokens":22,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["permitted","prohibited","insufficient"],"gold":"permitted","row_len":89,"state_len":22,"decide":88,"opts":[59,70,87],"row_ids_sha256":"ffd666aacc6955c58f20d4cc62a060d61a0612101a0074f38b526f0e4b0057c5","oracle":{"probs":[0.9116050601005554,0.04669686034321785,0.041698139160871506],"logits_raw":[3.3476483821868896,-3.8020009994506836,-4.07441520690918],"argmax":0,"top2_margin":0.8649082183837891,"near_tie":false},"graph":{"p":[0.9114686846733093,0.04673686996102333,0.041794419288635254],"max_abs_dp":0.00013637542724609375,"argmax_equal":true}}]},{"id":"semif_6db17cc22bd656d78558","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The record mentions the desk clerk and ledger but supplies neither deposit status.","questions":{"decision":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; in conflicts, the signed ledger overrides the desk clerk. May it be released?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=3476d3f4f32290fe57c3","provenance":{"family":"rule_application","variant":"missing","group_id":"3476d3f4f32290fe57c3","label":2,"option_order":["prohibited","permitted","insufficient"]},"state_tokens":15,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","permitted","insufficient"],"gold":"insufficient","row_len":82,"state_len":15,"decide":81,"opts":[52,63,80],"row_ids_sha256":"7c0a2a68b07e4a137fc9db575af59d5283475e8e425154f6291fa442d470c095","oracle":{"probs":[0.21013669669628143,0.08963154256343842,0.7002317905426025],"logits_raw":[-1.7671465873718262,-3.81722354888916,1.1289030313491821],"argmax":2,"top2_margin":0.4900950789451599,"near_tie":false},"graph":{"p":[0.21013160049915314,0.09089477360248566,0.6989736557006836],"max_abs_dp":0.0012632310390472412,"argmax_equal":true}}]},{"id":"semif_ef3e0f8e657bf59f8a29","source":"semif_authored144","request":{"model":"kev-0.8b","state":"For this task, the current signed checklist controls readiness when it conflicts with chat. Candidate A: Chat says ready, but the signed checklist says not ready. Candidate B: Chat says not ready, but the signed checklist says ready.","questions":{"decision":{"type":"choice","instructions":"Select the candidate ready according to the controlling signed checklist.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=original group=c979db7416b206c1eb09","provenance":{"family":"candidate_selection","variant":"original","group_id":"c979db7416b206c1eb09","label":2,"option_order":["A","insufficient","B"]},"state_tokens":47,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["A","insufficient","B"],"gold":"B","row_len":83,"state_len":47,"decide":82,"opts":[64,75,81],"row_ids_sha256":"ef971f766aa0224e96a172095f7a5710922d9404b00b48ad827261cb85e30793","oracle":{"probs":[0.18707191944122314,0.26139143109321594,0.5515366196632385],"logits_raw":[-3.251271963119507,-2.4463858604431152,-0.6498144865036011],"argmax":2,"top2_margin":0.2901451885700226,"near_tie":false},"graph":{"p":[0.18623726069927216,0.26510271430015564,0.5486600399017334],"max_abs_dp":0.0037112832069396973,"argmax_equal":true}}]},{"id":"semif_f756e1cb04e11766341e","source":"semif_authored144","request":{"model":"kev-0.8b","state":"For this task, the current signed checklist controls readiness when it conflicts with chat. Candidate A: Chat says ready, but the signed checklist says not ready. Candidate B: Chat says not ready, but the signed checklist says ready.","questions":{"decision":{"type":"choice","instructions":"Under the alternative protocol, chat controls conflicts. Select the candidate ready according to chat.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=criterion_reversal group=c979db7416b206c1eb09","provenance":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"c979db7416b206c1eb09","label":1,"option_order":["insufficient","A","B"]},"state_tokens":47,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","A","B"],"gold":"A","row_len":89,"state_len":47,"decide":88,"opts":[75,81,87],"row_ids_sha256":"6dc89c7fd1363897a742ad9efd4427ad9121915d8e8d3bbec749d3daadd3997c","oracle":{"probs":[0.22178898751735687,0.697046160697937,0.08116486668586731],"logits_raw":[-2.6175615787506104,0.13766664266586304,-5.036229133605957],"argmax":1,"top2_margin":0.47525715827941895,"near_tie":false},"graph":{"p":[0.220899298787117,0.698333740234375,0.08076696842908859],"max_abs_dp":0.0012875795364379883,"argmax_equal":true}}]},{"id":"semif_d167e51bb1f499099645","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The signed checklist controls conflicts. Candidate A: The checklist says ready and chat says not ready. Candidate B: The checklist says not ready and chat says ready.","questions":{"decision":{"type":"choice","instructions":"Select the candidate ready according to the controlling signed checklist.","criteria":{"A":"Candidate A","B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=evidence_change group=c979db7416b206c1eb09","provenance":{"family":"candidate_selection","variant":"evidence_change","group_id":"c979db7416b206c1eb09","label":0,"option_order":["A","B","insufficient"]},"state_tokens":33,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["A","B","insufficient"],"gold":"A","row_len":69,"state_len":33,"decide":68,"opts":[50,56,67],"row_ids_sha256":"4745f581913bc4e29d689d61c30b2f16aca4d2bce8a8dc531b6c9ea99d0aa2f1","oracle":{"probs":[0.3331868648529053,0.187632754445076,0.4791804254055023],"logits_raw":[-2.8611555099487305,-4.2427496910095215,-1.9868600368499756],"argmax":2,"top2_margin":0.14599356055259705,"near_tie":false},"graph":{"p":[0.33311814069747925,0.18755042552947998,0.47933149337768555],"max_abs_dp":0.00015106797218322754,"argmax_equal":true}}]},{"id":"semif_674ca4e9346216255466","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The signed checklist controls conflicts, but Candidate A and Candidate B provide only project names without checklist or chat status.","questions":{"decision":{"type":"choice","instructions":"Select the candidate ready according to the controlling signed checklist.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=c979db7416b206c1eb09","provenance":{"family":"candidate_selection","variant":"missing","group_id":"c979db7416b206c1eb09","label":0,"option_order":["insufficient","B","A"]},"state_tokens":23,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","B","A"],"gold":"insufficient","row_len":59,"state_len":23,"decide":58,"opts":[45,51,57],"row_ids_sha256":"efc1e4177b6102966beb1b7dda52084cf2d7a7b7e10719df05411ae43d5addc4","oracle":{"probs":[0.9019967913627625,0.04058562591671944,0.0574176162481308],"logits_raw":[3.6157891750335693,-3.8458456993103027,-3.0110976696014404],"argmax":0,"top2_margin":0.8445791602134705,"near_tie":false},"graph":{"p":[0.9002598524093628,0.041080765426158905,0.05865932255983353],"max_abs_dp":0.0017369389533996582,"argmax_equal":true}}]},{"id":"semif_1df851a202994a07f123","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The agreed handover requires both the source files and the user guide. The delivery record confirms the source files arrived and explicitly says the guide was not delivered.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the agreed handover is complete.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=f7403e39d6796072491c","provenance":{"family":"evidence_interpretation","variant":"original","group_id":"f7403e39d6796072491c","label":0,"option_order":["contradicted","insufficient","supported"]},"state_tokens":32,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["contradicted","insufficient","supported"],"gold":"contradicted","row_len":77,"state_len":32,"decide":76,"opts":[55,66,75],"row_ids_sha256":"18e776677c03a3bf6a6f01a4802a9922a8fb1ca2eb60cb5ff18c4847e2c51fae","oracle":{"probs":[0.9839341044425964,0.012210285291075706,0.003855565097182989],"logits_raw":[4.97550630569458,-5.585321426391602,-8.358922004699707],"argmax":0,"top2_margin":0.9717237949371338,"near_tie":false},"graph":{"p":[0.9835630059242249,0.01249196007847786,0.003945149481296539],"max_abs_dp":0.00037109851837158203,"argmax_equal":true}}]},{"id":"semif_b5e6788d95b8395d2888","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The agreed handover requires both the source files and the user guide. The delivery record confirms the source files arrived and explicitly says the guide was not delivered.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the source files were delivered.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=f7403e39d6796072491c","provenance":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"f7403e39d6796072491c","label":2,"option_order":["insufficient","contradicted","supported"]},"state_tokens":32,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","contradicted","supported"],"gold":"supported","row_len":76,"state_len":32,"decide":75,"opts":[54,65,74],"row_ids_sha256":"acbcb34439ad0b4430c175b5b9d065cf8499b8b58bfc76c254b6eaa636934199","oracle":{"probs":[0.004545513540506363,0.0017992692301049829,0.9936552047729492],"logits_raw":[-7.4253997802734375,-9.655232429504395,5.536591529846191],"argmax":2,"top2_margin":0.9891096949577332,"near_tie":false},"graph":{"p":[0.0046126870438456535,0.0018359230598434806,0.9935513734817505],"max_abs_dp":0.00010383129119873047,"argmax_equal":true}}]},{"id":"semif_32f36cd0fbbfb3604586","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The agreed handover requires the source files and user guide. The delivery record confirms that both arrived.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the agreed handover is complete.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=f7403e39d6796072491c","provenance":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"f7403e39d6796072491c","label":0,"option_order":["supported","insufficient","contradicted"]},"state_tokens":21,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","insufficient","contradicted"],"gold":"supported","row_len":66,"state_len":21,"decide":65,"opts":[42,53,64],"row_ids_sha256":"6cced2be44ed60b120708b07b8da1a49ae98f780a689e29214a767cbbaf580dc","oracle":{"probs":[0.8223103880882263,0.17248377203941345,0.005205787252634764],"logits_raw":[1.617693543434143,-2.140110731124878,-10.562565803527832],"argmax":0,"top2_margin":0.6498266458511353,"near_tie":false},"graph":{"p":[0.8247637748718262,0.17001566290855408,0.0052205719985067844],"max_abs_dp":0.002468109130859375,"argmax_equal":true}}]},{"id":"semif_843185706adff0c60f8b","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The agreement requires source files and a user guide, but the delivery record gives no information about either item.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the agreed handover is complete.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=f7403e39d6796072491c","provenance":{"family":"evidence_interpretation","variant":"missing","group_id":"f7403e39d6796072491c","label":2,"option_order":["contradicted","supported","insufficient"]},"state_tokens":22,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["contradicted","supported","insufficient"],"gold":"insufficient","row_len":67,"state_len":22,"decide":66,"opts":[45,54,65],"row_ids_sha256":"a2dad40c2f7171c6997b0df7b4ddbe95bc65f9f881082ea0a0b24370db9a11da","oracle":{"probs":[0.7637298703193665,0.016523797065019608,0.21974632143974304],"logits_raw":[1.5758347511291504,-7.647547721862793,-1.4214788675308228],"argmax":0,"top2_margin":0.5439835786819458,"near_tie":false},"graph":{"p":[0.768615186214447,0.016451317816972733,0.21493349969387054],"max_abs_dp":0.004885315895080566,"argmax_equal":true}}]},{"id":"semif_b531dd603bf080dab820","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The contractor supplied the translated manual but explicitly did not supply the glossary.","questions":{"decision":{"type":"choice","instructions":"Approve final handover exactly when both the translated manual and glossary have been supplied. May final handover be approved?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=a060fd4621be415e9f1a","provenance":{"family":"rule_application","variant":"original","group_id":"a060fd4621be415e9f1a","label":0,"option_order":["prohibited","permitted","insufficient"]},"state_tokens":16,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","permitted","insufficient"],"gold":"prohibited","row_len":82,"state_len":16,"decide":81,"opts":[52,63,80],"row_ids_sha256":"addd5dea73e9b1d9f1d46394bec540d0f4eb3fb566c3050728db320e6389dbff","oracle":{"probs":[0.8360788226127625,0.020044254139065742,0.1438768059015274],"logits_raw":[1.251565933227539,-7.724878311157227,-2.9825172424316406],"argmax":0,"top2_margin":0.6922020316123962,"near_tie":false},"graph":{"p":[0.8354669809341431,0.02037554420530796,0.14415745437145233],"max_abs_dp":0.0006118416786193848,"argmax_equal":true}}]},{"id":"semif_be2dd67bc733c1213bef","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The contractor supplied the translated manual but explicitly did not supply the glossary.","questions":{"decision":{"type":"choice","instructions":"Approve the translation milestone when the manual has been supplied; the glossary is not required for this milestone. May this milestone be approved?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=a060fd4621be415e9f1a","provenance":{"family":"rule_application","variant":"criterion_reversal","group_id":"a060fd4621be415e9f1a","label":1,"option_order":["prohibited","permitted","insufficient"]},"state_tokens":16,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","permitted","insufficient"],"gold":"permitted","row_len":85,"state_len":16,"decide":84,"opts":[55,66,83],"row_ids_sha256":"9158dec01c4a73b37a359b753aff0cb29bb7e454ff339306fd9aab78f8c3da64","oracle":{"probs":[0.03954993560910225,0.8046994805335999,0.15575064718723297],"logits_raw":[-5.723263263702393,1.525936484336853,-2.4253089427948],"argmax":1,"top2_margin":0.6489488482475281,"near_tie":false},"graph":{"p":[0.03999071195721626,0.8049589991569519,0.15505023300647736],"max_abs_dp":0.0007004141807556152,"argmax_equal":true}}]},{"id":"semif_b8e2750307df9761e716","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The contractor supplied both the translated manual and the glossary.","questions":{"decision":{"type":"choice","instructions":"Approve final handover exactly when both the translated manual and glossary have been supplied. May final handover be approved?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=a060fd4621be415e9f1a","provenance":{"family":"rule_application","variant":"evidence_change","group_id":"a060fd4621be415e9f1a","label":0,"option_order":["permitted","prohibited","insufficient"]},"state_tokens":13,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["permitted","prohibited","insufficient"],"gold":"permitted","row_len":79,"state_len":13,"decide":78,"opts":[49,60,77],"row_ids_sha256":"13ab4587463e6e3344bb4d9347eabbbc767e22989a9b7cdf9f4bb4e01b1b916c","oracle":{"probs":[0.8804212808609009,0.018181541934609413,0.10139720141887665],"logits_raw":[0.3456186354160309,-8.989839553833008,-4.854709148406982],"argmax":0,"top2_margin":0.779024064540863,"near_tie":false},"graph":{"p":[0.8795632719993591,0.018443984910845757,0.10199267417192459],"max_abs_dp":0.000858008861541748,"argmax_equal":true}}]},{"id":"semif_5d31b4bf2d0fc193f53b","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The contractor reports working on the handover but supplies no delivery record for either item.","questions":{"decision":{"type":"choice","instructions":"Approve final handover exactly when both the translated manual and glossary have been supplied. May final handover be approved?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=a060fd4621be415e9f1a","provenance":{"family":"rule_application","variant":"missing","group_id":"a060fd4621be415e9f1a","label":1,"option_order":["permitted","insufficient","prohibited"]},"state_tokens":18,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["permitted","insufficient","prohibited"],"gold":"insufficient","row_len":84,"state_len":18,"decide":83,"opts":[54,71,82],"row_ids_sha256":"9511d0bf33c1ce54d56bb2aa0e0533d6e9692cf8c2a39803b7810b7d78aeb985","oracle":{"probs":[0.08856970816850662,0.4373311698436737,0.4740990400314331],"logits_raw":[-5.153538227081299,-1.3113149404525757,-1.117085337638855],"argmax":2,"top2_margin":0.0367678701877594,"near_tie":false},"graph":{"p":[0.08921642601490021,0.4372306764125824,0.4735529124736786],"max_abs_dp":0.0006467178463935852,"argmax_equal":true}}]},{"id":"semif_a9230edfc6ec9e524af0","source":"semif_authored144","request":{"model":"kev-0.8b","state":"A complete incident package requires both a timeline and a remedy report. Candidate A: Both documents are attached. Candidate B: Only the timeline is attached; the remedy report is absent.","questions":{"decision":{"type":"choice","instructions":"Select the complete incident package.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=2b0f8eda8903e38a47de","provenance":{"family":"candidate_selection","variant":"original","group_id":"2b0f8eda8903e38a47de","label":2,"option_order":["B","insufficient","A"]},"state_tokens":37,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","insufficient","A"],"gold":"A","row_len":68,"state_len":37,"decide":67,"opts":[49,60,66],"row_ids_sha256":"f98562f94738f448c5a6768298321efd728ae4a0baa32163844de37a00b0550e","oracle":{"probs":[0.003151868237182498,0.005357069429010153,0.9914910197257996],"logits_raw":[-7.426509857177734,-6.150289535522461,6.411199569702148],"argmax":2,"top2_margin":0.9861339330673218,"near_tie":false},"graph":{"p":[0.0031851644162088633,0.005380942951887846,0.99143385887146],"max_abs_dp":5.716085433959961e-05,"argmax_equal":true}}]},{"id":"semif_c100e84585149a04be49","source":"semif_authored144","request":{"model":"kev-0.8b","state":"A complete incident package requires both a timeline and a remedy report. Candidate A: Both documents are attached. Candidate B: Only the timeline is attached; the remedy report is absent.","questions":{"decision":{"type":"choice","instructions":"Select the package containing the timeline but missing the remedy report.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=2b0f8eda8903e38a47de","provenance":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"2b0f8eda8903e38a47de","label":0,"option_order":["B","A","insufficient"]},"state_tokens":37,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","A","insufficient"],"gold":"B","row_len":74,"state_len":37,"decide":73,"opts":[55,61,72],"row_ids_sha256":"9f026054f18c944de4dd8dd39b1347d3784f30affdf036fab668363d30fd14e4","oracle":{"probs":[0.9777088761329651,0.007992812432348728,0.014298345893621445],"logits_raw":[5.091994762420654,-6.473091125488281,-5.07373046875],"argmax":0,"top2_margin":0.9634105563163757,"near_tie":false},"graph":{"p":[0.9776706695556641,0.007985871285200119,0.014343539252877235],"max_abs_dp":4.519335925579071e-05,"argmax_equal":true}}]},{"id":"semif_579beed184ffc1104aa9","source":"semif_authored144","request":{"model":"kev-0.8b","state":"A complete package requires both documents. Candidate A: The timeline is attached but the remedy report is absent. Candidate B: Both the timeline and remedy report are attached.","questions":{"decision":{"type":"choice","instructions":"Select the complete incident package.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=2b0f8eda8903e38a47de","provenance":{"family":"candidate_selection","variant":"evidence_change","group_id":"2b0f8eda8903e38a47de","label":0,"option_order":["B","A","insufficient"]},"state_tokens":34,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","A","insufficient"],"gold":"B","row_len":65,"state_len":34,"decide":64,"opts":[46,52,63],"row_ids_sha256":"31a1b2ac219902b2bd6603a1e1cd6e9b25ed1ae94f1d65da2873d9fde0e6f867","oracle":{"probs":[0.9845778346061707,0.005327304825186729,0.010094886645674706],"logits_raw":[6.350813388824463,-6.207245826721191,-4.669338226318359],"argmax":0,"top2_margin":0.9744829535484314,"near_tie":false},"graph":{"p":[0.9845483303070068,0.005348298233002424,0.010103299282491207],"max_abs_dp":2.950429916381836e-05,"argmax_equal":true}}]},{"id":"semif_c4e9bd463b6fde5be743","source":"semif_authored144","request":{"model":"kev-0.8b","state":"A complete package requires both documents. Candidate A and Candidate B give only package titles and no attachment information.","questions":{"decision":{"type":"choice","instructions":"Select the complete incident package.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=2b0f8eda8903e38a47de","provenance":{"family":"candidate_selection","variant":"missing","group_id":"2b0f8eda8903e38a47de","label":1,"option_order":["B","insufficient","A"]},"state_tokens":22,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","insufficient","A"],"gold":"insufficient","row_len":53,"state_len":22,"decide":52,"opts":[34,45,51],"row_ids_sha256":"7fbde744dd915c77f8180818bb94e0fca263c1d4d20624992493219ecf1fd412","oracle":{"probs":[0.03872605413198471,0.8984104990959167,0.06286338716745377],"logits_raw":[-4.132961273193359,3.4319353103637695,-2.9673476219177246],"argmax":1,"top2_margin":0.8355470895767212,"near_tie":false},"graph":{"p":[0.03889284282922745,0.8983819484710693,0.06272526830434799],"max_abs_dp":0.00016678869724273682,"argmax_equal":true}}]},{"id":"semif_b863998d4be2721a5dfd","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The routing rule assigns repairs to the in-house workshop when it can perform the work; use the outside workshop only when the in-house workshop cannot. Both workshops confirm they can perform this repair.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the routing rule assigns this repair to the outside workshop.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=01020a1eaf46ec2fba5d","provenance":{"family":"evidence_interpretation","variant":"original","group_id":"01020a1eaf46ec2fba5d","label":2,"option_order":["supported","insufficient","contradicted"]},"state_tokens":39,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["supported","insufficient","contradicted"],"gold":"contradicted","row_len":88,"state_len":39,"decide":87,"opts":[64,75,86],"row_ids_sha256":"21b18c5e535ca85148d63ee076a362fc3d0a8cfd05f46c98ef029aa3e9c20343","oracle":{"probs":[0.006588317919522524,0.9400573372840881,0.05335437133908272],"logits_raw":[-10.358909606933594,1.5766448974609375,-5.3262763023376465],"argmax":1,"top2_margin":0.8867029547691345,"near_tie":false},"graph":{"p":[0.006659606471657753,0.9404995441436768,0.05284080281853676],"max_abs_dp":0.0005135685205459595,"argmax_equal":true}}]},{"id":"semif_37cebc3792e98049439f","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The routing rule assigns repairs to the in-house workshop when it can perform the work; use the outside workshop only when the in-house workshop cannot. Both workshops confirm they can perform this repair.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the outside workshop can perform this repair.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=01020a1eaf46ec2fba5d","provenance":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"01020a1eaf46ec2fba5d","label":1,"option_order":["insufficient","supported","contradicted"]},"state_tokens":39,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","supported","contradicted"],"gold":"supported","row_len":85,"state_len":39,"decide":84,"opts":[63,72,83],"row_ids_sha256":"ba6897fc90ab5cf41fb81a60c94e401876f937af78cb07e6820ab67533ad95cd","oracle":{"probs":[0.05080816149711609,0.9332149028778076,0.015977030619978905],"logits_raw":[-3.0753042697906494,3.9276931285858154,-5.858875751495361],"argmax":1,"top2_margin":0.8824067115783691,"near_tie":false},"graph":{"p":[0.05265524238348007,0.93086838722229,0.016476329416036606],"max_abs_dp":0.002346515655517578,"argmax_equal":true}}]},{"id":"semif_64d63f86e328aff1481c","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The routing rule assigns repairs to the in-house workshop when it can perform the work; use the outside workshop only when the in-house workshop cannot. The in-house workshop cannot perform this repair; the outside workshop can.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the routing rule assigns this repair to the outside workshop.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=01020a1eaf46ec2fba5d","provenance":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"01020a1eaf46ec2fba5d","label":2,"option_order":["contradicted","insufficient","supported"]},"state_tokens":44,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["contradicted","insufficient","supported"],"gold":"supported","row_len":93,"state_len":44,"decide":92,"opts":[71,82,91],"row_ids_sha256":"55c58c2d586a6eef7672e5a016dd4519ad270a53474a221027e9ccc89181fb0e","oracle":{"probs":[0.009599825367331505,0.032133471220731735,0.9582666754722595],"logits_raw":[-6.995548248291016,-4.088671684265137,4.080416202545166],"argmax":2,"top2_margin":0.9261332154273987,"near_tie":false},"graph":{"p":[0.009896211326122284,0.03336342051625252,0.9567403793334961],"max_abs_dp":0.0015262961387634277,"argmax_equal":true}}]},{"id":"semif_6d64a8459f64eac7f4cc","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The routing rule assigns repairs to the in-house workshop when it can perform the work; use the outside workshop only when the in-house workshop cannot. Neither workshop's ability to perform this repair is supplied.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the routing rule assigns this repair to the outside workshop.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=01020a1eaf46ec2fba5d","provenance":{"family":"evidence_interpretation","variant":"missing","group_id":"01020a1eaf46ec2fba5d","label":0,"option_order":["insufficient","contradicted","supported"]},"state_tokens":41,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","contradicted","supported"],"gold":"insufficient","row_len":90,"state_len":41,"decide":89,"opts":[68,79,88],"row_ids_sha256":"280d9ead876b8b4769f9bfdd3020f843c32d5910e63682521808e15a5d3c038f","oracle":{"probs":[0.980688214302063,0.012366223149001598,0.006945554167032242],"logits_raw":[1.8035119771957397,-8.718832015991211,-10.106803894042969],"argmax":0,"top2_margin":0.9683219790458679,"near_tie":false},"graph":{"p":[0.9802917242050171,0.012652859091758728,0.007055384106934071],"max_abs_dp":0.00039649009704589844,"argmax_equal":true}}]},{"id":"semif_2708193212d8a4a523c7","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The town hall and school both confirm they are available and suitable for the meeting.","questions":{"decision":{"type":"choice","instructions":"Book the school only as a fallback when the town hall is unavailable or unsuitable; otherwise book the town hall. May the school be booked under this policy?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=6a06a2f278301c1412ee","provenance":{"family":"rule_application","variant":"original","group_id":"6a06a2f278301c1412ee","label":0,"option_order":["prohibited","permitted","insufficient"]},"state_tokens":17,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","permitted","insufficient"],"gold":"prohibited","row_len":90,"state_len":17,"decide":89,"opts":[60,71,88],"row_ids_sha256":"da6e99c923b48bcc95e6f524a6c925d2d811ba5536bcd265824c139eb03ff273","oracle":{"probs":[0.2302512675523758,0.3750152289867401,0.3947334587574005],"logits_raw":[-4.4337263107299805,-3.260066032409668,-3.136770009994507],"argmax":2,"top2_margin":0.0197182297706604,"near_tie":true},"graph":{"p":[0.23067454993724823,0.37633171677589417,0.3929937481880188],"max_abs_dp":0.0017397105693817139,"argmax_equal":true}}]},{"id":"semif_5a37d1dfcad20810a76a","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The town hall and school both confirm they are available and suitable for the meeting.","questions":{"decision":{"type":"choice","instructions":"Under the alternative policy, either suitable available venue may be booked without priority. May the school be booked?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=6a06a2f278301c1412ee","provenance":{"family":"rule_application","variant":"criterion_reversal","group_id":"6a06a2f278301c1412ee","label":1,"option_order":["prohibited","permitted","insufficient"]},"state_tokens":17,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["prohibited","permitted","insufficient"],"gold":"permitted","row_len":79,"state_len":17,"decide":78,"opts":[49,60,77],"row_ids_sha256":"1c136cd93d3bb9ac540a248d5d9cd09897ac1d9fa5f9affacddf654ce9b86cd4","oracle":{"probs":[0.01263310294598341,0.9337266087532043,0.053640272468328476],"logits_raw":[-7.252532005310059,3.1003711223602295,-3.7734336853027344],"argmax":1,"top2_margin":0.880086362361908,"near_tie":false},"graph":{"p":[0.012668835930526257,0.9334210157394409,0.05391015484929085],"max_abs_dp":0.00030559301376342773,"argmax_equal":true}}]},{"id":"semif_acacd8b46acd25cb954b","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The town hall is confirmed unavailable, while the school is available and suitable.","questions":{"decision":{"type":"choice","instructions":"Book the school only as a fallback when the town hall is unavailable or unsuitable; otherwise book the town hall. May the school be booked under this policy?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=6a06a2f278301c1412ee","provenance":{"family":"rule_application","variant":"evidence_change","group_id":"6a06a2f278301c1412ee","label":1,"option_order":["insufficient","permitted","prohibited"]},"state_tokens":16,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","permitted","prohibited"],"gold":"permitted","row_len":89,"state_len":16,"decide":88,"opts":[65,76,87],"row_ids_sha256":"fd9b3203c3270d108f8547a230dae397d71610ef8856b4afc30438f6763b74a1","oracle":{"probs":[0.10890720039606094,0.8434890508651733,0.047603789716959],"logits_raw":[-4.144232749938965,0.781073272228241,-6.135441303253174],"argmax":1,"top2_margin":0.7345818281173706,"near_tie":false},"graph":{"p":[0.10746435075998306,0.845593273639679,0.04694243520498276],"max_abs_dp":0.0021042227745056152,"argmax_equal":true}}]},{"id":"semif_4089c199b8b796ec3ed8","source":"semif_authored144","request":{"model":"kev-0.8b","state":"The two venues are named, but neither availability nor suitability is provided.","questions":{"decision":{"type":"choice","instructions":"Book the school only as a fallback when the town hall is unavailable or unsuitable; otherwise book the town hall. May the school be booked under this policy?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=6a06a2f278301c1412ee","provenance":{"family":"rule_application","variant":"missing","group_id":"6a06a2f278301c1412ee","label":0,"option_order":["insufficient","prohibited","permitted"]},"state_tokens":15,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","prohibited","permitted"],"gold":"insufficient","row_len":88,"state_len":15,"decide":87,"opts":[64,75,86],"row_ids_sha256":"7226816bb9b9a270c6b1af208d45f22cfbd382bcf98b1dfc470cc56316b918d1","oracle":{"probs":[0.43762126564979553,0.1252461075782776,0.43713265657424927],"logits_raw":[-1.546178936958313,-4.556323528289795,-1.5488667488098145],"argmax":0,"top2_margin":0.0004886090755462646,"near_tie":true},"graph":{"p":[0.4364941418170929,0.1258741170167923,0.4376317262649536],"max_abs_dp":0.0011271238327026367,"argmax_equal":false}}]},{"id":"semif_02f2b95a5c5a4777a45c","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A is the preferred translator and is confirmed able to handle this language. Candidate B is the fallback translator and is also able to handle it.","questions":{"decision":{"type":"choice","instructions":"Choose A whenever A is able; choose B only when A is unable and B is able.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=2797f8c23d05c0ee06b3","provenance":{"family":"candidate_selection","variant":"original","group_id":"2797f8c23d05c0ee06b3","label":1,"option_order":["B","A","insufficient"]},"state_tokens":30,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["B","A","insufficient"],"gold":"A","row_len":74,"state_len":30,"decide":73,"opts":[55,61,72],"row_ids_sha256":"0d2062fc58d5921ab1f5849c1cf2f7debeb4b4caed2133d6bea1a0035c9eff48","oracle":{"probs":[0.006407043896615505,0.951797366142273,0.04179557412862778],"logits_raw":[-7.672119140625,4.360426425933838,-3.1598312854766846],"argmax":1,"top2_margin":0.910001814365387,"near_tie":false},"graph":{"p":[0.0064708092249929905,0.9516192078590393,0.04190998524427414],"max_abs_dp":0.00017815828323364258,"argmax_equal":true}}]},{"id":"semif_ebea4681e3dcc7d92ca2","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A is the preferred translator and is confirmed able to handle this language. Candidate B is the fallback translator and is also able to handle it.","questions":{"decision":{"type":"choice","instructions":"Under the alternative priority, choose B whenever B is able; choose A only when B is unable and A is able.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=2797f8c23d05c0ee06b3","provenance":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"2797f8c23d05c0ee06b3","label":1,"option_order":["insufficient","B","A"]},"state_tokens":30,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","B","A"],"gold":"B","row_len":79,"state_len":30,"decide":78,"opts":[65,71,77],"row_ids_sha256":"b8c376c2166b146132ed824ec4fce8e512337bcd658aa6472e57aa682f3a5cce","oracle":{"probs":[0.5002351999282837,0.18285870552062988,0.3169060945510864],"logits_raw":[-2.2899727821350098,-4.711336612701416,-3.3882691860198975],"argmax":0,"top2_margin":0.18332910537719727,"near_tie":false},"graph":{"p":[0.4998880922794342,0.181038498878479,0.3190733790397644],"max_abs_dp":0.0021672844886779785,"argmax_equal":true}}]},{"id":"semif_b8c5b9b285cbdddc9f8d","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A is the preferred translator but is confirmed unable to handle this language. Candidate B is the fallback and is confirmed able.","questions":{"decision":{"type":"choice","instructions":"Choose A whenever A is able; choose B only when A is unable and B is able.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=2797f8c23d05c0ee06b3","provenance":{"family":"candidate_selection","variant":"evidence_change","group_id":"2797f8c23d05c0ee06b3","label":2,"option_order":["A","insufficient","B"]},"state_tokens":26,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["A","insufficient","B"],"gold":"B","row_len":70,"state_len":26,"decide":69,"opts":[51,62,68],"row_ids_sha256":"076a43f86d0a10ab21f241762001d9500c5795fdf8f5be4d272666bd9fb7f9df","oracle":{"probs":[0.011849828064441681,0.05671675503253937,0.9314334392547607],"logits_raw":[-6.872406959533691,-3.105118751525879,3.628584623336792],"argmax":2,"top2_margin":0.8747166991233826,"near_tie":false},"graph":{"p":[0.011816387996077538,0.056630536913871765,0.9315531253814697],"max_abs_dp":0.00011968612670898438,"argmax_equal":true}}]},{"id":"semif_7ed75d854d99b3a81916","source":"semif_authored144","request":{"model":"kev-0.8b","state":"Candidate A is preferred and Candidate B is the fallback, but neither candidate's ability to handle the language is supplied.","questions":{"decision":{"type":"choice","instructions":"Choose A whenever A is able; choose B only when A is unable and B is able.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=2797f8c23d05c0ee06b3","provenance":{"family":"candidate_selection","variant":"missing","group_id":"2797f8c23d05c0ee06b3","label":0,"option_order":["insufficient","B","A"]},"state_tokens":24,"questions":[{"q":0,"qid":"decision","type":"choice","keys":["insufficient","B","A"],"gold":"insufficient","row_len":68,"state_len":24,"decide":67,"opts":[54,60,66],"row_ids_sha256":"cc8a89bed2c7eee3e38e319a82ef40cf549e11a9699bd182ab33f5e858dd9d73","oracle":{"probs":[0.35997670888900757,0.018347473815083504,0.6216757893562317],"logits_raw":[-1.089073657989502,-8.250795364379883,0.22554248571395874],"argmax":2,"top2_margin":0.2616990804672241,"near_tie":false},"graph":{"p":[0.3556850552558899,0.018363969400525093,0.6259509921073914],"max_abs_dp":0.004291653633117676,"argmax_equal":true}}]},{"id":"own_t01","source":"own_text","request":{"model":"kev-0.8b","state":"Ticket #4471 - opened 2026-03-14 09:12\nFrom: Dana (customer since 2023)\nSubject: Charged twice for March\n\nHi, I'm on the Pennimere Meal Kits weekly plan. My card statement shows two charges of 64.90 on March 1, but I only received one box. I already contacted my bank and they told me to ask you first. Please refund the duplicate charge. I'd also like to pause deliveries for two weeks while I'm travelling, starting March 20. Thanks, Dana","questions":{"team":{"type":"choice","instructions":"Which team should handle this ticket first?","criteria":{"billing":"Charges, invoices and refunds","deliveries":"Boxes, couriers and delivery dates","account":"Sign-in, plan changes and pauses","technical":"App or website faults","other":null}},"several_actions":{"type":"noul","instructions":"Does the ticket ask for more than one action?"},"upset":{"type":"score","instructions":"How upset is the customer?","criteria":["calm","mildly annoyed","frustrated","angry"]},"refund":{"type":"noul","instructions":"Refund request?","criteria":{"true":"The customer explicitly asks for money back","false":"No refund is requested"}}}},"gold":{"team":"billing","several_actions":"true","upset":"1","refund":"true"},"note":"support ticket","state_tokens":131,"questions":[{"q":0,"qid":"team","type":"choice","keys":["billing","deliveries","account","technical","other"],"gold":"billing","row_len":184,"state_len":131,"decide":183,"opts":[148,160,171,179,182],"row_ids_sha256":"83b620952b41d34c8ad6bff611d2764deef4aa3e6c74a1ac4dc2734a37893aa4","oracle":{"probs":[0.5710851550102234,0.0908634215593338,0.2018778771162033,0.026216482743620872,0.10995703935623169],"logits_raw":[3.334623336791992,-1.0881314277648926,0.8326312303543091,-4.078777313232422,-0.6292206048965454],"argmax":0,"top2_margin":0.3692072629928589,"near_tie":false},"graph":{"p":[0.57033771276474,0.09170949459075928,0.20198439061641693,0.026061683893203735,0.10990674793720245],"max_abs_dp":0.0008460730314254761,"argmax_equal":true}},{"q":1,"qid":"several_actions","type":"noul","keys":["false","true"],"gold":"true","row_len":149,"state_len":131,"decide":148,"opts":[144,147],"row_ids_sha256":"1bf3a37090c5826419fc4f776c23c71999c929f54e5cb0f0f823f295e4eac15f","oracle":{"probs":[0.24468646943569183,0.755313515663147],"logits_raw":[-1.440720796585083,1.2712711095809937],"argmax":1,"top2_margin":0.510627031326294,"near_tie":false},"graph":{"p":[0.24505110085010529,0.7549489140510559],"max_abs_dp":0.00036463141441345215,"argmax_equal":true}},{"q":2,"qid":"upset","type":"score","keys":["0","1","2","3"],"gold":"1","row_len":158,"state_len":131,"decide":157,"opts":[141,147,152,156],"row_ids_sha256":"f0e4d0f86185f26b7fba810e3866a53caca0da4f3d6b3e1b641f081b1b569261","oracle":{"probs":[0.3663862645626068,0.36807045340538025,0.1958460807800293,0.06969724595546722],"logits_raw":[0.07100933790206909,0.08204400539398193,-1.4360418319702148,-3.921896457672119],"argmax":1,"top2_margin":0.0016841888427734375,"near_tie":true},"graph":{"p":[0.36541780829429626,0.3677539825439453,0.1967996209859848,0.07002867013216019],"max_abs_dp":0.0009684562683105469,"argmax_equal":true}},{"q":3,"qid":"refund","type":"noul","keys":["false","true"],"gold":"true","row_len":156,"state_len":131,"decide":155,"opts":[143,154],"row_ids_sha256":"e222b432a32e9ea481b64966a4ccf24551ac8f3fef59c62df11ee80c7531f508","oracle":{"probs":[0.05692391097545624,0.9430760741233826],"logits_raw":[-2.8353071212768555,3.91951322555542],"argmax":1,"top2_margin":0.8861521482467651,"near_tie":false},"graph":{"p":[0.057627130299806595,0.9423729181289673],"max_abs_dp":0.0007032193243503571,"argmax_equal":true}}]},{"id":"own_t02","source":"own_text","text":"withheld (name screen)","request_sha256":"618344a156a1da5a7697a30fb0c159d753890ccfd0b3abc4d36da10050792a67","state_tokens":247,"questions":[{"q":0,"type":"score","n_options":4,"row_len":270,"state_len":247,"decide":269,"opts":[258,261,265,268],"row_ids_sha256":"40d88e8df0f859d17f4fc4cf804953d7e7f0aee88b62cb1eae690bfd35e74228","oracle":{"probs":[0.1914207637310028,0.3702922463417053,0.3727027177810669,0.0655842050909996],"logits_raw":[-1.7981369495391846,-0.21058037877082825,-0.194968581199646,-4.3753509521484375],"argmax":2,"top2_margin":0.0024104714393615723,"near_tie":true},"graph":{"p":[0.19304212927818298,0.37176093459129333,0.36983340978622437,0.06536351144313812],"max_abs_dp":0.0028693079948425293,"argmax_equal":false}},{"q":1,"type":"choice","n_options":5,"row_len":303,"state_len":247,"decide":302,"opts":[265,279,288,298,301],"row_ids_sha256":"ca0548bc4cd60939636a8c7a12770aa94bfb86c14d1f316a31e31437f91953b7","oracle":{"probs":[0.003948438912630081,0.9330828785896301,0.008283678442239761,0.008900881744921207,0.04578408971428871],"logits_raw":[-7.336916923522949,5.812563896179199,-5.554113864898682,-5.381207466125488,-1.4406118392944336],"argmax":1,"top2_margin":0.8872987627983093,"near_tie":false},"graph":{"p":[0.0039802840910851955,0.9328378438949585,0.008319015614688396,0.008998509496450424,0.04586433246731758],"max_abs_dp":0.00024503469467163086,"argmax_equal":true}},{"q":2,"type":"noul","n_options":2,"row_len":262,"state_len":247,"decide":261,"opts":[257,260],"row_ids_sha256":"f685d22a03316c7049e454dccb6ac9363e2c833dc3c59951a23d152339f9d667","oracle":{"probs":[0.9926832318305969,0.0073168231174349785],"logits_raw":[4.8031463623046875,-7.011125564575195],"argmax":0,"top2_margin":0.9853664040565491,"near_tie":false},"graph":{"p":[0.9926565289497375,0.0073434943333268166],"max_abs_dp":2.6702880859375e-05,"argmax_equal":true}}]},{"id":"own_t03","source":"own_text","text":"withheld (name screen)","request_sha256":"611f9ed0157a71806a58173aac94d173efb312d772a5e8972148a469d96a58bf","state_tokens":192,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":232,"state_len":192,"decide":231,"opts":[203,207,211,215,219,230],"row_ids_sha256":"5205ad7d6ddda90f69d58059f71706b8306cb0135da3484676ab81d1de5cc633","oracle":{"probs":[0.0020695424173027277,0.9451109170913696,0.010300913825631142,0.009771846234798431,0.0036126382183283567,0.029134098440408707],"logits_raw":[-10.684650421142578,4.049939155578613,-6.823168754577637,-6.950032711029053,-9.34421443939209,-4.321654796600342],"argmax":1,"top2_margin":0.9159768223762512,"near_tie":false},"graph":{"p":[0.0020746549125760794,0.9447587132453918,0.010351435281336308,0.009831058792769909,0.0036323717795312405,0.029351750388741493],"max_abs_dp":0.0003522038459777832,"argmax_equal":true}},{"q":1,"type":"noul","n_options":2,"row_len":214,"state_len":192,"decide":213,"opts":[201,212],"row_ids_sha256":"2b86aad2a4dc065103885eb595709259e0660d190d0f2e021b2c8e191e504ecc","oracle":{"probs":[0.055014487355947495,0.9449855089187622],"logits_raw":[-3.0350255966186523,3.806753158569336],"argmax":1,"top2_margin":0.8899710178375244,"near_tie":false},"graph":{"p":[0.05563857778906822,0.9443614482879639],"max_abs_dp":0.0006240904331207275,"argmax_equal":true}},{"q":2,"type":"choice","n_options":4,"row_len":245,"state_len":192,"decide":244,"opts":[213,225,234,243],"row_ids_sha256":"7c5d36278672c7a4a55cd82f7729f0fb325108ce3b1edf105f0fb43a5d68d22d","oracle":{"probs":[0.8466532230377197,0.07334506511688232,0.045452676713466644,0.03454908728599548],"logits_raw":[2.119715690612793,-3.765761375427246,-4.917064666748047,-5.577021598815918],"argmax":0,"top2_margin":0.7733081579208374,"near_tie":false},"graph":{"p":[0.8460888266563416,0.07343357056379318,0.04575064405798912,0.03472694382071495],"max_abs_dp":0.0005643963813781738,"argmax_equal":true}}]},{"id":"own_t04","source":"own_text","request":{"model":"kev-0.8b","state":"Quennick Labs - mobile team stand-up, Wednesday\nPresent: Aiko, Ben, Chloé, Femi\nAiko: finished the offline sync screen; waiting on design review before merging.\nBen: the crash on login for Android 12 users is still open; he can reproduce it only on one test phone. Blocked until the device lab returns the second phone on Friday.\nChloé: release notes for 4.2 drafted; needs numbers from Femi.\nFemi: analytics export fixed; will send Chloé the weekly active user numbers today.\nDecisions: release 4.2 moves from Thursday to Monday because of the login crash. Ben owns the crash; Aiko will pair with him tomorrow.","questions":{"crash_owner":{"type":"choice","instructions":"Who owns the login crash?","criteria":{"aiko":"Aiko","ben":"Ben","chloe":"Chloé","femi":"Femi"}},"blocked":{"type":"noul","instructions":"Is any team member blocked?"},"release_day":{"type":"choice","instructions":"When is release 4.2 now planned?","criteria":{"thursday":"Thursday","friday":"Friday","monday":"Monday","unknown":"Not decided"}},"blockers":{"type":"score","instructions":"How many people report a blocker?","criteria":["none","one","two or more"]}}},"gold":{"crash_owner":"ben","blocked":"true","release_day":"monday","blockers":"1"},"note":"meeting notes","state_tokens":155,"questions":[{"q":0,"qid":"crash_owner","type":"choice","keys":["aiko","ben","chloe","femi"],"gold":"ben","row_len":190,"state_len":155,"decide":189,"opts":[168,173,181,188],"row_ids_sha256":"c836e824b242d82efc9bb874dcc94fc634b6e668f5245c509f330c1daeb210d7","oracle":{"probs":[0.00562314735725522,0.9882436990737915,0.0030788234435021877,0.0030543364118784666],"logits_raw":[-6.87905740737915,5.557906150817871,-8.328327178955078,-8.347539901733398],"argmax":1,"top2_margin":0.9826205372810364,"near_tie":false},"graph":{"p":[0.0056945267133414745,0.9881189465522766,0.0031004406046122313,0.0030861124396324158],"max_abs_dp":0.00012475252151489258,"argmax_equal":true}},{"q":1,"qid":"blocked","type":"noul","keys":["false","true"],"gold":"true","row_len":169,"state_len":155,"decide":168,"opts":[164,167],"row_ids_sha256":"c42166f9d6c411776304b4cc7028ad340aa2a740694ebaf11a4446caf5f0a0ce","oracle":{"probs":[0.23977839946746826,0.7602216005325317],"logits_raw":[-1.651745319366455,1.1245832443237305],"argmax":1,"top2_margin":0.5204432010650635,"near_tie":false},"graph":{"p":[0.24016612768173218,0.759833812713623],"max_abs_dp":0.0003877878189086914,"argmax_equal":true}},{"q":2,"qid":"release_day","type":"choice","keys":["thursday","friday","monday","unknown"],"gold":"monday","row_len":191,"state_len":155,"decide":190,"opts":[171,177,183,189],"row_ids_sha256":"71c30ae67039b35000c37ce534a15965ec5c4395c7da8e2d99c3abcdd4e2b66a","oracle":{"probs":[0.007304235827177763,0.0132474759593606,0.9195667505264282,0.0598815456032753],"logits_raw":[-6.999885559082031,-5.567437171936035,4.634444713592529,-1.9377630949020386],"argmax":2,"top2_margin":0.8596851825714111,"near_tie":false},"graph":{"p":[0.0071001905016601086,0.012997070327401161,0.9210289120674133,0.05887376517057419],"max_abs_dp":0.0014621615409851074,"argmax_equal":true}},{"q":3,"qid":"blockers","type":"score","keys":["0","1","2"],"gold":"1","row_len":175,"state_len":155,"decide":174,"opts":[165,168,173],"row_ids_sha256":"905f802ad561e6041e8025e5f1bf46ae4a2364918a79f3bbc3f68db7c2e30961","oracle":{"probs":[0.08063077181577682,0.8613259792327881,0.058043234050273895],"logits_raw":[-2.969484329223633,2.72946834564209,-3.760334014892578],"argmax":1,"top2_margin":0.7806951999664307,"near_tie":false},"graph":{"p":[0.08202023059129715,0.8593794107437134,0.05860035866498947],"max_abs_dp":0.001946568489074707,"argmax_equal":true}}]},{"id":"own_t05","source":"own_text","text":"withheld (name screen)","request_sha256":"6b142cbabc69002becc9d793b7cab1ef1ec5d25426571e93a0acb16e39aed3f2","state_tokens":102,"questions":[{"q":0,"type":"score","n_options":5,"row_len":132,"state_len":102,"decide":131,"opts":[114,118,122,126,130],"row_ids_sha256":"f63dcba0291d3bdf63551857c41078d7c0e0676f652a3b3bd522f7d60fc560e6","oracle":{"probs":[0.05646618455648422,0.8141573071479797,0.12698572874069214,0.0021074225660413504,0.0002832839672919363],"logits_raw":[-2.792074680328369,3.628497838973999,-0.8421327471733093,-10.703591346740723,-15.531982421875],"argmax":1,"top2_margin":0.6871715784072876,"near_tie":false},"graph":{"p":[0.05719476565718651,0.8122361302375793,0.12815095484256744,0.0021282939705997705,0.0002898216771427542],"max_abs_dp":0.0019211769104003906,"argmax_equal":true}},{"q":1,"type":"choice","n_options":6,"row_len":158,"state_len":102,"decide":157,"opts":[117,126,132,138,147,156],"row_ids_sha256":"63b39fbac5a70d0c7298a5714b57f6e4ea7fcb6ef95fb8be4920b41f92b485c4","oracle":{"probs":[0.004670754540711641,0.8968867063522339,0.032181646674871445,0.04655159264802933,0.0015213091392070055,0.018188145011663437],"logits_raw":[-7.326563835144043,5.323505401611328,-2.682705879211426,-1.7944765090942383,-10.025548934936523,-4.055661201477051],"argmax":1,"top2_margin":0.8503351211547852,"near_tie":false},"graph":{"p":[0.004693141672760248,0.8964901566505432,0.03221879154443741,0.04691139981150627,0.0015386538580060005,0.01814776472747326],"max_abs_dp":0.00039654970169067383,"argmax_equal":true}},{"q":2,"type":"noul","n_options":2,"row_len":117,"state_len":102,"decide":116,"opts":[112,115],"row_ids_sha256":"99711b488da8291395ef03d4c768c36861d4e305d5ade9115c99679d11b7cba1","oracle":{"probs":[0.9959360361099243,0.004063933622092009],"logits_raw":[5.4003400802612305,-7.836619853973389],"argmax":0,"top2_margin":0.9918721318244934,"near_tie":false},"graph":{"p":[0.9958563446998596,0.004143686965107918],"max_abs_dp":7.97533430159092e-05,"argmax_equal":true}},{"q":3,"type":"noul","n_options":2,"row_len":125,"state_len":102,"decide":124,"opts":[113,123],"row_ids_sha256":"499a60c75e1a70407df503feba0ab821da5a68108e7c249d502313ebad29d8f8","oracle":{"probs":[0.0051238215528428555,0.9948762059211731],"logits_raw":[-6.761048316955566,5.915749549865723],"argmax":1,"top2_margin":0.9897524118423462,"near_tie":false},"graph":{"p":[0.005191130563616753,0.9948087930679321],"max_abs_dp":6.74128532409668e-05,"argmax_equal":true}}]},{"id":"own_t06","source":"own_text","request":{"model":"kev-0.8b","state":"[19:02] Guest (room 314): hi, the heating in my room isn't working and it's freezing\n[19:03] Front desk (Wendlecote Hotel): I'm sorry to hear that! I'll send someone from maintenance right away.\n[19:04] Guest: thanks. also can I keep the room until 2pm tomorrow? my train is at 3\n[19:06] Front desk: Late checkout until 14:00 is possible for 20 €, shall I add it?\n[19:06] Guest: yes please\n[19:25] Guest: still nobody came and it's getting colder\n[19:26] Front desk: Apologies, our technician is on another call. He will be with you in 10 minutes, and we can move you to room 402 if you prefer.","questions":{"first_request":{"type":"choice","instructions":"What did the guest ask for first?","criteria":{"room_service":"Food or drinks to the room","late_checkout":"Leave the room later than usual","towels":"Extra towels or linen","maintenance":"Something in the room is broken","billing":"A question about the bill","booking_change":"Change the dates or room type","lost_item":"Something was lost","noise":"A noise complaint","transport":"Help with a taxi or train","other":null}},"fixed":{"type":"noul","instructions":"Has the heating problem been fixed by the end of the chat?"},"patience":{"type":"score","instructions":"How patient is the guest by the end?","criteria":["patient","impatient","angry"]}}},"gold":{"first_request":"maintenance","fixed":"false","patience":"1"},"note":"chat log, 10-option choice","state_tokens":202,"questions":[{"q":0,"qid":"first_request","type":"choice","keys":["room_service","late_checkout","towels","maintenance","billing","booking_change","lost_item","noise","transport","other"],"gold":"maintenance","row_len":301,"state_len":202,"decide":300,"opts":[221,232,241,251,260,271,279,286,296,299],"row_ids_sha256":"29e30b4b4114a003a4dabfdbb2cb2bf207297dec7de8bd9d2102471b7b981e3e","oracle":{"probs":[0.003737571183592081,0.04860172048211098,0.004495671484619379,0.7091311812400818,0.004683794919401407,0.027817819267511368,0.002698435913771391,0.006418914068490267,0.014914719387888908,0.17750012874603271],"logits_raw":[-9.635528564453125,-3.4634742736816406,-9.191181182861328,2.9856576919555664,-9.092548370361328,-4.806006908416748,-10.419332504272461,-8.334305763244629,-6.3057661056518555,-0.3468876779079437],"argmax":3,"top2_margin":0.5316310524940491,"near_tie":false},"graph":{"p":[0.0037599981296807528,0.048349376767873764,0.004497551824897528,0.7088735699653625,0.004692920949310064,0.02785314992070198,0.0027120320592075586,0.0064357612282037735,0.014978147111833096,0.17784744501113892],"max_abs_dp":0.00034731626510620117,"argmax_equal":true}},{"q":1,"qid":"fixed","type":"noul","keys":["false","true"],"gold":"false","row_len":223,"state_len":202,"decide":222,"opts":[218,221],"row_ids_sha256":"14e6abc3c4cd132f7f2f8d8a5d5cc4edf569788a0e9ac610af8a6f2caf216e06","oracle":{"probs":[0.9824784994125366,0.017521429806947708],"logits_raw":[1.9351263046264648,-7.753203392028809],"argmax":0,"top2_margin":0.964957058429718,"near_tie":false},"graph":{"p":[0.9826158285140991,0.017384182661771774],"max_abs_dp":0.0001373291015625,"argmax_equal":true}},{"q":2,"qid":"patience","type":"score","keys":["0","1","2"],"gold":"1","row_len":224,"state_len":202,"decide":223,"opts":[214,218,222],"row_ids_sha256":"186f111eefc4f19a1312fafca0e573857ecda997592c8e680f03bee7af597223","oracle":{"probs":[0.08255375921726227,0.7847334742546082,0.13271275162696838],"logits_raw":[-2.01625394821167,3.401916742324829,-0.8740119338035583],"argmax":1,"top2_margin":0.6520206928253174,"near_tie":false},"graph":{"p":[0.0832199975848198,0.7840104699134827,0.13276946544647217],"max_abs_dp":0.0007230043411254883,"argmax_equal":true}}]},{"id":"own_t07","source":"own_text","request":{"model":"kev-0.8b","state":"Ostlebury Dental - voicemail transcript, Friday 16:48\n'Hello, this is Jonas. I have a cleaning appointment on Tuesday at 9:30 with Dr Amal. Something came up at work, so I need to move it. I can do Wednesday after 15:00 or any time on Thursday morning. Please call me back to confirm. I can't check messages during the day, so after 18:00 is best. Thank you.'\nFree slots next week: Tue 11:00, Wed 15:30, Wed 17:00, Thu 08:30, Thu 12:30.","questions":{"slot":{"type":"choice","instructions":"Which of these free slots fits the caller?","criteria":{"tue_1100":"Tuesday 11:00","wed_1530":"Wednesday 15:30","thu_1230":"Thursday 12:30","none":"None of them fits"}},"callback":{"type":"noul","criteria":{"true":"The caller wants to be phoned back","false":"No call back is needed"}},"callback_time":{"type":"choice","criteria":{"morning":"Call back in the morning","afternoon":"Call back in the afternoon","evening":"Call back after 18:00"}}}},"gold":{"slot":"wed_1530","callback":"true","callback_time":"evening"},"note":"voicemail + scheduling","state_tokens":151,"questions":[{"q":0,"qid":"slot","type":"choice","keys":["tue_1100","wed_1530","thu_1230","none"],"gold":"wed_1530","row_len":219,"state_len":151,"decide":218,"opts":[177,193,209,217],"row_ids_sha256":"f74cd37346faa8848e611324a08d39518de9f69639cce75caa4ee65b2d6e5b24","oracle":{"probs":[0.054513849318027496,0.39418861269950867,0.2716035842895508,0.27969399094581604],"logits_raw":[-4.342125415802002,0.4179430603981018,-0.47827690839767456,-0.4076530635356903],"argmax":1,"top2_margin":0.11449462175369263,"near_tie":false},"graph":{"p":[0.05542980507016182,0.39896801114082336,0.26678210496902466,0.27882006764411926],"max_abs_dp":0.004821479320526123,"argmax_equal":true}},{"q":1,"qid":"callback","type":"noul","keys":["false","true"],"gold":"true","row_len":174,"state_len":151,"decide":173,"opts":[160,172],"row_ids_sha256":"901662360835425dca65b2a282ec58ed82f1f667ce197c307f6c871fe06a804a","oracle":{"probs":[0.05446076765656471,0.9455391764640808],"logits_raw":[-2.163533926010132,4.703993797302246],"argmax":1,"top2_margin":0.8910784125328064,"near_tie":false},"graph":{"p":[0.05473453551530838,0.9452654719352722],"max_abs_dp":0.0002737678587436676,"argmax_equal":true}},{"q":2,"qid":"callback_time","type":"choice","keys":["morning","afternoon","evening"],"gold":"evening","row_len":187,"state_len":151,"decide":186,"opts":[161,171,185],"row_ids_sha256":"467b7f5e1e08ba3a99ebcfddeeda815ce641f64e01aa23c62ddf265afe0d1bef","oracle":{"probs":[0.07689440250396729,0.07516466081142426,0.847940981388092],"logits_raw":[-3.503873825073242,-3.5586156845092773,2.2715554237365723],"argmax":2,"top2_margin":0.7710465788841248,"near_tie":false},"graph":{"p":[0.07805769890546799,0.0760919526219368,0.84585040807724],"max_abs_dp":0.0020905733108520508,"argmax_equal":true}}]},{"id":"own_t08","source":"own_text","text":"withheld (name screen)","request_sha256":"fa7821c465f476d33b05cd4ea2cf428445e8d60d8424184abbcd953b3b3d3259","state_tokens":128,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":155,"state_len":128,"decide":154,"opts":[143,153],"row_ids_sha256":"6b7a6d9b2b486633968818bc9314a41189fb0e1d0851439df4e6e5a82fd37e0a","oracle":{"probs":[0.05369734764099121,0.9463026523590088],"logits_raw":[-4.65091609954834,2.2525196075439453],"argmax":1,"top2_margin":0.8926053047180176,"near_tie":false},"graph":{"p":[0.05451211333274841,0.945487916469574],"max_abs_dp":0.0008147656917572021,"argmax_equal":true}},{"q":1,"type":"choice","n_options":4,"row_len":155,"state_len":128,"decide":154,"opts":[141,145,149,153],"row_ids_sha256":"d2f30fc3c75a07fe50ef7cd6c92013f0b5790c78b9da0a9f895f77a3060d7302","oracle":{"probs":[0.017629867419600487,0.9719675779342651,0.005944828037172556,0.004457809031009674],"logits_raw":[-7.118899345397949,2.5287063121795654,-9.734451293945312,-10.427066802978516],"argmax":1,"top2_margin":0.9543377161026001,"near_tie":false},"graph":{"p":[0.01789722591638565,0.9716569781303406,0.0059607429429888725,0.004485053010284901],"max_abs_dp":0.00031059980392456055,"argmax_equal":true}},{"q":2,"type":"score","n_options":4,"row_len":170,"state_len":128,"decide":169,"opts":[140,148,160,168],"row_ids_sha256":"c1551bfc5200e4689e7a7a9da5b7aed41ac57cbf3478984ae368955c0b75976c","oracle":{"probs":[0.32353493571281433,0.1330980509519577,0.40054214000701904,0.14282479882240295],"logits_raw":[-3.6947197914123535,-5.83182430267334,-3.180999755859375,-5.662119388580322],"argmax":2,"top2_margin":0.07700720429420471,"near_tie":false},"graph":{"p":[0.3242354393005371,0.1339685469865799,0.4002193212509155,0.14157669246196747],"max_abs_dp":0.0012481063604354858,"argmax_equal":true}}]},{"id":"own_t09","source":"own_text","text":"withheld (name screen)","request_sha256":"15472ea434bd49bc396e88dab5249fe69bad7cbec0d0b0879c795a32eaf9c508","state_tokens":106,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":122,"state_len":106,"decide":121,"opts":[117,120],"row_ids_sha256":"29bebbf33491958a5e6b9f2f86a430c567601a5a36d2abde2395f894e93ecd86","oracle":{"probs":[0.12070854753255844,0.8792914748191833],"logits_raw":[-1.7208327054977417,3.0569512844085693],"argmax":1,"top2_margin":0.7585829496383667,"near_tie":false},"graph":{"p":[0.12082564830780029,0.8791743516921997],"max_abs_dp":0.00011712312698364258,"argmax_equal":true}},{"q":1,"type":"choice","n_options":4,"row_len":151,"state_len":106,"decide":150,"opts":[123,132,141,149],"row_ids_sha256":"67b90692181aef08d9df4354787e70218579a8bcefb1fe7e6ee258655a00598c","oracle":{"probs":[0.968799352645874,0.0070528313517570496,0.004022529348731041,0.02012527361512184],"logits_raw":[4.552127361297607,-7.291962623596191,-8.643003463745117,-4.769104957580566],"argmax":0,"top2_margin":0.9486740827560425,"near_tie":false},"graph":{"p":[0.9686601161956787,0.007093985099345446,0.004041136708110571,0.020204778760671616],"max_abs_dp":0.0001392364501953125,"argmax_equal":true}}]},{"id":"own_t10","source":"own_text","request":{"model":"kev-0.8b","state":"To the managing agent, Pennimere Court\n\nI live in flat 9 and I am writing, for the third time this month, about the noise from the gym equipment in the basement. The machines are used until after midnight, and the thumping comes up through the floor of my bedroom. I have asked for the opening hours to be limited to 07:00-22:00, as the house rules say. If nothing changes by the end of next week, I will raise it with the residents' committee.\n\nYours, Teodora (flat 9)","questions":{"topic":{"type":"choice","instructions":"What is the letter about?","criteria":{"noise":"Noise","repairs":"Repairs","rent":"Rent or charges","parking":"Parking","pets":"Pets"}},"first_time":{"type":"noul","instructions":"Is this the first time the writer complains?"},"escalation":{"type":"score","instructions":"How far is the writer prepared to take this?","criteria":["no further step","a reminder","the residents' committee","a lawyer or court"]}}},"gold":{"topic":"noise","first_time":"false","escalation":"2"},"note":"complaint letter","state_tokens":123,"questions":[{"q":0,"qid":"topic","type":"choice","keys":["noise","repairs","rent","parking","pets"],"gold":"noise","row_len":161,"state_len":123,"decide":160,"opts":[134,141,148,154,159],"row_ids_sha256":"7fcf9323713af9c4ef2611d7d952aaec63879bf2856e1688a047c971c3d82cbf","oracle":{"probs":[0.9628740549087524,0.015841851010918617,0.01494741439819336,0.003785195527598262,0.002551453188061714],"logits_raw":[3.5208754539489746,-6.361414909362793,-6.501247882843018,-9.805814743041992,-10.754842758178711],"argmax":0,"top2_margin":0.9470322132110596,"near_tie":false},"graph":{"p":[0.9630509614944458,0.01577630825340748,0.01482540275901556,0.003795523662120104,0.0025519123300909996],"max_abs_dp":0.00017690658569335938,"argmax_equal":true}},{"q":1,"qid":"first_time","type":"noul","keys":["false","true"],"gold":"false","row_len":140,"state_len":123,"decide":139,"opts":[135,138],"row_ids_sha256":"ee3121e333da5f04b77b1f8592b0c606df1a601805e3d8fb6544172c002089ba","oracle":{"probs":[0.9098619818687439,0.0901380404829979],"logits_raw":[1.4258140325546265,-4.136854648590088],"argmax":0,"top2_margin":0.8197239637374878,"near_tie":false},"graph":{"p":[0.911821186542511,0.0881788358092308],"max_abs_dp":0.00195920467376709,"argmax_equal":true}},{"q":2,"qid":"escalation","type":"score","keys":["0","1","2","3"],"gold":"2","row_len":156,"state_len":123,"decide":155,"opts":[138,142,148,154],"row_ids_sha256":"9264179d83bc2602350f2618be7a610f834ab923db2161d5dfea31d73240e999","oracle":{"probs":[0.09475820511579514,0.061989884823560715,0.8251891732215881,0.01806284859776497],"logits_raw":[-2.2471165657043457,-3.268141269683838,2.960259199142456,-6.235074996948242],"argmax":2,"top2_margin":0.7304309606552124,"near_tie":false},"graph":{"p":[0.09489418566226959,0.06194428727030754,0.8249611258506775,0.018200475722551346],"max_abs_dp":0.00022804737091064453,"argmax_equal":true}}]},{"id":"own_j01","source":"own_json","request":{"model":"kev-0.8b","state":{"invoice":"FPW-2026-0412","issuer":"Fennistone Print Works","customer":"Ostlebury Community Choir","issued":"2026-04-02","due":"2026-05-02","status":"unpaid","reminders_sent":2,"lines":[{"item":"Concert posters A2","qty":40,"unit_price":1.85},{"item":"Programme booklets","qty":250,"unit_price":0.62},{"item":"Delivery","qty":1,"unit_price":12.0}],"subtotal":241.0,"vat_rate":0.2,"total":289.2,"payment_terms":"bank transfer within 30 days","today":"2026-05-19"},"questions":{"overdue":{"type":"noul","instructions":"Is the invoice overdue?"},"method":{"type":"choice","instructions":"How is the customer expected to pay?","criteria":{"card":null,"bank_transfer":null,"cash":null,"cheque":null}},"size":{"type":"score","instructions":"How large is the invoice total?","criteria":["under 100","100 to 250","250 to 500","over 500"]},"reminded":{"type":"noul","criteria":{"true":"At least one reminder has already been sent","false":"No reminder has been sent yet"}}}},"gold":{"overdue":"true","method":"bank_transfer","size":"2","reminded":"true"},"note":"invoice (object state)","state_tokens":199,"questions":[{"q":0,"qid":"overdue","type":"noul","keys":["false","true"],"gold":"true","row_len":212,"state_len":199,"decide":211,"opts":[207,210],"row_ids_sha256":"88ee02a817b9884fa9884b3bb74469357d2c1503f98b94736eefd467da3a589f","oracle":{"probs":[0.2744416296482086,0.7255584001541138],"logits_raw":[-1.6488921642303467,0.6902769207954407],"argmax":1,"top2_margin":0.45111677050590515,"near_tie":false},"graph":{"p":[0.27856868505477905,0.721431314945221],"max_abs_dp":0.004127085208892822,"argmax_equal":true}},{"q":1,"qid":"method","type":"choice","keys":["card","bank_transfer","cash","cheque"],"gold":"bank_transfer","row_len":223,"state_len":199,"decide":222,"opts":[210,214,217,221],"row_ids_sha256":"2ff072010fbe7a47f0c8bf747effdd55e393ec007c718c681872aa8067a6c625","oracle":{"probs":[0.007005597930401564,0.9563279151916504,0.007846099324524403,0.02882051281630993],"logits_raw":[-9.035520553588867,2.793562412261963,-8.762899398803711,-5.6324567794799805],"argmax":1,"top2_margin":0.9275074005126953,"near_tie":false},"graph":{"p":[0.007046387065201998,0.9563867449760437,0.007804200053215027,0.0287625789642334],"max_abs_dp":5.882978439331055e-05,"argmax_equal":true}},{"q":2,"qid":"size","type":"score","keys":["0","1","2","3"],"gold":"2","row_len":242,"state_len":199,"decide":241,"opts":[213,223,233,240],"row_ids_sha256":"343418b3a145b928b88a6e85a14d151db402077c4a66ba1df4c649b6b32584fa","oracle":{"probs":[0.0037643350660800934,0.01599075086414814,0.9721170663833618,0.008127870038151741],"logits_raw":[-13.686630249023438,-10.206424713134766,-0.32365682721138,-11.834627151489258],"argmax":2,"top2_margin":0.95612633228302,"near_tie":false},"graph":{"p":[0.003796817036345601,0.016234805807471275,0.9718188047409058,0.008149465546011925],"max_abs_dp":0.0002982616424560547,"argmax_equal":true}},{"q":3,"qid":"reminded","type":"noul","keys":["false","true"],"gold":"true","row_len":223,"state_len":199,"decide":222,"opts":[209,221],"row_ids_sha256":"49289b99a83c757988232a1ea13d55d8d8c64b74e19a5bb8c79b96a25c1406d9","oracle":{"probs":[0.00762958824634552,0.9923703670501709],"logits_raw":[-8.951849937438965,2.7609517574310303],"argmax":1,"top2_margin":0.9847407937049866,"near_tie":false},"graph":{"p":[0.007804870139807463,0.9921951293945312],"max_abs_dp":0.00017528189346194267,"argmax_equal":true}}]},{"id":"own_j02","source":"own_json","request":{"model":"kev-0.8b","state":{"order_id":"PF-88213","shop":"Pennimere Foods","placed":"2026-06-03 18:22","items":[{"sku":"OAT-1KG","name":"Rolled oats 1 kg","qty":2,"status":"shipped"},{"sku":"HNY-450","name":"Wildflower honey 450 g","qty":1,"status":"out of stock"},{"sku":"TEA-GRN","name":"Green tea, 50 bags","qty":3,"status":"shipped"}],"shipping":{"carrier":"regional courier","tracking_status":"delivered","delivered_at":"2026-06-05 11:40"},"payment":{"method":"card","captured":27.4,"refunded":0.0},"customer_note":"Where is my honey? I was charged for it."},"questions":{"next_action":{"type":"choice","instructions":"What should the shop do next?","criteria":{"refund_missing_item":"Refund the item that was not sent","resend_order":"Send the whole order again","track_parcel":"Ask the courier where the parcel is","cancel_order":"Cancel the order","change_address":"Update the delivery address","discount_code":"Give a discount code","replace_damaged":"Replace damaged goods","confirm_delivery":"Tell the customer the parcel was delivered","fraud_review":"Flag the order for fraud review","no_action":"Nothing needs to be done"}},"partial":{"type":"noul","instructions":"Was only part of the order shipped?"},"charged_claim":{"type":"noul","instructions":"Does the customer say they paid for something they did not receive?"}}},"gold":{"next_action":"refund_missing_item","partial":"true","charged_claim":"true"},"note":"order (object state), 10-option choice","state_tokens":216,"questions":[{"q":0,"qid":"next_action","type":"choice","keys":["refund_missing_item","resend_order","track_parcel","cancel_order","change_address","discount_code","replace_damaged","confirm_delivery","fraud_review","no_action"],"gold":"refund_missing_item","row_len":332,"state_len":216,"decide":331,"opts":[237,248,261,269,278,287,296,308,320,330],"row_ids_sha256":"5473d95b8a20b4261af1e98edf2011c97f71382032d071f08d2fec9d31362e53","oracle":{"probs":[0.5714622139930725,0.012004455551505089,0.02538725733757019,0.01634262129664421,0.0023002540692687035,0.015482096001505852,0.0015259168576449156,0.15178827941417694,0.006725217215716839,0.19698171317577362],"logits_raw":[2.6227478981018066,-6.671631813049316,-4.869573593139648,-5.92936897277832,-10.64704704284668,-6.059516906738281,-11.634549140930176,-0.5669807195663452,-8.065729141235352,0.06009429693222046],"argmax":0,"top2_margin":0.3744804859161377,"near_tie":false},"graph":{"p":[0.5706941485404968,0.012268897145986557,0.025951405987143517,0.016407674178481102,0.0023494751658290625,0.0157048087567091,0.0015448365593329072,0.15252424776554108,0.0068222093395888805,0.1957322657108307],"max_abs_dp":0.0012494474649429321,"argmax_equal":true}},{"q":1,"qid":"partial","type":"noul","keys":["false","true"],"gold":"true","row_len":232,"state_len":216,"decide":231,"opts":[227,230],"row_ids_sha256":"edb9c6df7238a356b186cffe68ec288eac0072aaac44acfe24486380155b38d5","oracle":{"probs":[0.2644047737121582,0.7355952262878418],"logits_raw":[-0.8009281158447266,1.660939335823059],"argmax":1,"top2_margin":0.4711904525756836,"near_tie":false},"graph":{"p":[0.2659972012042999,0.7340027689933777],"max_abs_dp":0.0015924572944641113,"argmax_equal":true}},{"q":2,"qid":"charged_claim","type":"noul","keys":["false","true"],"gold":"true","row_len":237,"state_len":216,"decide":236,"opts":[232,235],"row_ids_sha256":"6657cdfa92ef11fb0aa412ddafeb2292425f230b979107289a8c018be651699b","oracle":{"probs":[0.20972037315368652,0.7902796268463135],"logits_raw":[-0.7295165061950684,2.4623777866363525],"argmax":1,"top2_margin":0.580559253692627,"near_tie":false},"graph":{"p":[0.21011322736740112,0.7898867130279541],"max_abs_dp":0.000392913818359375,"argmax_equal":true}}]},{"id":"own_j03","source":"own_json","text":"withheld (name screen)","request_sha256":"cd3ac92dfcc934109f0271196b3d501f126b4d0b9077a1cace48b1b136226080","state_tokens":297,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":380,"state_len":297,"decide":379,"opts":[321,339,358,378],"row_ids_sha256":"294817e7c127d91cca3f150ea793a6b4520105401ab1aca4d56a41033c3e234a","oracle":{"probs":[0.8703437447547913,0.030941059812903404,0.05064396932721138,0.04807121306657791],"logits_raw":[2.0026354789733887,-6.025881767272949,-4.840334415435791,-4.965777397155762],"argmax":0,"top2_margin":0.819699764251709,"near_tie":false},"graph":{"p":[0.8688412308692932,0.03112843818962574,0.05100588500499725,0.04902442917227745],"max_abs_dp":0.0015025138854980469,"argmax_equal":true}},{"q":1,"type":"score","n_options":7,"row_len":333,"state_len":297,"decide":332,"opts":[310,314,317,321,324,328,331],"row_ids_sha256":"2f47ca93b1034ef86e528fa4c26a6f3df48bb13aba93870760d054bac25cca71","oracle":{"probs":[0.04908178746700287,0.03114940971136093,0.06207752600312233,0.12242617458105087,0.21290619671344757,0.24308079481124878,0.27927809953689575],"logits_raw":[-2.3178517818450928,-3.411865234375,-1.7526800632476807,-0.11867350339889526,1.2126986980438232,1.5316017866134644,1.8655955791473389],"argmax":6,"top2_margin":0.03619730472564697,"near_tie":false},"graph":{"p":[0.0502658411860466,0.032050326466560364,0.06384274363517761,0.12494443356990814,0.21241280436515808,0.2402854859828949,0.2761983573436737],"max_abs_dp":0.003079742193222046,"argmax_equal":true}},{"q":2,"type":"noul","n_options":2,"row_len":314,"state_len":297,"decide":313,"opts":[309,312],"row_ids_sha256":"94cc0b84c74e88681d01096f0d2a8208ea9a9f9770f7fd99291e0f994492f8ac","oracle":{"probs":[0.06999809294939041,0.9300019145011902],"logits_raw":[-2.4225239753723145,3.801250457763672],"argmax":1,"top2_margin":0.8600038290023804,"near_tie":false},"graph":{"p":[0.07136131823062897,0.9286386966705322],"max_abs_dp":0.001363225281238556,"argmax_equal":true}}]},{"id":"own_j04","source":"own_json","text":"withheld (name screen)","request_sha256":"9264d1704e2ebe3e6882d2f9df50f13e9fc79ccfdeb4cc782631806e9ade432d","state_tokens":157,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":180,"state_len":157,"decide":179,"opts":[166,170,175,178],"row_ids_sha256":"5df13c1b110c602244de90cd027eed7d4d6737bd13dcde282fc2290b113f2cc9","oracle":{"probs":[0.0015651179710403085,0.0006018762942403555,0.9939358830451965,0.0038971668109297752],"logits_raw":[-9.678363800048828,-11.977741241455078,5.849587917327881,-7.4833526611328125],"argmax":2,"top2_margin":0.9900386929512024,"near_tie":false},"graph":{"p":[0.0016145478002727032,0.0006216289475560188,0.9937606453895569,0.0040031312964856625],"max_abs_dp":0.00017523765563964844,"argmax_equal":true}},{"q":1,"type":"noul","n_options":2,"row_len":171,"state_len":157,"decide":170,"opts":[166,169],"row_ids_sha256":"80b7f06ac68dbd141e3eb691f29217780e1a09568772578cd8bcc361036be22d","oracle":{"probs":[0.9941788911819458,0.005821166094392538],"logits_raw":[4.661320686340332,-7.7067790031433105],"argmax":0,"top2_margin":0.9883577227592468,"near_tie":false},"graph":{"p":[0.9941465854644775,0.005853442009538412],"max_abs_dp":3.230571746826172e-05,"argmax_equal":true}},{"q":2,"type":"score","n_options":4,"row_len":196,"state_len":157,"decide":195,"opts":[171,178,186,194],"row_ids_sha256":"de439c5db9ee41557077f11d05e0dddb9000ec21e7b10cd4147e6f6d942ca884","oracle":{"probs":[0.14629992842674255,0.16009673476219177,0.2818213105201721,0.41178205609321594],"logits_raw":[-4.013598918914795,-3.796766996383667,-2.436157703399658,-1.5237332582473755],"argmax":3,"top2_margin":0.12996074557304382,"near_tie":false},"graph":{"p":[0.1466304510831833,0.16114985942840576,0.2829095423221588,0.40931013226509094],"max_abs_dp":0.002471923828125,"argmax_equal":true}}]},{"id":"own_j05","source":"own_json","text":"withheld (name screen)","request_sha256":"f8e41850826034ff1ae4ef229b0cdf169dd2f83a3cbb07ba71ff61298e1df487","state_tokens":133,"questions":[{"q":0,"type":"choice","n_options":3,"row_len":198,"state_len":133,"decide":197,"opts":[162,179,196],"row_ids_sha256":"a9c10f4ff674b883fe0483b48243f69873a663e685d894ffd804a9f08330fb0f","oracle":{"probs":[0.8338309526443481,0.10375454276800156,0.062414463609457016],"logits_raw":[1.0057718753814697,-4.0084428787231445,-5.231271743774414],"argmax":0,"top2_margin":0.7300764322280884,"near_tie":false},"graph":{"p":[0.833026111125946,0.10391916334629059,0.06305476278066635],"max_abs_dp":0.0008048415184020996,"argmax_equal":true}},{"q":1,"type":"noul","n_options":2,"row_len":153,"state_len":133,"decide":152,"opts":[142,151],"row_ids_sha256":"5ce6b272a9fd3d8dd304e70fbb53994ad5fc4467ce284a224dcc8385c35388a2","oracle":{"probs":[0.13677017390727997,0.8632298707962036],"logits_raw":[-2.5969133377075195,1.8359428644180298],"argmax":1,"top2_margin":0.7264596819877625,"near_tie":false},"graph":{"p":[0.13816700875759125,0.8618329763412476],"max_abs_dp":0.0013968944549560547,"argmax_equal":true}},{"q":2,"type":"score","n_options":7,"row_len":169,"state_len":133,"decide":168,"opts":[145,150,153,157,160,164,167],"row_ids_sha256":"2f4b13996697864c2f9a2157477fed51ada5fbb034c163dc94091ada66a854b9","oracle":{"probs":[0.11629471182823181,0.05286753550171852,0.1182284727692604,0.2122940570116043,0.21082766354084015,0.16927093267440796,0.12021666765213013],"logits_raw":[-1.6264948844909668,-3.5232760906219482,-1.586815595626831,-0.17842620611190796,-0.19510328769683838,-0.7233285903930664,-1.546690821647644],"argmax":3,"top2_margin":0.0014663934707641602,"near_tie":true},"graph":{"p":[0.11733843386173248,0.05309362709522247,0.11859815567731857,0.2128337174654007,0.21034017205238342,0.16821452975273132,0.11958133429288864],"max_abs_dp":0.0010564029216766357,"argmax_equal":true}}]},{"id":"own_a01","source":"own_json","request":{"model":"kev-0.8b","state":[{"ticket":"HD-101","summary":"Printer on floor 2 jams on every job","reported":"09:05","affects":"1 team"},{"ticket":"HD-102","summary":"Card readers at the main entrance reject all badges","reported":"09:12","affects":"whole building"},{"ticket":"HD-103","summary":"Request for a second monitor","reported":"09:30","affects":"1 person"},{"ticket":"HD-104","summary":"Smell of burning plastic near the server room","reported":"09:41","affects":"unknown"}],"questions":{"first":{"type":"choice","instructions":"Which ticket should be handled first?","criteria":{"HD-101":"Printer jams","HD-102":"Entrance card readers","HD-103":"Second monitor","HD-104":"Burning smell"}},"hazard":{"type":"noul","instructions":"Does any ticket describe a possible safety hazard?"},"load":{"type":"score","instructions":"Is the queue overloaded?","criteria":["normal","overloaded"]}}},"gold":{"first":"HD-104","hazard":"true","load":"0"},"note":"helpdesk queue (top-level array state), 2 levels","state_tokens":149,"questions":[{"q":0,"qid":"first","type":"choice","keys":["HD-101","HD-102","HD-103","HD-104"],"gold":"HD-104","row_len":199,"state_len":149,"decide":198,"opts":[166,177,187,197],"row_ids_sha256":"2d0e633f74a562c7bf0d26de3400d7edabad48fd756c6af6f0aabc8dfd5b8d26","oracle":{"probs":[0.05393663048744202,0.5021153092384338,0.009280506521463394,0.4346676170825958],"logits_raw":[-4.424733638763428,0.94321209192276,-8.659125328063965,0.5961437225341797],"argmax":1,"top2_margin":0.06744769215583801,"near_tie":false},"graph":{"p":[0.05473655089735985,0.4961101710796356,0.009358023293316364,0.43979528546333313],"max_abs_dp":0.006005138158798218,"argmax_equal":true}},{"q":1,"qid":"hazard","type":"noul","keys":["false","true"],"gold":"true","row_len":166,"state_len":149,"decide":165,"opts":[161,164],"row_ids_sha256":"202a2840d93239cd50fa1530a967ccd33e77f7e230225bb6ec3a90a413ce413d","oracle":{"probs":[0.09005801379680634,0.9099420309066772],"logits_raw":[-3.0041215419769287,2.5608959197998047],"argmax":1,"top2_margin":0.8198840022087097,"near_tie":false},"graph":{"p":[0.09090164303779602,0.9090983867645264],"max_abs_dp":0.0008436441421508789,"argmax_equal":true}},{"q":2,"qid":"load","type":"score","keys":["0","1"],"gold":"0","row_len":163,"state_len":149,"decide":162,"opts":[157,161],"row_ids_sha256":"a6187c673f8c1444978efb7b9c830a045865d83f0c8184a632793a1d4af3c9a2","oracle":{"probs":[0.5434985160827637,0.45650145411491394],"logits_raw":[-2.759812355041504,-3.1795120239257812],"argmax":0,"top2_margin":0.08699706196784973,"near_tie":false},"graph":{"p":[0.542426586151123,0.45757338404655457],"max_abs_dp":0.001071929931640625,"argmax_equal":true}}]},{"id":"own_m01","source":"own_text","request":{"model":"kev-0.8b","state":"Flat to let - Brindlecourt Road, Ostlebury\nTwo-bedroom flat on the third floor of a building with a lift. Rent 1,150 per month, deposit equal to five weeks' rent, minimum tenancy 12 months. Available from 1 September. Bills are not included except water. Pets considered by agreement (small dogs and cats only, no other animals). The flat is unfurnished apart from a fridge and a washing machine. Bike storage in the basement; no parking space. Viewings on Saturdays between 10:00 and 13:00. Contact the agent, Pilar, to book a viewing.","questions":{"bedrooms":{"type":"choice","instructions":"How many bedrooms does the flat have?","criteria":{"one":null,"two":null,"three":null}},"lift":{"type":"noul","instructions":"Is there a lift?"},"furnished":{"type":"score","instructions":"How furnished is the flat?","criteria":["unfurnished","partly furnished","fully furnished"]},"rabbit":{"type":"choice","instructions":"Could a tenant keep a rabbit?","criteria":{"yes":"Allowed","by_agreement":"Only with the landlord's agreement","no":"Not allowed"}},"parking":{"type":"noul","instructions":"Does the flat come with a parking space?"},"bill":{"type":"choice","instructions":"Which bill is included in the rent?","criteria":{"electricity":null,"gas":null,"water":null,"internet":null,"none":"No bill is included"}},"weekday_viewing":{"type":"noul","instructions":"Can the flat be viewed on a weekday?"},"term":{"type":"score","instructions":"How long is the minimum tenancy?","criteria":["under 6 months","6 months","12 months","over 12 months"]}}},"gold":{"bedrooms":"two","lift":"true","furnished":"0","rabbit":"no","parking":"false","bill":"water","weekday_viewing":"false","term":"2"},"note":"flat listing, 8 questions","state_tokens":137,"questions":[{"q":0,"qid":"bedrooms","type":"choice","keys":["one","two","three"],"gold":"two","row_len":156,"state_len":137,"decide":155,"opts":[148,151,154],"row_ids_sha256":"0e5e4c256b6680393a094fec5b76789fd6cf60963915ba5615bd182208596f23","oracle":{"probs":[0.007502803113311529,0.9874116778373718,0.005085615441203117],"logits_raw":[-8.346052169799805,3.395014762878418,-9.281670570373535],"argmax":1,"top2_margin":0.9799088835716248,"near_tie":false},"graph":{"p":[0.007515734527260065,0.9873793125152588,0.005104941315948963],"max_abs_dp":3.236532211303711e-05,"argmax_equal":true}},{"q":1,"qid":"lift","type":"noul","keys":["false","true"],"gold":"true","row_len":150,"state_len":137,"decide":149,"opts":[145,148],"row_ids_sha256":"284d55ae3bb6b3f8eb459e12594bc2ca51bd5fa1e868ccdea58f664699e743c1","oracle":{"probs":[0.010419528000056744,0.9895804524421692],"logits_raw":[-6.440659523010254,4.515527725219727],"argmax":1,"top2_margin":0.9791609048843384,"near_tie":false},"graph":{"p":[0.010549414902925491,0.9894506335258484],"max_abs_dp":0.0001298869028687477,"argmax_equal":true}},{"q":2,"qid":"furnished","type":"score","keys":["0","1","2"],"gold":"0","row_len":159,"state_len":137,"decide":158,"opts":[148,153,157],"row_ids_sha256":"74b9dd6b2b72a4e8425630ee462b6e85e2446ed497ef6f8d345926bbf80fd8b2","oracle":{"probs":[0.5909679532051086,0.40521174669265747,0.0038202833384275436],"logits_raw":[2.0557289123535156,1.147801160812378,-10.074219703674316],"argmax":0,"top2_margin":0.18575620651245117,"near_tie":false},"graph":{"p":[0.5908618569374084,0.4052184522151947,0.003919620998203754],"max_abs_dp":0.00010609626770019531,"argmax_equal":true}},{"q":3,"qid":"rabbit","type":"choice","keys":["yes","by_agreement","no"],"gold":"no","row_len":169,"state_len":137,"decide":168,"opts":[149,161,167],"row_ids_sha256":"7ea31b649b32f2538db0e70ad0996fb14f9109ba167c910322c009c6436aabac","oracle":{"probs":[0.02416291832923889,0.14040842652320862,0.8354287147521973],"logits_raw":[-6.959805965423584,-2.725792646408081,1.565131664276123],"argmax":2,"top2_margin":0.695020318031311,"near_tie":false},"graph":{"p":[0.023538656532764435,0.136661097407341,0.8398002982139587],"max_abs_dp":0.004371583461761475,"argmax_equal":true}},{"q":4,"qid":"parking","type":"noul","keys":["false","true"],"gold":"false","row_len":154,"state_len":137,"decide":153,"opts":[149,152],"row_ids_sha256":"ec5341eb73d150091b76f0622bf38aac9a8b2b8f176decad5b99d6d166a0d9d4","oracle":{"probs":[0.9960099458694458,0.003990098834037781],"logits_raw":[6.953341484069824,-6.327914237976074],"argmax":0,"top2_margin":0.9920198321342468,"near_tie":false},"graph":{"p":[0.9959755539894104,0.004024453926831484],"max_abs_dp":3.439188003540039e-05,"argmax_equal":true}},{"q":5,"qid":"bill","type":"choice","keys":["electricity","gas","water","internet","none"],"gold":"water","row_len":168,"state_len":137,"decide":167,"opts":[149,152,155,158,166],"row_ids_sha256":"53677a9e92596f0d1be3616a512a93597b6090c49d5c6c5900bb9c4775f9add2","oracle":{"probs":[0.0037448140792548656,0.004332194570451975,0.9345604181289673,0.0073315599001944065,0.05003092437982559],"logits_raw":[-10.10379409790039,-9.753226280212402,3.1768903732299805,-8.487370491027832,-3.866663932800293],"argmax":2,"top2_margin":0.8845294713973999,"near_tie":false},"graph":{"p":[0.0037995914462953806,0.0043738954700529575,0.9341897368431091,0.007371884770691395,0.050264909863471985],"max_abs_dp":0.0003706812858581543,"argmax_equal":true}},{"q":6,"qid":"weekday_viewing","type":"noul","keys":["false","true"],"gold":"false","row_len":154,"state_len":137,"decide":153,"opts":[149,152],"row_ids_sha256":"a5c38b3987e72bcbcdccf3836f96d0732efb0adf42a16b79dace04c92209140c","oracle":{"probs":[0.9371252059936523,0.06287479400634766],"logits_raw":[4.44718074798584,-2.0531764030456543],"argmax":0,"top2_margin":0.8742504119873047,"near_tie":false},"graph":{"p":[0.9369328618049622,0.06306716054677963],"max_abs_dp":0.00019236654043197632,"argmax_equal":true}},{"q":7,"qid":"term","type":"score","keys":["0","1","2","3"],"gold":"2","row_len":169,"state_len":137,"decide":168,"opts":[151,155,160,167],"row_ids_sha256":"cf213e634f7fa28ed351d56320a67f69a6325a7f1864dc3ff4b83d453bb010e6","oracle":{"probs":[0.005555816926062107,0.006325599271804094,0.9755743741989136,0.012544273398816586],"logits_raw":[-7.278313636779785,-6.9661054611206055,5.156588077545166,-5.318780899047852],"argmax":2,"top2_margin":0.9630300998687744,"near_tie":false},"graph":{"p":[0.005726401228457689,0.006419702433049679,0.9749292135238647,0.012924584560096264],"max_abs_dp":0.0006451606750488281,"argmax_equal":true}}]},{"id":"own_L01","source":"own_long","request":{"model":"kev-0.8b","state":"BRINDLECOURT LOGISTICS - parcel hub 2, cross-belt sorter S1, controller log (excerpt)\nshift: early (06:00-14:00)   supervisor on duty: Yusuf   maintenance on call: Malik\n\n06:00:00 INFO  throughput 1712 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n06:03:00 INFO  throughput 1698 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n06:06:00 INFO  throughput 1731 parcels/h; lanes 1-12 active; scanner read rate 99.5%\n06:09:00 INFO  throughput 1704 parcels/h; lanes 1-12 active; scanner read rate 99.7%\n06:09:41 INFO  diverter D4 calibration check passed (offset 0.39 mm)\n06:12:00 INFO  throughput 1689 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n06:12:17 WARN  scanner T2 no-read on parcel 48367; parcel sent to manual lane\n06:15:00 INFO  throughput 1720 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n06:18:00 INFO  throughput 1716 parcels/h; lanes 1-12 active; scanner read rate 99.5%\n06:21:00 INFO  throughput 1702 parcels/h; lanes 1-12 active; scanner read rate 99.7%\n06:24:00 INFO  throughput 1695 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n06:27:00 INFO  throughput 1727 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n06:27:41 INFO  diverter D4 calibration check passed (offset 0.57 mm)\n06:30:00 INFO  throughput 1709 parcels/h; lanes 1-12 active; scanner read rate 99.5%\n06:33:00 INFO  throughput 1683 parcels/h; lanes 1-12 active; scanner read rate 99.7%\n06:33:17 WARN  scanner T2 no-read on parcel 48640; parcel sent to manual lane\n06:36:00 INFO  throughput 1718 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n06:39:00 INFO  throughput 1700 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n06:40:00 INFO  throughput 1694 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n06:41:12 WARN  lane 7 drive motor current 8.9 A (limit 8.0 A) for 5 s\n06:41:58 WARN  lane 7 drive motor current 9.4 A (limit 8.0 A) for 12 s\n06:42:40 WARN  lane 7 belt speed 1.62 m/s (setpoint 1.80 m/s)\n06:43:05 ERROR lane 7 jam detected: photo-eye PE7-3 blocked for more than 3 s\n06:43:06 ERROR sorter S1 stopped by safety controller (cause: jam lane 7); all lanes halted\n06:43:06 INFO  infeed conveyors paused; parcels held on accumulation belts\n06:43:40 INFO  alarm acknowledged by Yusuf at the control desk\n06:44:15 INFO  maintenance call raised to Malik (priority 1)\n06:46:02 INFO  scanner T1 and T2 idle (no parcels); self-test passed\n06:47:30 INFO  accumulation belts at 64% capacity\n06:50:12 INFO  Malik on site at lane 7; lock-out applied to lane 7 drive\n06:52:48 INFO  accumulation belts at 81% capacity; infeed doors 3 and 4 closed to new trailers\n06:55:20 INFO  jam cause reported by Malik: crushed carton wedged under the lane 7 belt at the second roller\n06:58:41 INFO  carton removed from lane 7; roller R7-2 inspected, no damage found\n07:00:05 INFO  lane 7 belt tension measured below specification; adjustment started\n07:03:12 INFO  lane 7 belt tension adjusted; test run at low speed 0.5 m/s for 60 s passed\n07:04:25 INFO  lock-out removed from lane 7 drive\n07:05:30 INFO  sorter S1 restarted by Yusuf; stoppage 22 min 24 s\n07:05:31 INFO  infeed conveyors resumed; infeed doors 3 and 4 reopened\n07:06:00 INFO  lane 7 drive motor current 6.1 A (limit 8.0 A)\n07:09:00 INFO  throughput 1840 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.0%\n07:12:00 INFO  throughput 1866 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.3%\n07:15:00 INFO  throughput 1851 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.6%\n07:15:44 WARN  scanner T1 no-read on parcel 49870; parcel sent to manual lane\n07:18:00 INFO  throughput 1792 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.0%\n07:21:00 INFO  throughput 1748 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.3%\n07:21:09 INFO  accumulation belts back below 10% capacity\n07:24:00 INFO  throughput 1731 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.6%\n07:27:00 INFO  throughput 1722 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.0%\n07:27:30 INFO  lane 7 drive motor current 5.9 A (limit 8.0 A); belt speed 1.80 m/s\n07:30:00 INFO  throughput 1715 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.3%\n07:33:00 INFO  shift note by Yusuf: lane 7 belt tension to be checked again at 10:00; carton supplier to be told about weak boxes\n07:33:05 INFO  sorter S1 running normally; no active alarms","questions":{"component":{"type":"choice","instructions":"Which component caused the stoppage?","criteria":{"scanner":"Barcode scanner","lane_7":"Conveyor lane 7","diverter":"Diverter arm","power":"Power supply","software":"Control software"}},"normal_at_end":{"type":"noul","instructions":"Was the sorter running normally at the end of the log?"},"stoppage":{"type":"score","instructions":"How long was the sorter fully stopped?","criteria":["under 2 minutes","2 to 5 minutes","5 to 10 minutes","10 to 20 minutes","20 to 40 minutes","over 40 minutes"]}}},"gold":{"component":"lane_7","normal_at_end":"true","stoppage":"4"},"note":"long state: sorter log","state_tokens":1746,"questions":[{"q":0,"qid":"component","type":"choice","keys":["scanner","lane_7","diverter","power","software"],"gold":"lane_7","row_len":1793,"state_len":1746,"decide":1792,"opts":[1759,1769,1779,1785,1791],"row_ids_sha256":"1e9677b0f901ba2b8f62149138ca290b171cf88e47165ee53d74266c7a28899e","oracle":{"probs":[0.005787316709756851,0.9585761427879333,0.006416614167392254,0.0078064980916678905,0.021413419395685196],"logits_raw":[-7.726229667663574,4.56815767288208,-7.477872848510742,-7.006128311157227,-4.578275680541992],"argmax":1,"top2_margin":0.9371626973152161,"near_tie":false},"graph":{"p":[0.005789552815258503,0.9584445953369141,0.0064184414222836494,0.007774425204843283,0.02157297357916832],"max_abs_dp":0.00015955418348312378,"argmax_equal":true}},{"q":1,"qid":"normal_at_end","type":"noul","keys":["false","true"],"gold":"true","row_len":1766,"state_len":1746,"decide":1765,"opts":[1761,1764],"row_ids_sha256":"46af37073f0e2c88202f8f5ea48afa14d205a542713d8e51308fed55dad5863f","oracle":{"probs":[0.050303347408771515,0.9496966600418091],"logits_raw":[-4.6396989822387695,2.4294466972351074],"argmax":1,"top2_margin":0.8993933200836182,"near_tie":false},"graph":{"p":[0.0502963550388813,0.9497036337852478],"max_abs_dp":6.992369890213013e-06,"argmax_equal":true}},{"q":2,"qid":"stoppage","type":"score","keys":["0","1","2","3","4","5"],"gold":"4","row_len":1802,"state_len":1746,"decide":1801,"opts":[1760,1767,1775,1784,1793,1800],"row_ids_sha256":"e22b1a28755e4ce18e14dfb2091c08da5d75dc6e776ad65cfbc091109fdfd391","oracle":{"probs":[0.03952183201909065,0.13238869607448578,0.13758593797683716,0.22705300152301788,0.4256904721260071,0.03776004537940025],"logits_raw":[-8.104804992675781,-5.196158409118652,-5.103509902954102,-3.89823579788208,-2.3859634399414062,-8.214526176452637],"argmax":4,"top2_margin":0.1986374706029892,"near_tie":false},"graph":{"p":[0.04059120640158653,0.133712500333786,0.1368081271648407,0.22729499638080597,0.42359697818756104,0.037996143102645874],"max_abs_dp":0.002093493938446045,"argmax_equal":true}}]},{"id":"own_L02","source":"own_long","text":"withheld (name screen)","request_sha256":"49a1faf4a2fcbd6cb7ff2b9b342355b1908cceb9d64c89bc763ab265b89754f7","state_tokens":1477,"questions":[{"q":0,"type":"score","n_options":5,"row_len":1518,"state_len":1477,"decide":1517,"opts":[1495,1500,1505,1510,1516],"row_ids_sha256":"de12001f10d52572cb39c14f22e60175edff20a769f0931f6c98b4abca0f02e0","oracle":{"probs":[0.03940492123365402,0.0024226040113717318,0.002537293592467904,0.9507192969322205,0.004915900994092226],"logits_raw":[-1.8277928829193115,-8.538381576538086,-8.427087783813477,5.831454277038574,-6.835782051086426],"argmax":3,"top2_margin":0.9113143682479858,"near_tie":false},"graph":{"p":[0.03975889831781387,0.0024820088874548674,0.002613229211419821,0.9501180052757263,0.0050277989357709885],"max_abs_dp":0.0006012916564941406,"argmax_equal":true}},{"q":1,"type":"noul","n_options":2,"row_len":1497,"state_len":1477,"decide":1496,"opts":[1492,1495],"row_ids_sha256":"44138ee3e4a88a5fd88be6dae900a8f170b6024e319179025258b43ca6f317da","oracle":{"probs":[0.9881781935691833,0.011821863241493702],"logits_raw":[5.39301061630249,-5.255955696105957],"argmax":0,"top2_margin":0.9763563275337219,"near_tie":false},"graph":{"p":[0.9880210757255554,0.011978930793702602],"max_abs_dp":0.0001571178436279297,"argmax_equal":true}},{"q":2,"type":"choice","n_options":4,"row_len":1539,"state_len":1477,"decide":1538,"opts":[1506,1522,1530,1537],"row_ids_sha256":"cb13ba9675e3162d6680f7d1b00966e3a4ee7446c4b2cc7cd39b5fc927152ee5","oracle":{"probs":[0.9412030577659607,0.0037783070001751184,0.043812789022922516,0.011205767281353474],"logits_raw":[3.1966605186462402,-10.079641342163086,-4.183255195617676,-7.463897705078125],"argmax":0,"top2_margin":0.8973902463912964,"near_tie":false},"graph":{"p":[0.9412140846252441,0.0038059502840042114,0.04376097023487091,0.011219055391848087],"max_abs_dp":5.1818788051605225e-05,"argmax_equal":true}},{"q":3,"type":"noul","n_options":2,"row_len":1499,"state_len":1477,"decide":1498,"opts":[1494,1497],"row_ids_sha256":"25eb0e47d6882f607461f2e5184cd637900841e540d893a414493ee65b42d8f7","oracle":{"probs":[0.030631551519036293,0.9693683981895447],"logits_raw":[-3.4746932983398438,4.837280750274658],"argmax":1,"top2_margin":0.9387368559837341,"near_tie":false},"graph":{"p":[0.030866537243127823,0.9691334366798401],"max_abs_dp":0.00023498572409152985,"argmax_equal":true}}]},{"id":"own_L03","source":"own_long","request":{"model":"kev-0.8b","state":"WENDLECOTE RESIDENTS' ASSOCIATION\nMinutes of the Annual General Meeting held on Thursday 12 March 2026 at 19:30 in Wendlecote Village Hall\n\nPresent: 47 members (attendance sheet attached) and 3 guests, among them the ward councillor, Adaeze. In the chair: Gareth. Minutes: Priyanka (secretary).\nApologies: 11 members sent their apologies.\n\n1. Welcome and quorum\nThe chair welcomed members and guests and thanked the hall committee for the use of the room. The constitution requires 30 members to be present for a general meeting to be quorate. With 47 members present, the meeting was declared quorate.\n\n2. Minutes of the last annual general meeting\nThe minutes of the meeting held on 13 March 2025 were approved as a true record, proposed by Hamid and seconded by Ruth. There were no matters arising that were not already on the agenda.\n\n3. Chair's report\nGareth summarised the year. The association ran the summer fair, which raised 2,140 for the hall roof fund, and organised two litter picks along the river path, each with more than twenty volunteers. The campaign for a pedestrian crossing on Mill Lane continued, and the council has now agreed to carry out a traffic survey in May. Membership rose from 212 to 236 households. He thanked the committee and the volunteers who deliver the newsletter to every house four times a year.\n\n4. Treasurer's report\nThe outgoing treasurer, Colm, presented the accounts for the year. Income was 6,480: membership fees 2,832, the summer fair 2,140 and donations 1,508. Expenditure was 4,915, mostly newsletter printing (1,620), insurance (740), hall hire (600) and equipment for the river path clean-up (1,955). The year closed with a surplus of 1,565 and a bank balance of 9,310. The accounts had been checked by an independent examiner, who found no issues. Members approved the accounts by a show of hands, with no votes against.\n\n5. Reports from the working groups\nEnvironment group (Hannah): the group planted 150 native bulbs on the verge outside the school and repaired two of the four bird boxes on the green. It would like to survey the river path for invasive plants in the summer and is looking for two more volunteers.\nEvents group (Dominic): besides the summer fair, the group ran a quiz night in November that broke even. A film night in the hall is planned for April if a licence can be arranged at a reasonable price.\nRoads and transport group (Hamid): apart from the crossing campaign, the group reported three damaged street lights and a blocked drain on Station Road to the council; two of the lights have been repaired.\n\n6. Hall roof fund\nRuth reported for the hall committee that the roof repair has been quoted at 18,400. With the 2,140 from the summer fair, the fund now stands at 11,750. The committee has applied to two charitable trusts for the rest and expects their answers by the end of May. Work cannot start before September because bats roost in the roof space during the summer, and an ecologist will check the roof before any work begins. Members asked whether the association would cover any shortfall; Ruth said that the committee would come back to a general meeting before committing any of the association's money. One member offered to organise a sponsored walk for the fund in June, and several members volunteered to help.\n\n7. Election of officers\nGareth agreed to stand again as chair and was re-elected unopposed. Priyanka was re-elected as secretary unopposed. Colm stood down as treasurer after six years. Two members were nominated for the post: Odette and Lewis. After short statements from both candidates, a secret ballot was held. Odette received 29 votes and Lewis 16, with 2 spoiled papers. Odette was declared elected as treasurer. The meeting thanked Colm warmly for his years of service.\n\n8. Motion 1: bicycle racks on the village green\nProposed by Ines, seconded by Dominic: \"That the association spends up to 1,200 from its reserves on four bicycle racks at the edge of the village green, subject to council permission.\" Ines explained that bicycles are currently chained to the railings, which damages the paint and blocks the footpath for pushchairs. One member asked whether the racks would attract theft; Ines replied that they would stand in view of the shop's camera. Vote: 31 for, 9 against, 7 abstentions. The motion was carried.\n\n9. Motion 2: annual membership fee\nProposed by the committee: \"That the annual fee rises from 12 to 15 per household from 1 April 2026.\" The treasurer-elect said that this year's surplus makes an increase unnecessary for now. Several members said that a higher fee would put off new households, especially in the new flats on Station Road. Vote: 18 for, 26 against, 3 abstentions. The motion was defeated, and the fee stays at 12.\n\n10. Motion 3: community garden\nProposed by Kwame, seconded by Hannah: \"That the association creates a community garden on the unused plot behind the hall.\" Members supported the idea in principle but asked who would look after the garden and whether the hall committee owns the plot. It was agreed to set up a working group of Kwame, Hannah and Lewis to report to the next general meeting. No vote was taken.\n\n11. Any other business\n(a) Speeding on Mill Lane: several members reported cars driving well above the limit at school times. The secretary will write to the council asking for the May survey to include speed data.\n(b) Newsletter: volunteers are needed to deliver on the east side of the village.\n(c) The ward councillor said that the council's grant scheme for community projects opens in June and that the garden might qualify.\n(d) Recycling: the council will collect glass separately from July; the collection days will be printed in the next newsletter.\n(e) Defibrillator: Hamid reported that the defibrillator outside the shop was checked in February and that its pads expire in October; the committee will order replacements.\n\n12. Date of the next meeting\nThe next general meeting will be held in September; the date will be announced in the summer newsletter. The meeting closed at 21:10.","questions":{"fee_motion":{"type":"choice","instructions":"What happened to the motion on the annual fee?","criteria":{"approved":null,"rejected":null,"deferred":"Postponed to a later meeting or a working group","withdrawn":"Withdrawn before the vote"}},"quorate":{"type":"noul","instructions":"Was the meeting quorate?"},"treasurer":{"type":"choice","instructions":"Who was elected treasurer?","criteria":{"odette":"Odette","lewis":"Lewis"}},"racks_support":{"type":"score","instructions":"How strong was the support for the bicycle racks motion?","criteria":["opposed by most","split","supported by most","unanimous"]}}},"gold":{"fee_motion":"rejected","quorate":"true","treasurer":"odette","racks_support":"2"},"note":"long state: meeting minutes","state_tokens":1431,"questions":[{"q":0,"qid":"fee_motion","type":"choice","keys":["approved","rejected","deferred","withdrawn"],"gold":"rejected","row_len":1475,"state_len":1431,"decide":1474,"opts":[1444,1447,1463,1473],"row_ids_sha256":"281f9a5c14075feb01c99dee3fe2d79d9631848c80b2c62eaa3f2356308a2a79","oracle":{"probs":[0.002276319544762373,0.9744077324867249,0.010191483423113823,0.013124536722898483],"logits_raw":[-10.380332946777344,4.198572635650635,-6.773682594299316,-6.165117263793945],"argmax":1,"top2_margin":0.9612832069396973,"near_tie":false},"graph":{"p":[0.0023102741688489914,0.9743324518203735,0.010206841863691807,0.013150446116924286],"max_abs_dp":7.528066635131836e-05,"argmax_equal":true}},{"q":1,"qid":"quorate","type":"noul","keys":["false","true"],"gold":"true","row_len":1445,"state_len":1431,"decide":1444,"opts":[1440,1443],"row_ids_sha256":"54e159a869f1a86ac6653627e83429dffe8cffcc943f03856cf5487cc624e2d2","oracle":{"probs":[0.013143135234713554,0.9868568778038025],"logits_raw":[-6.618133068084717,3.772696018218994],"argmax":1,"top2_margin":0.973713755607605,"near_tie":false},"graph":{"p":[0.01321406476199627,0.9867859482765198],"max_abs_dp":7.092952728271484e-05,"argmax_equal":true}},{"q":2,"qid":"treasurer","type":"choice","keys":["odette","lewis"],"gold":"odette","row_len":1451,"state_len":1431,"decide":1450,"opts":[1443,1449],"row_ids_sha256":"99eb4bc74f422e56854e7de804a63f14b869a212bf13de2c6a2e69d16501507e","oracle":{"probs":[0.993480920791626,0.006519101560115814],"logits_raw":[5.133867263793945,-6.960090637207031],"argmax":0,"top2_margin":0.986961841583252,"near_tie":false},"graph":{"p":[0.9933815002441406,0.006618451327085495],"max_abs_dp":9.942054748535156e-05,"argmax_equal":true}},{"q":3,"qid":"racks_support","type":"score","keys":["0","1","2","3"],"gold":"2","row_len":1463,"state_len":1431,"decide":1462,"opts":[1448,1451,1456,1461],"row_ids_sha256":"0e44cad639992f9a1ec1452bc48643479c5181dbbfdfb4d0e421c568f2fae8ab","oracle":{"probs":[0.019440198317170143,0.144740492105484,0.813139796257019,0.022679556161165237],"logits_raw":[-8.097293853759766,-3.266909122467041,0.8858382701873779,-7.726471424102783],"argmax":2,"top2_margin":0.6683993339538574,"near_tie":false},"graph":{"p":[0.01962282508611679,0.14531941711902618,0.8120021820068359,0.02305556833744049],"max_abs_dp":0.0011376142501831055,"argmax_equal":true}}]}],"heldout":{"what":"130 transfer-v4 development records after the fixture's slices (mmlu records 61-100, records 21-30 of each other source, score records 21-40); used to test choices made on the fixture, never to make them","records":[{"id":"tv4h_00","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":60,"_meta":{"row":282,"text_sha256":"f764b3d9187ad0e0c6e98ca818eebbf3481f738e0f2e562377c125f27af714f2","row_sha256":"2048a5ee51c3949f2022371d25b2190325890748995426e82689d42e865c0268","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/282","group_id":"mmlu/test/282","variant":"clean"}},"state_tokens":12,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":59,"state_len":12,"decide":58,"opts":[30,41,48,57],"row_ids_sha256":"5dd37369d5cd2cbb9a084a7aa9833c11149d3a58d9081f4eef004eae85d649ed","oracle":{"probs":[0.4746726155281067,0.04107768088579178,0.05167515203356743,0.432574599981308],"logits_raw":[0.022543281316757202,-5.865447044372559,-5.313229560852051,-0.20090797543525696],"argmax":0,"top2_margin":0.042098015546798706,"near_tie":false},"graph":{"p":[0.4736717939376831,0.04168267920613289,0.052147939801216125,0.4324975907802582],"max_abs_dp":0.001000821590423584,"argmax_equal":true}}]},{"id":"tv4h_01","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":61,"_meta":{"row":8524,"text_sha256":"822364408ed0625c0b79c047029facb8b81fe5b5e08ad2add9a92236a5d5b269","row_sha256":"20e11bd9c0a720cb3a8f6acb69e9a3b4fefa0594c4c6d6317476c296ea756038","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/8524","group_id":"mmlu/test/8524","variant":"clean"}},"state_tokens":86,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":127,"state_len":86,"decide":126,"opts":[100,108,116,125],"row_ids_sha256":"269e84ae132257cbb148ba7d9a16a3c3fb426ced21f8ca7696b4e3e1d1c384b2","oracle":{"probs":[0.30009880661964417,0.28078126907348633,0.1210072785615921,0.298112690448761],"logits_raw":[-2.6951358318328857,-2.855224370956421,-4.880457401275635,-2.7111124992370605],"argmax":0,"top2_margin":0.0019861161708831787,"near_tie":true},"graph":{"p":[0.29999375343322754,0.2815800905227661,0.12054955214262009,0.29787659645080566],"max_abs_dp":0.0007988214492797852,"argmax_equal":true}}]},{"id":"tv4h_02","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":62,"_meta":{"row":2715,"text_sha256":"58baa2441381d1ab03a94d50b013407342206bbe038af9f52b19e3755feeb54f","row_sha256":"2184b9fdf77f4a55d63eeb73ab996cf18d39f76749859d27707940163931cfb8","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/2715","group_id":"mmlu/test/2715","variant":"clean"}},"state_tokens":16,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":84,"state_len":16,"decide":83,"opts":[41,54,67,82],"row_ids_sha256":"488a5d9f60d3ee394c60889ab9abdd731276901941dda5ed54554a24563897bd","oracle":{"probs":[0.08315582573413849,0.043452635407447815,0.02014969289302826,0.8532417416572571],"logits_raw":[-4.914401054382324,-6.476035118103027,-8.325042724609375,0.6876688599586487],"argmax":3,"top2_margin":0.7700859308242798,"near_tie":false},"graph":{"p":[0.08537283539772034,0.044342078268527985,0.020496735349297523,0.8497883081436157],"max_abs_dp":0.0034534335136413574,"argmax_equal":true}}]},{"id":"tv4h_03","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":63,"_meta":{"row":7350,"text_sha256":"4c6704fa4aecbfc95541b77dafcbfea7dac268e7aafcca3031e7ae007f5651d0","row_sha256":"2202beac91b1bdbc4e4a3048b09c6bc85f42db942a7b2afd39652b4f6055cc3d","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/7350","group_id":"mmlu/test/7350","variant":"clean"}},"state_tokens":21,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":67,"state_len":21,"decide":66,"opts":[37,49,57,65],"row_ids_sha256":"bbbef282ce19e530a49437e266e59fb257836c752ada5d7193980dcd83ea4221","oracle":{"probs":[0.13216856122016907,0.3864857256412506,0.22925467789173126,0.2520909905433655],"logits_raw":[-3.893962860107422,-1.3122309446334839,-2.5688180923461914,-2.3403477668762207],"argmax":1,"top2_margin":0.13439473509788513,"near_tie":false},"graph":{"p":[0.13283975422382355,0.3843117952346802,0.23035162687301636,0.2524968087673187],"max_abs_dp":0.0021739304065704346,"argmax_equal":true}}]},{"id":"tv4h_04","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":64,"_meta":{"row":4527,"text_sha256":"fa2936944c569913a16b3b985b605e5ad0bb3b42d27eea2a892d448bfbe0ed57","row_sha256":"224bf654f98530f4b7096d42d6917feb13dfbdf561cb6a377b129a4b414a210c","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4527","group_id":"mmlu/test/4527","variant":"clean"}},"state_tokens":43,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":140,"state_len":43,"decide":139,"opts":[72,94,116,138],"row_ids_sha256":"3f9b0ba6e8a28690c10c3008209df1e601a43c802d532d652dbd1dd2bac2e763","oracle":{"probs":[0.007607422303408384,0.008333966135978699,0.972274124622345,0.011784583330154419],"logits_raw":[-7.4775543212890625,-7.258086204528809,4.193024158477783,-6.424502372741699],"argmax":2,"top2_margin":0.9604895114898682,"near_tie":false},"graph":{"p":[0.007699638605117798,0.008409772999584675,0.9720083475112915,0.011882305145263672],"max_abs_dp":0.0002657771110534668,"argmax_equal":true}}]},{"id":"tv4h_05","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":65,"_meta":{"row":6364,"text_sha256":"7371f764a68ff91e345aba45d7bfb1528e39c959702df4406faea501c20fa2cf","row_sha256":"225b3e9b8e5698a9c1631bdfab0fb9b31a57be67b50b55cdd661b8c3e8cf279f","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6364","group_id":"mmlu/test/6364","variant":"clean"}},"state_tokens":16,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":55,"state_len":16,"decide":54,"opts":[30,37,45,53],"row_ids_sha256":"186557d036240b5273db6a9d372a5327e54ebf2e8e8c2104af1dfbf09395edee","oracle":{"probs":[0.19632546603679657,0.35237929224967957,0.025799747556447983,0.425495445728302],"logits_raw":[-2.279385805130005,-0.8720046281814575,-7.162245750427246,-0.41835325956344604],"argmax":3,"top2_margin":0.07311615347862244,"near_tie":false},"graph":{"p":[0.19585418701171875,0.3528103530406952,0.026212720200419426,0.4251227080821991],"max_abs_dp":0.0004712790250778198,"argmax_equal":true}}]},{"id":"tv4h_06","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":66,"_meta":{"row":4563,"text_sha256":"5331cc1333fb17a37a60d99714e5693ae650cae7f65955370422a480a217f7c8","row_sha256":"23f533cd5a96dd4943b45ee9c63c10f80bbcd70bef2db77925ca499bf52d1cf5","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4563","group_id":"mmlu/test/4563","variant":"clean"}},"state_tokens":23,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":90,"state_len":23,"decide":89,"opts":[46,61,77,88],"row_ids_sha256":"55c5afdb81fa3089e1acfef3d2201bc87300ea5327cbe1d0e081d77a6f5b723f","oracle":{"probs":[0.3130136728286743,0.08826939761638641,0.08595883101224899,0.5127580761909485],"logits_raw":[-4.65197229385376,-7.697678565979004,-7.761498928070068,-3.464448928833008],"argmax":3,"top2_margin":0.19974440336227417,"near_tie":false},"graph":{"p":[0.3182791769504547,0.08864191919565201,0.0875382348895073,0.5055406093597412],"max_abs_dp":0.007217466831207275,"argmax_equal":true}}]},{"id":"tv4h_07","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":67,"_meta":{"row":4441,"text_sha256":"9f80c384e23e619a896a7a680e54dffae97e8e03de73c0ad2094a9032227371e","row_sha256":"241ad0a238016dd3f459f6e5468b9ba43b2d14d94e5945ff512733bea84c7283","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4441","group_id":"mmlu/test/4441","variant":"clean"}},"state_tokens":54,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":107,"state_len":54,"decide":106,"opts":[72,83,94,105],"row_ids_sha256":"3765268055bfb0b111764f982d2c23b7e25e7058faa3622b3a74cf813de7c87b","oracle":{"probs":[0.24486219882965088,0.2556079924106598,0.24320857226848602,0.2563212811946869],"logits_raw":[-2.209820032119751,-2.1064817905426025,-2.226123809814453,-2.0997767448425293],"argmax":3,"top2_margin":0.0007132887840270996,"near_tie":true},"graph":{"p":[0.24486662447452545,0.2566481828689575,0.24294181168079376,0.25554341077804565],"max_abs_dp":0.0010401904582977295,"argmax_equal":false}}]},{"id":"tv4h_08","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":68,"_meta":{"row":6910,"text_sha256":"08add8ef339b19f632d4cd3ed1c45f824889679fc9e7268120b865e29107b853","row_sha256":"243c6556acdb14a65b6e210646059f3904579d224d730ff3e1db1854006fc802","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6910","group_id":"mmlu/test/6910","variant":"clean"}},"state_tokens":50,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":87,"state_len":50,"decide":86,"opts":[64,71,78,85],"row_ids_sha256":"8f7c6940e6b2bc381d74ab811383476bc3eb336bdeaac188d9724bbf8808ec5f","oracle":{"probs":[0.06892278790473938,0.31066766381263733,0.37211811542510986,0.24829146265983582],"logits_raw":[-5.719019889831543,-2.0961415767669678,-1.661879539489746,-2.6353864669799805],"argmax":2,"top2_margin":0.061450451612472534,"near_tie":false},"graph":{"p":[0.06917395442724228,0.30998945236206055,0.3737071752548218,0.24712945520877838],"max_abs_dp":0.001589059829711914,"argmax_equal":true}}]},{"id":"tv4h_09","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":69,"_meta":{"row":8800,"text_sha256":"811bf5bf013a07b39b29b5f0e44cf591f87cd435de584630cfdc16120e0258d6","row_sha256":"24905e1af1615c518414f5cd372639176ddf690d141547b1ac73c59abc4592a5","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/8800","group_id":"mmlu/test/8800","variant":"clean"}},"state_tokens":77,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":118,"state_len":77,"decide":117,"opts":[91,99,107,116],"row_ids_sha256":"3662e94c064af6501080d8280e11a4c446e0a5845109921b04188d2f18f38f39","oracle":{"probs":[0.49143186211586,0.26663830876350403,0.18579144775867462,0.056138359010219574],"logits_raw":[-3.413367748260498,-4.884499549865723,-5.753728866577148,-8.633302688598633],"argmax":0,"top2_margin":0.22479355335235596,"near_tie":false},"graph":{"p":[0.49374285340309143,0.26620209217071533,0.1840510070323944,0.05600402131676674],"max_abs_dp":0.0023109912872314453,"argmax_equal":true}}]},{"id":"tv4h_10","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":70,"_meta":{"row":4472,"text_sha256":"e74dabb87afea0a70e21f777eeaf42a44d31d8d97f43f2d08389fe444f57a4de","row_sha256":"251010676810510fb50874da455582cb17fe90795b76223c2a0dd62d52d95724","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4472","group_id":"mmlu/test/4472","variant":"clean"}},"state_tokens":93,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":179,"state_len":93,"decide":178,"opts":[122,145,168,177],"row_ids_sha256":"9d8032cb6fe609b8f796944c319a1a6294b2ff3c27b44bfcd166a9da5c4a678c","oracle":{"probs":[0.10908767580986023,0.3102836012840271,0.47853392362594604,0.10209472477436066],"logits_raw":[-4.916651725769043,-2.4015235900878906,-1.3591256141662598,-5.076054573059082],"argmax":2,"top2_margin":0.16825032234191895,"near_tie":false},"graph":{"p":[0.11093530058860779,0.309591680765152,0.4766119718551636,0.10286109149456024],"max_abs_dp":0.0019219517707824707,"argmax_equal":true}}]},{"id":"tv4h_11","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":71,"_meta":{"row":5801,"text_sha256":"3796fbb987fe807698dcb5b074df04ea08fb2286ddba3337c78225e1c48e4367","row_sha256":"258e79933164b8b9c20d756f1819cac2f8c79959d082348e5b605da73f607886","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/5801","group_id":"mmlu/test/5801","variant":"clean"}},"state_tokens":256,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":299,"state_len":256,"decide":298,"opts":[272,281,289,297],"row_ids_sha256":"23a2922d504dbe69ae91738fd3982e05357105d7fc4035f0c9363af7361766f9","oracle":{"probs":[0.0029606495518237352,0.04206644743680954,0.0012705973349511623,0.9537022709846497],"logits_raw":[-7.591586112976074,-1.2063102722167969,-9.62691593170166,6.303215026855469],"argmax":3,"top2_margin":0.9116358160972595,"near_tie":false},"graph":{"p":[0.0029773868154734373,0.04180196672677994,0.0012884802417829633,0.9539321064949036],"max_abs_dp":0.00026448071002960205,"argmax_equal":true}}]},{"id":"tv4h_12","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":72,"_meta":{"row":8718,"text_sha256":"a32bd31ba4a0d8b9564f68c3f60078adb94db71005b4df272123d4eee902801a","row_sha256":"25a1f2b97bec0c728d9151e77cd9765acbdbff78d4d76321dc548d73d67b397a","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/8718","group_id":"mmlu/test/8718","variant":"clean"}},"state_tokens":90,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":131,"state_len":90,"decide":130,"opts":[104,112,120,129],"row_ids_sha256":"b01c0d6017210f13cdd59d0a58bca50ca78a43046bd545355f578606a3bf94b6","oracle":{"probs":[0.1745661199092865,0.011046789586544037,0.8087263107299805,0.0056608556769788265],"logits_raw":[0.6133867502212524,-6.027705192565918,4.302238941192627,-7.63630485534668],"argmax":2,"top2_margin":0.6341601610183716,"near_tie":false},"graph":{"p":[0.176786869764328,0.01111019216477871,0.8064323663711548,0.005670508835464716],"max_abs_dp":0.0022939443588256836,"argmax_equal":true}}]},{"id":"tv4h_13","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":73,"_meta":{"row":3557,"text_sha256":"198dd2b76fe680e1dacbd1335668b2cf4cb3d4f9fffc588672e7d77a07901af8","row_sha256":"26845c1670c6e75a2d4a6f5cedb8efe497d67a7b7731c69be410252e96d3decc","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/3557","group_id":"mmlu/test/3557","variant":"clean"}},"state_tokens":36,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":76,"state_len":36,"decide":75,"opts":[50,58,66,74],"row_ids_sha256":"d9307742cfd6805219f32d1c2fa979767fefbeb5243fababa5c9b94c4524a5b4","oracle":{"probs":[0.0015315022319555283,0.010210219770669937,0.976097583770752,0.012160630896687508],"logits_raw":[-11.12237548828125,-6.557762145996094,4.414243221282959,-6.137148857116699],"argmax":2,"top2_margin":0.9639369249343872,"near_tie":false},"graph":{"p":[0.001583208329975605,0.010368592105805874,0.975667417049408,0.01238083653151989],"max_abs_dp":0.00043016672134399414,"argmax_equal":true}}]},{"id":"tv4h_14","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":74,"_meta":{"row":13887,"text_sha256":"ad12a30dc2689720958e98441b0bec2fd12b917195620d41c3d708b9952ad1de","row_sha256":"280a540cd9fecd12ca41f8a28ecbd2a22f9f99dee5bb95f89913b8b5faf5c874","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13887","group_id":"mmlu/test/13887","variant":"clean"}},"state_tokens":26,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":60,"state_len":26,"decide":59,"opts":[39,45,51,58],"row_ids_sha256":"46a64590e9110ff6360e87fccefaef5e58b9a12655eece576492fc5cecb27a26","oracle":{"probs":[0.2270582914352417,0.009543998166918755,0.6782194375991821,0.08517825603485107],"logits_raw":[1.7244136333465576,-5.90106725692749,4.3572678565979,-0.6346234083175659],"argmax":2,"top2_margin":0.45116114616394043,"near_tie":false},"graph":{"p":[0.22867625951766968,0.009659957140684128,0.6763442754745483,0.08531957119703293],"max_abs_dp":0.001875162124633789,"argmax_equal":true}}]},{"id":"tv4h_15","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":75,"_meta":{"row":8047,"text_sha256":"e52382280110bfcd67d3a91beccfadf2876e36a576891d8a75b3d9961b4d0449","row_sha256":"28254f59ed7f5a97aa710645d9ab88f8094ef152d148725988eee0ef5c1ac4f3","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/8047","group_id":"mmlu/test/8047","variant":"clean"}},"state_tokens":15,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":49,"state_len":15,"decide":48,"opts":[28,35,41,47],"row_ids_sha256":"5e40d1c70395e357511c59e7cd82ae5600c4e28963e8c0483d56e903baeaae87","oracle":{"probs":[0.03339442238211632,0.0067278980277478695,0.9462159276008606,0.013661758974194527],"logits_raw":[-3.1036505699157715,-6.958446502685547,4.94237756729126,-5.254149436950684],"argmax":2,"top2_margin":0.9128215312957764,"near_tie":false},"graph":{"p":[0.03283531591296196,0.006783580407500267,0.9465219974517822,0.013859054073691368],"max_abs_dp":0.0005591064691543579,"argmax_equal":true}}]},{"id":"tv4h_16","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":76,"_meta":{"row":9900,"text_sha256":"8375b63d6a821982197076188109d3204c7e6732d1d3ff9b7bb610145fef13c0","row_sha256":"293e97cddb795f8c71f60dfa1b09e9c57613ddef1b10b9e9f273f84c6cbad0e1","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/9900","group_id":"mmlu/test/9900","variant":"clean"}},"state_tokens":19,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":76,"state_len":19,"decide":75,"opts":[38,48,58,74],"row_ids_sha256":"6d8fd779d738c549668bb2a7916989a4e004519bfca80050d2ac993f2ac36e31","oracle":{"probs":[0.2362508326768875,0.6714919805526733,0.047110483050346375,0.04514671862125397],"logits_raw":[-0.6603691577911377,1.8530099391937256,-4.539880752563477,-4.642325401306152],"argmax":1,"top2_margin":0.435241162776947,"near_tie":false},"graph":{"p":[0.23636241257190704,0.6705271005630493,0.04738248512148857,0.04572790488600731],"max_abs_dp":0.0009648799896240234,"argmax_equal":true}}]},{"id":"tv4h_17","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":77,"_meta":{"row":12517,"text_sha256":"f2f9b485c469f4be1682eb28ca7e240e67c830cd1c0cff7138ee98b090fccf3f","row_sha256":"29bbf79ec6a3ae443184c74b2792f0167e7745df3c7c59ef3ef2388cd14ba690","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12517","group_id":"mmlu/test/12517","variant":"clean"}},"state_tokens":18,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":57,"state_len":18,"decide":56,"opts":[32,40,47,55],"row_ids_sha256":"0ada28aed66a41b119fbd2e4fe2647c16fe0c9c18ddba0bda3aa23e8f7735d37","oracle":{"probs":[0.04952378198504448,0.5375540256500244,0.32905113697052,0.083871029317379],"logits_raw":[-10.115594863891602,-4.378185272216797,-5.559113502502441,-8.8480224609375],"argmax":1,"top2_margin":0.2085028886795044,"near_tie":false},"graph":{"p":[0.049933139234781265,0.535900354385376,0.32991936802864075,0.08424711227416992],"max_abs_dp":0.0016536712646484375,"argmax_equal":true}}]},{"id":"tv4h_18","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":78,"_meta":{"row":6949,"text_sha256":"257ad45a0015775345cabbacaa6a55bfcbff4f409fb831ba6f533bed3ccbbb38","row_sha256":"2c346d2340c7eebe04b1ac82f63e627335efcdaa19acdece5f30ee104c67cb6e","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6949","group_id":"mmlu/test/6949","variant":"clean"}},"state_tokens":13,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":62,"state_len":13,"decide":61,"opts":[29,41,51,60],"row_ids_sha256":"02c8a5ade2ff2d01143fed6f28dd984a657dc8d76b2d419dade06b5b516801f6","oracle":{"probs":[0.0054949428886175156,0.04613437131047249,0.9388146996498108,0.009555952623486519],"logits_raw":[-6.583436012268066,-1.4640107154846191,5.785561561584473,-5.252080917358398],"argmax":2,"top2_margin":0.8926803469657898,"near_tie":false},"graph":{"p":[0.005571609828621149,0.04660714045166969,0.9381680488586426,0.009653222747147083],"max_abs_dp":0.0006466507911682129,"argmax_equal":true}}]},{"id":"tv4h_19","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":79,"_meta":{"row":4674,"text_sha256":"f72f2f388107a78803b8f5852f79102fe5d0884cd4d62e91641e86443db15aa0","row_sha256":"2c549e4d1c079994107ca839d795cff8fa4ec4915fe76db3404085878da3b0fe","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4674","group_id":"mmlu/test/4674","variant":"clean"}},"state_tokens":52,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":97,"state_len":52,"decide":96,"opts":[67,75,83,95],"row_ids_sha256":"d9f46aa9ef0eb025aeba827b691123432dde398874bfa7e2f7981f961c3009b7","oracle":{"probs":[0.06165368854999542,0.43808281421661377,0.118699811398983,0.38156363368034363],"logits_raw":[-6.117246627807617,-1.3992836475372314,-4.541128158569336,-1.7316322326660156],"argmax":1,"top2_margin":0.05651918053627014,"near_tie":false},"graph":{"p":[0.0615038238465786,0.43500256538391113,0.11751250922679901,0.38598108291625977],"max_abs_dp":0.004417449235916138,"argmax_equal":true}}]},{"id":"tv4h_20","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":480,"_meta":{"row":13292,"text_sha256":"e4f765d62c6ed44999ddbaa502f1b865f19197d1e492af2abef30ac5dfd47750","row_sha256":"010a5fda50061ef619d06b589cf689715a9774f817996fb54d6d627b8499f882","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13292/none_present","group_id":"mmlu/test/13292","variant":"none_present","parent_id":"mmlu/test/13292","none_key":"none_of_these"}},"state_tokens":30,"questions":[{"q":0,"type":"choice","n_options":5,"row_len":108,"state_len":30,"decide":107,"opts":[48,66,80,95,106],"row_ids_sha256":"729b02a558bace7b9485db78176e77bbe8302ab5c1b3bc5131ba795fe7bc601a","oracle":{"probs":[0.30655643343925476,0.13829496502876282,0.3095454275608063,0.12007450312376022,0.12552863359451294],"logits_raw":[-0.2663119435310364,-2.1815590858459473,-0.24296605587005615,-2.521477222442627,-2.4145967960357666],"argmax":2,"top2_margin":0.0029889941215515137,"near_tie":true},"graph":{"p":[0.3085102140903473,0.13865268230438232,0.30858394503593445,0.1196388304233551,0.12461429089307785],"max_abs_dp":0.0019537806510925293,"argmax_equal":true}}]},{"id":"tv4h_21","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":481,"_meta":{"row":13292,"text_sha256":"e4f765d62c6ed44999ddbaa502f1b865f19197d1e492af2abef30ac5dfd47750","row_sha256":"010a5fda50061ef619d06b589cf689715a9774f817996fb54d6d627b8499f882","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13292/none_absent","group_id":"mmlu/test/13292","variant":"none_absent","parent_id":"mmlu/test/13292","none_key":"none_of_these"}},"state_tokens":30,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":97,"state_len":30,"decide":96,"opts":[55,70,84,95],"row_ids_sha256":"4a74de3aadef48fe8faf1264fa388d7ef92d8f39887b3a2e0ff36739756d4beb","oracle":{"probs":[0.2595764696598053,0.18810304999351501,0.3622538149356842,0.19006666541099548],"logits_raw":[-1.5396920442581177,-2.3145880699157715,-0.7377707362174988,-2.2896013259887695],"argmax":2,"top2_margin":0.1026773452758789,"near_tie":false},"graph":{"p":[0.26031792163848877,0.1881757229566574,0.3621172308921814,0.18938903510570526],"max_abs_dp":0.0007414519786834717,"argmax_equal":true}}]},{"id":"tv4h_22","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":482,"_meta":{"row":13292,"text_sha256":"e4f765d62c6ed44999ddbaa502f1b865f19197d1e492af2abef30ac5dfd47750","row_sha256":"010a5fda50061ef619d06b589cf689715a9774f817996fb54d6d627b8499f882","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13292/permuted","group_id":"mmlu/test/13292","variant":"permuted","parent_id":"mmlu/test/13292"}},"state_tokens":30,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":94,"state_len":30,"decide":93,"opts":[55,66,81,92],"row_ids_sha256":"510ccc4bc5c352d20db36d3be41cb214b057980f421453556bf75bc2c586a2f3","oracle":{"probs":[0.2582307457923889,0.29809504747390747,0.23003582656383514,0.21363838016986847],"logits_raw":[-1.6219303607940674,-1.2765207290649414,-1.9001144170761108,-2.078042507171631],"argmax":1,"top2_margin":0.039864301681518555,"near_tie":false},"graph":{"p":[0.2593357563018799,0.2998739182949066,0.22956937551498413,0.21122096478939056],"max_abs_dp":0.0024174153804779053,"argmax_equal":true}}]},{"id":"tv4h_23","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":483,"_meta":{"row":588,"text_sha256":"6f26999eaee4ab138243c58aa5c0360817cb583c90caaac957438ec0b09f9d90","row_sha256":"010f9883d7f34928b7cf5b3fe2ecf5f7e662cf3cc29611c5790da5e6b28ef96a","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/588/none_present","group_id":"mmlu/test/588","variant":"none_present","parent_id":"mmlu/test/588","none_key":"none_of_these"}},"state_tokens":16,"questions":[{"q":0,"type":"choice","n_options":5,"row_len":77,"state_len":16,"decide":76,"opts":[33,43,52,66,75],"row_ids_sha256":"a2219354d42d1e7ea533c87012d3e1d1ddfdb831f3af214ce95a6224a24efed2","oracle":{"probs":[0.000971917062997818,0.9429154992103577,0.006515315268188715,0.04427322372794151,0.005323998164385557],"logits_raw":[-10.6028470993042,5.944669246673584,-6.024998664855957,-1.4144673347473145,-6.510855197906494],"argmax":1,"top2_margin":0.8986423015594482,"near_tie":false},"graph":{"p":[0.0009957918664440513,0.9424670338630676,0.006589335389435291,0.04457817226648331,0.005369630642235279],"max_abs_dp":0.00044846534729003906,"argmax_equal":true}}]},{"id":"tv4h_24","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":484,"_meta":{"row":588,"text_sha256":"6f26999eaee4ab138243c58aa5c0360817cb583c90caaac957438ec0b09f9d90","row_sha256":"010f9883d7f34928b7cf5b3fe2ecf5f7e662cf3cc29611c5790da5e6b28ef96a","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/588/none_absent","group_id":"mmlu/test/588","variant":"none_absent","parent_id":"mmlu/test/588","none_key":"none_of_these"}},"state_tokens":16,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":67,"state_len":16,"decide":66,"opts":[32,46,56,65],"row_ids_sha256":"6c45f6d6f555776cd196f2cfe89d04f016f80888b30b28efdf678e5714cfad8e","oracle":{"probs":[0.025834083557128906,0.9423138499259949,0.003542400663718581,0.02830960601568222],"logits_raw":[-8.089265823364258,0.564439058303833,-12.86982250213623,-7.869096279144287],"argmax":1,"top2_margin":0.9140042662620544,"near_tie":false},"graph":{"p":[0.026049287989735603,0.9418726563453674,0.003622700460255146,0.028455309569835663],"max_abs_dp":0.0004411935806274414,"argmax_equal":true}}]},{"id":"tv4h_25","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":485,"_meta":{"row":588,"text_sha256":"6f26999eaee4ab138243c58aa5c0360817cb583c90caaac957438ec0b09f9d90","row_sha256":"010f9883d7f34928b7cf5b3fe2ecf5f7e662cf3cc29611c5790da5e6b28ef96a","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/588/permuted","group_id":"mmlu/test/588","variant":"permuted","parent_id":"mmlu/test/588"}},"state_tokens":16,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":63,"state_len":16,"decide":62,"opts":[33,42,52,61],"row_ids_sha256":"8e08fe176fb8c488b1a0460e7d0270643ed8486413b86ee80517980edd365883","oracle":{"probs":[0.9816269278526306,0.008899662643671036,0.0029223980382084846,0.006550904363393784],"logits_raw":[5.149112701416016,-6.167016506195068,-8.846414566040039,-6.904254913330078],"argmax":0,"top2_margin":0.9727272391319275,"near_tie":false},"graph":{"p":[0.9813184142112732,0.009050602093338966,0.002972218906506896,0.006658829282969236],"max_abs_dp":0.0003085136413574219,"argmax_equal":true}}]},{"id":"tv4h_26","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":486,"_meta":{"row":4589,"text_sha256":"b21eb6a47835c694dbb4c55d0c6eaf127143103b5be6e901b3bc081a1875e03a","row_sha256":"0140ae7a6c63a4b72a607fe028843ded2c762ed70839dd64f1e26c4f29fd8813","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4589/none_present","group_id":"mmlu/test/4589","variant":"none_present","parent_id":"mmlu/test/4589","none_key":"none_of_these"}},"state_tokens":18,"questions":[{"q":0,"type":"choice","n_options":5,"row_len":110,"state_len":18,"decide":109,"opts":[39,53,67,88,108],"row_ids_sha256":"510b899fe0c090069e732364c8345bbf5f83c35c58c7d1638b5db2ce4b21d995","oracle":{"probs":[0.7710285782814026,0.1971021443605423,0.010083111003041267,0.00975391548126936,0.012032205238938332],"logits_raw":[2.4771676063537598,-0.8046927452087402,-7.957542419433594,-8.037406921386719,-7.532334327697754],"argmax":0,"top2_margin":0.5739264488220215,"near_tie":false},"graph":{"p":[0.7696431279182434,0.19812098145484924,0.010222558863461018,0.009860778227448463,0.012152659706771374],"max_abs_dp":0.0013854503631591797,"argmax_equal":true}}]},{"id":"tv4h_27","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":487,"_meta":{"row":4589,"text_sha256":"b21eb6a47835c694dbb4c55d0c6eaf127143103b5be6e901b3bc081a1875e03a","row_sha256":"0140ae7a6c63a4b72a607fe028843ded2c762ed70839dd64f1e26c4f29fd8813","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4589/none_absent","group_id":"mmlu/test/4589","variant":"none_absent","parent_id":"mmlu/test/4589","none_key":"none_of_these"}},"state_tokens":18,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":96,"state_len":18,"decide":95,"opts":[39,60,80,94],"row_ids_sha256":"f128e82c49b920af84a92ae872ec4f536fff7e619c144fd7728fb68b3f9cba3b","oracle":{"probs":[0.08604273200035095,0.05114022269845009,0.05304817855358124,0.8097688555717468],"logits_raw":[-5.932554244995117,-7.184356212615967,-7.096224784851074,-0.5384196043014526],"argmax":3,"top2_margin":0.7237261533737183,"near_tie":false},"graph":{"p":[0.08518121391534805,0.05151889845728874,0.053536634892225266,0.8097631931304932],"max_abs_dp":0.0008615180850028992,"argmax_equal":true}}]},{"id":"tv4h_28","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":488,"_meta":{"row":4589,"text_sha256":"b21eb6a47835c694dbb4c55d0c6eaf127143103b5be6e901b3bc081a1875e03a","row_sha256":"0140ae7a6c63a4b72a607fe028843ded2c762ed70839dd64f1e26c4f29fd8813","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4589/permuted","group_id":"mmlu/test/4589","variant":"permuted","parent_id":"mmlu/test/4589"}},"state_tokens":18,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":96,"state_len":18,"decide":95,"opts":[39,53,74,94],"row_ids_sha256":"d1353ef0e56096aef885705eebdd6c352a134fd8201d4b4c55a488b0ab26b3b1","oracle":{"probs":[0.9461724758148193,0.017222043126821518,0.01631193421781063,0.02029358223080635],"logits_raw":[2.4616715908050537,-7.177529335021973,-7.308160781860352,-6.782661437988281],"argmax":0,"top2_margin":0.9258788824081421,"near_tie":false},"graph":{"p":[0.945670485496521,0.017428243532776833,0.016417725011706352,0.020483531057834625],"max_abs_dp":0.0005019903182983398,"argmax_equal":true}}]},{"id":"tv4h_29","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":489,"_meta":{"row":6099,"text_sha256":"d11693badfd9d9aeb495b1bbd2b030536f3badf884e3c5c968350c222af0c1bf","row_sha256":"018fc4e2f2bc5d033d1203c9298ea91deb852be048cc2234325c3f764b24db98","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6099/none_present","group_id":"mmlu/test/6099","variant":"none_present","parent_id":"mmlu/test/6099","none_key":"none_of_these"}},"state_tokens":20,"questions":[{"q":0,"type":"choice","n_options":5,"row_len":89,"state_len":20,"decide":88,"opts":[39,52,61,73,87],"row_ids_sha256":"ac0623c83d30b1606232f98e3c16ed7f02d4c88868e2c2b5fcfd401c2172676c","oracle":{"probs":[0.37717702984809875,0.05788011848926544,0.03039967268705368,0.015718869864940643,0.5188242793083191],"logits_raw":[-0.7183589935302734,-5.228116035461426,-6.777472972869873,-8.364431381225586,0.04881155490875244],"argmax":4,"top2_margin":0.14164724946022034,"near_tie":false},"graph":{"p":[0.37413108348846436,0.05918622761964798,0.03063354641199112,0.015855351462960243,0.520193874835968],"max_abs_dp":0.0030459463596343994,"argmax_equal":true}}]},{"id":"tv4h_30","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":490,"_meta":{"row":6099,"text_sha256":"d11693badfd9d9aeb495b1bbd2b030536f3badf884e3c5c968350c222af0c1bf","row_sha256":"018fc4e2f2bc5d033d1203c9298ea91deb852be048cc2234325c3f764b24db98","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6099/none_absent","group_id":"mmlu/test/6099","variant":"none_absent","parent_id":"mmlu/test/6099","none_key":"none_of_these"}},"state_tokens":20,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":77,"state_len":20,"decide":76,"opts":[40,52,66,75],"row_ids_sha256":"cf0b750c88af581a6cf0fe0fbfb21925274c8d3435c46702ded7e628bb3251c8","oracle":{"probs":[0.15701055526733398,0.0459911935031414,0.7767869830131531,0.02021127939224243],"logits_raw":[-3.8808417320251465,-6.835141181945801,-0.03392109274864197,-8.813417434692383],"argmax":2,"top2_margin":0.6197764277458191,"near_tie":false},"graph":{"p":[0.15826265513896942,0.046458642929792404,0.7747957110404968,0.020482974126935005],"max_abs_dp":0.00199127197265625,"argmax_equal":true}}]},{"id":"tv4h_31","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":491,"_meta":{"row":6099,"text_sha256":"d11693badfd9d9aeb495b1bbd2b030536f3badf884e3c5c968350c222af0c1bf","row_sha256":"018fc4e2f2bc5d033d1203c9298ea91deb852be048cc2234325c3f764b24db98","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6099/permuted","group_id":"mmlu/test/6099","variant":"permuted","parent_id":"mmlu/test/6099"}},"state_tokens":20,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":75,"state_len":20,"decide":74,"opts":[39,52,61,73],"row_ids_sha256":"6705f1ec5a84aac94cb25786299381f4facbbb87e1e6a97234173522fed5df15","oracle":{"probs":[0.7859917879104614,0.11991894990205765,0.06216959282755852,0.0319196954369545],"logits_raw":[-0.633526623249054,-5.157214164733887,-6.737868309020996,-8.34184455871582],"argmax":0,"top2_margin":0.6660728454589844,"near_tie":false},"graph":{"p":[0.7817237973213196,0.12305353581905365,0.06289203464984894,0.03233053907752037],"max_abs_dp":0.004267990589141846,"argmax_equal":true}}]},{"id":"tv4h_32","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":492,"_meta":{"row":6139,"text_sha256":"67fe59443c44fef770b56204f7dd4ac9a42a3d1d492a4249ec7e742c7c97566f","row_sha256":"022dbb0545abb6ee6edcb5b28dbfdc7099785c2a5b106aa0536906349b11988d","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6139/none_present","group_id":"mmlu/test/6139","variant":"none_present","parent_id":"mmlu/test/6139","none_key":"none_of_these"}},"state_tokens":14,"questions":[{"q":0,"type":"choice","n_options":5,"row_len":80,"state_len":14,"decide":79,"opts":[34,44,54,64,78],"row_ids_sha256":"b97c178760bb057d05bf2e4d353d2e75d1d004f7d2bf1a8a7022c0ef8be4f81d","oracle":{"probs":[0.8921193480491638,0.006419079378247261,0.007241999730467796,0.0022946768440306187,0.0919247567653656],"logits_raw":[4.557070732116699,-7.315162658691406,-7.024938583374023,-9.790223121643066,-0.9109893441200256],"argmax":0,"top2_margin":0.8001946210861206,"near_tie":false},"graph":{"p":[0.8906072974205017,0.006508223246783018,0.007350253872573376,0.0023425850085914135,0.09319155663251877],"max_abs_dp":0.0015120506286621094,"argmax_equal":true}}]},{"id":"tv4h_33","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":493,"_meta":{"row":6139,"text_sha256":"67fe59443c44fef770b56204f7dd4ac9a42a3d1d492a4249ec7e742c7c97566f","row_sha256":"022dbb0545abb6ee6edcb5b28dbfdc7099785c2a5b106aa0536906349b11988d","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6139/none_absent","group_id":"mmlu/test/6139","variant":"none_absent","parent_id":"mmlu/test/6139","none_key":"none_of_these"}},"state_tokens":14,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":67,"state_len":14,"decide":66,"opts":[31,41,51,65],"row_ids_sha256":"b0a8c88817781b16d26ac5ea544dd2009f64451576b71398cda7961b3960d5ad","oracle":{"probs":[0.017466573044657707,0.034380074590444565,0.029218384996056557,0.9189349412918091],"logits_raw":[-9.7601900100708,-8.130841255187988,-8.522254943847656,-0.2251906394958496],"argmax":3,"top2_margin":0.8845548629760742,"near_tie":false},"graph":{"p":[0.017713723704218864,0.03509470820426941,0.02965562790632248,0.9175359010696411],"max_abs_dp":0.0013990402221679688,"argmax_equal":true}}]},{"id":"tv4h_34","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":494,"_meta":{"row":6139,"text_sha256":"67fe59443c44fef770b56204f7dd4ac9a42a3d1d492a4249ec7e742c7c97566f","row_sha256":"022dbb0545abb6ee6edcb5b28dbfdc7099785c2a5b106aa0536906349b11988d","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6139/permuted","group_id":"mmlu/test/6139","variant":"permuted","parent_id":"mmlu/test/6139"}},"state_tokens":14,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":66,"state_len":14,"decide":65,"opts":[34,44,54,64],"row_ids_sha256":"d7c2a308e17ad244dd7f665fdfbe3cc4dcaa1620cb2195ce2301b228a3e89a7e","oracle":{"probs":[0.9828706979751587,0.006867693271487951,0.007817881181836128,0.002443697303533554],"logits_raw":[4.617333889007568,-7.325454235076904,-7.013665199279785,-9.811664581298828],"argmax":0,"top2_margin":0.9750528335571289,"near_tie":false},"graph":{"p":[0.9825628995895386,0.0069816079922020435,0.007953842170536518,0.002501638373360038],"max_abs_dp":0.0003077983856201172,"argmax_equal":true}}]},{"id":"tv4h_35","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":495,"_meta":{"row":13810,"text_sha256":"d6ab903bb64d97ef4252906a9c35b394c2b135cc82173b9a98a562ae4941e714","row_sha256":"03bd43e6287a70dfe5cc0e5fe2bf7bfe2cdf49e4a895d91a13fdde9bbb4ff5d9","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13810/none_present","group_id":"mmlu/test/13810","variant":"none_present","parent_id":"mmlu/test/13810","none_key":"none_of_these"}},"state_tokens":24,"questions":[{"q":0,"type":"choice","n_options":5,"row_len":95,"state_len":24,"decide":94,"opts":[48,58,67,81,93],"row_ids_sha256":"d6e64d3083c7b5005a52332798d25b320caae3c6e90437a19250ed8a7469a069","oracle":{"probs":[0.0038243632297962904,0.01246864628046751,0.06401437520980835,0.16853144764900208,0.7511610984802246],"logits_raw":[-8.704784393310547,-5.8612542152404785,-1.9252197742462158,0.4038718342781067,3.9997081756591797],"argmax":4,"top2_margin":0.5826296806335449,"near_tie":false},"graph":{"p":[0.0038542102556675673,0.012653816491365433,0.06386478990316391,0.1680692583322525,0.7515579462051392],"max_abs_dp":0.00046218931674957275,"argmax_equal":true}}]},{"id":"tv4h_36","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":496,"_meta":{"row":13810,"text_sha256":"d6ab903bb64d97ef4252906a9c35b394c2b135cc82173b9a98a562ae4941e714","row_sha256":"03bd43e6287a70dfe5cc0e5fe2bf7bfe2cdf49e4a895d91a13fdde9bbb4ff5d9","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13810/none_absent","group_id":"mmlu/test/13810","variant":"none_absent","parent_id":"mmlu/test/13810","none_key":"none_of_these"}},"state_tokens":24,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":86,"state_len":24,"decide":85,"opts":[41,58,72,84],"row_ids_sha256":"833401362d1905fb95b5f7fd315dfc23c754b8e7015246477d858ef500d4b30d","oracle":{"probs":[0.02284201793372631,0.006753668654710054,0.2085867077112198,0.7618175745010376],"logits_raw":[-4.7897539138793945,-7.721564292907715,0.5318347811698914,3.64851713180542],"argmax":3,"top2_margin":0.553230881690979,"near_tie":false},"graph":{"p":[0.023109644651412964,0.006752655375748873,0.2086181640625,0.7615194916725159],"max_abs_dp":0.0002980828285217285,"argmax_equal":true}}]},{"id":"tv4h_37","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":497,"_meta":{"row":13810,"text_sha256":"d6ab903bb64d97ef4252906a9c35b394c2b135cc82173b9a98a562ae4941e714","row_sha256":"03bd43e6287a70dfe5cc0e5fe2bf7bfe2cdf49e4a895d91a13fdde9bbb4ff5d9","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13810/permuted","group_id":"mmlu/test/13810","variant":"permuted","parent_id":"mmlu/test/13810"}},"state_tokens":24,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":81,"state_len":24,"decide":80,"opts":[48,57,67,79],"row_ids_sha256":"27c2519fd47a6803794ecb4db00775fe05d79a6e04d09e3ba0f7a80284ef9ef3","oracle":{"probs":[0.005203627049922943,0.13635699450969696,0.014195338822901249,0.8442439436912537],"logits_raw":[-8.672183990478516,-0.8142155408859253,-6.25757360458374,3.5724105834960938],"argmax":3,"top2_margin":0.7078869342803955,"near_tie":false},"graph":{"p":[0.005249860696494579,0.1359417587518692,0.014429774135351181,0.844378650188446],"max_abs_dp":0.0004152357578277588,"argmax_equal":true}}]},{"id":"tv4h_38","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":498,"_meta":{"row":12434,"text_sha256":"6415833a28d2dcfe5791e61f69f47411f55fa901fdbcb63f0f2a39b718ad434b","row_sha256":"04184bb438912221313ece06f12d94cc5b9eb0500c9a978a14d7e05f5fcbbedf","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12434/none_present","group_id":"mmlu/test/12434","variant":"none_present","parent_id":"mmlu/test/12434","none_key":"none_of_these"}},"state_tokens":194,"questions":[{"q":0,"type":"choice","n_options":5,"row_len":251,"state_len":194,"decide":250,"opts":[209,216,230,238,249],"row_ids_sha256":"d973a468e9621c7b87d7160cf941c129721c24a6d22ea1ed1c2d2ccf659a79db","oracle":{"probs":[0.008912339806556702,0.43816065788269043,0.46972036361694336,0.07187951356172562,0.011327093467116356],"logits_raw":[-9.454204559326172,-0.08228227496147156,0.08506307005882263,-4.4314446449279785,-8.877328872680664],"argmax":2,"top2_margin":0.03155970573425293,"near_tie":false},"graph":{"p":[0.008930154144763947,0.4381140470504761,0.4708839952945709,0.07062732428312302,0.01144446898251772],"max_abs_dp":0.0012521892786026,"argmax_equal":true}}]},{"id":"tv4h_39","source":"transfer_v4_dev_head_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":499,"_meta":{"row":12434,"text_sha256":"6415833a28d2dcfe5791e61f69f47411f55fa901fdbcb63f0f2a39b718ad434b","row_sha256":"04184bb438912221313ece06f12d94cc5b9eb0500c9a978a14d7e05f5fcbbedf","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12434/none_absent","group_id":"mmlu/test/12434","variant":"none_absent","parent_id":"mmlu/test/12434","none_key":"none_of_these"}},"state_tokens":194,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":244,"state_len":194,"decide":243,"opts":[215,223,231,242],"row_ids_sha256":"37c8ae00ca7a425ae240cc15fe29db455aedd15a02a4152781fde0ed80a2b806","oracle":{"probs":[0.8717966675758362,0.09659945219755173,0.01973087526857853,0.011872980743646622],"logits_raw":[-0.04016801714897156,-5.333436965942383,-9.155179023742676,-10.37725830078125],"argmax":0,"top2_margin":0.7751972079277039,"near_tie":false},"graph":{"p":[0.872444748878479,0.09579639881849289,0.019769929349422455,0.011988932266831398],"max_abs_dp":0.0008030533790588379,"argmax_equal":true}}]},{"id":"tv4xh_emotion_00","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":100,"_meta":{"row":688,"text_sha256":"4066949ec1b6e92f58345cb5fd30ddf7893f1aba241fdb6a6e1cbec273a084af","row_sha256":"0e67443af7734973a4fa055e05dfcc7d5df45ea2e63aebc622f0471ed7ee2e56","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/688","group_id":"emotion/test/688","variant":"clean"}},"state_tokens":8,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":38,"state_len":8,"decide":37,"opts":[19,22,25,28,32,36],"row_ids_sha256":"91a4d89d649301b39afab12ba190e89bf93414673f311e8f7aa2379c19a62eb9","oracle":{"probs":[0.5085260272026062,0.0018038589041680098,0.001034665503539145,0.42738568782806396,0.04593098163604736,0.015318728983402252],"logits_raw":[-0.14802521467208862,-13.721967697143555,-15.059370994567871,-0.566267728805542,-5.933074951171875,-8.575071334838867],"argmax":0,"top2_margin":0.08114033937454224,"near_tie":false},"graph":{"p":[0.5093666911125183,0.0018207560060545802,0.0010503723751753569,0.42610299587249756,0.04623694345355034,0.015422253869473934],"max_abs_dp":0.0012826919555664062,"argmax_equal":true}}]},{"id":"tv4xh_emotion_01","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":101,"_meta":{"row":1156,"text_sha256":"5c94328e0abcde5f76e9ececdcf831bdfcca8340bde56410a2ee6ea63ede8efa","row_sha256":"0eb690bd9b84ae63ee53771271da390f4f3568f09398ef81aaf16e75aa955ec4","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1156","group_id":"emotion/test/1156","variant":"clean"}},"state_tokens":38,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":68,"state_len":38,"decide":67,"opts":[49,52,55,58,62,66],"row_ids_sha256":"82c15132db56d48fb8278a8e0cb113b17887329fb25595c24bb1ca518476e9df","oracle":{"probs":[0.0013593222247436643,0.7950971722602844,0.016568588092923164,0.0015821980778127909,0.003986588679254055,0.1814061999320984],"logits_raw":[-12.159557342529297,3.1705362796783447,-6.143176555633545,-11.79425048828125,-9.570770263671875,-0.38494545221328735],"argmax":1,"top2_margin":0.613690972328186,"near_tie":false},"graph":{"p":[0.0013625134015455842,0.7955824136734009,0.016516100615262985,0.0015990566462278366,0.003982404246926308,0.18095755577087402],"max_abs_dp":0.0004852414131164551,"argmax_equal":true}}]},{"id":"tv4xh_emotion_02","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":102,"_meta":{"row":1762,"text_sha256":"ad39fc80381cfdb8445d4823ad85ad62a450b9aa9e3e6c2fac218c5af8bbcdff","row_sha256":"0ed670b67b6b443b13254f6f3ceba1594a40b3d5efa018cb847a1bf4cc09de92","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1762","group_id":"emotion/test/1762","variant":"clean"}},"state_tokens":23,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":53,"state_len":23,"decide":52,"opts":[34,37,40,43,47,51],"row_ids_sha256":"39687f0934a0e3785bbbc30d01500167b6b75c38913e7018f7175a4d9e954fe2","oracle":{"probs":[0.0870099738240242,0.09608414769172668,0.38784700632095337,0.016179822385311127,0.37590187788009644,0.036977097392082214],"logits_raw":[-4.160104274749756,-3.9214203357696533,-0.5640504956245422,-8.207700729370117,-0.639318585395813,-6.219019412994385],"argmax":2,"top2_margin":0.011945128440856934,"near_tie":true},"graph":{"p":[0.08653748035430908,0.09677823632955551,0.3890906572341919,0.016074268147349358,0.37463128566741943,0.03688807040452957],"max_abs_dp":0.001270592212677002,"argmax_equal":true}}]},{"id":"tv4xh_emotion_03","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":103,"_meta":{"row":485,"text_sha256":"fa4cd7567c7ddff76ea7698ba5b6216ea8436250b6bbd3dc121b9ac95e96339a","row_sha256":"0f5c1699eaf05166edeb880862d73f5e69b9b19355324a0191a5aee53c028243","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/485","group_id":"emotion/test/485","variant":"clean"}},"state_tokens":19,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":49,"state_len":19,"decide":48,"opts":[30,33,36,39,43,47],"row_ids_sha256":"c8dbef3d019173294a43b591cec2e2228f360aed19d8955b0d70eb9ca8eb05d0","oracle":{"probs":[0.01655125431716442,0.28051453828811646,0.6239371299743652,0.021700970828533173,0.02811867743730545,0.029177462682127953],"logits_raw":[-9.446250915527344,-2.6367359161376953,-0.7132816314697266,-8.794464111328125,-8.171112060546875,-8.082178115844727],"argmax":2,"top2_margin":0.3434225916862488,"near_tie":false},"graph":{"p":[0.01667707785964012,0.28317660093307495,0.620174765586853,0.021976936608552933,0.02852446585893631,0.029470134526491165],"max_abs_dp":0.003762364387512207,"argmax_equal":true}}]},{"id":"tv4xh_emotion_04","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":104,"_meta":{"row":482,"text_sha256":"72cedf6ff6119d3cdfd596952a47db17640ed14c4e7bc994ef8e6f0524f3e54f","row_sha256":"0f8ea79039cfe96a6aafe20683ec52f9058868b3fbc01fef4fc0f1993302d2b6","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/482","group_id":"emotion/test/482","variant":"clean"}},"state_tokens":12,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":42,"state_len":12,"decide":41,"opts":[23,26,29,32,36,40],"row_ids_sha256":"6c2b1a101f79930555acbe63b91bcb97021f61482e95d7cab6b238b2a46592d1","oracle":{"probs":[0.31804049015045166,0.10317728668451309,0.05449162423610687,0.1416085958480835,0.12308572232723236,0.2595963180065155],"logits_raw":[-5.237696170806885,-7.946258544921875,-9.482285499572754,-7.184459686279297,-7.521753787994385,-5.7262468338012695],"argmax":0,"top2_margin":0.05844417214393616,"near_tie":false},"graph":{"p":[0.3179806172847748,0.10273018479347229,0.0545700378715992,0.14199510216712952,0.1234908476471901,0.2592332363128662],"max_abs_dp":0.000447101891040802,"argmax_equal":true}}]},{"id":"tv4xh_emotion_05","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":105,"_meta":{"row":959,"text_sha256":"24c2a3b49262128f8e7f4493cf92e2df90ac37301bef8868e23a0655b278617e","row_sha256":"0f9c21d82eb205905d920e2ebee7fd94a0a69419bf1fbfcc21411261819828c2","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/959","group_id":"emotion/test/959","variant":"clean"}},"state_tokens":7,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":37,"state_len":7,"decide":36,"opts":[18,21,24,27,31,35],"row_ids_sha256":"3a7855f429959099e89a47c7409a033ce997fda0b97f1c9338947f0a73f46290","oracle":{"probs":[0.003014862071722746,0.5557619333267212,0.4135562777519226,0.0022964468225836754,0.0034126671962440014,0.021957872435450554],"logits_raw":[-12.127875328063965,0.4239720106124878,-0.2871275246143341,-12.782780647277832,-11.829669952392578,-7.350490570068359],"argmax":1,"top2_margin":0.14220565557479858,"near_tie":false},"graph":{"p":[0.00306810368783772,0.555546224117279,0.41354238986968994,0.0023339008912444115,0.0034685200080275536,0.022040801122784615],"max_abs_dp":0.00021570920944213867,"argmax_equal":true}}]},{"id":"tv4xh_emotion_06","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":106,"_meta":{"row":573,"text_sha256":"40b80bafb098041e5616de300eb5307603e3c1d76ffcb2a93a8d1f335e72b0b3","row_sha256":"0fd082dba48e728637716ff7ccbde925e36a81b67e911fe071cbcd9c163c9601","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/573","group_id":"emotion/test/573","variant":"clean"}},"state_tokens":13,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":43,"state_len":13,"decide":42,"opts":[24,27,30,33,37,41],"row_ids_sha256":"773e717b5e971b1248ccbae0fa09b292ecc2c1544f8e31586ce33d4c0b9054da","oracle":{"probs":[0.9329598546028137,0.0013065768871456385,0.001540723955258727,0.014746990986168385,0.016231199726462364,0.033214639872312546],"logits_raw":[3.0498557090759277,-12.760181427001953,-12.363565444946289,-6.928811073303223,-6.6980791091918945,-4.97521448135376],"argmax":0,"top2_margin":0.8997452259063721,"near_tie":false},"graph":{"p":[0.9325535297393799,0.0013172473991289735,0.0015509974909946322,0.014699457213282585,0.016301950439810753,0.033576760441064835],"max_abs_dp":0.0004063248634338379,"argmax_equal":true}}]},{"id":"tv4xh_emotion_07","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":107,"_meta":{"row":588,"text_sha256":"567395b877b01a51dda43199e8133c4ca29b4c2aec64fe320ea22003806b2e07","row_sha256":"0fe855203820afc20abc794e92cf670acd215f362b31c7cd545a6f6f318bb48e","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/588","group_id":"emotion/test/588","variant":"clean"}},"state_tokens":34,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":64,"state_len":34,"decide":63,"opts":[45,48,51,54,58,62],"row_ids_sha256":"ad37d9c23711f2a527bafb878ff6b1dab881753691b1ebf00c159f230efda122","oracle":{"probs":[0.3895173966884613,0.2042623609304428,0.1390550583600998,0.07947912067174911,0.11499961465597153,0.07268646359443665],"logits_raw":[-4.925124645233154,-6.478237628936768,-7.403449058532715,-8.749334335327148,-7.860456466674805,-8.964288711547852],"argmax":0,"top2_margin":0.1852550357580185,"near_tie":false},"graph":{"p":[0.39104703068733215,0.2032199651002884,0.13880321383476257,0.07958526909351349,0.11488436907529831,0.07246013730764389],"max_abs_dp":0.0015296339988708496,"argmax_equal":true}}]},{"id":"tv4xh_emotion_08","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":108,"_meta":{"row":1176,"text_sha256":"6ae2fe2353747ab43c1fb4639d95c826246c76ce3f9cb7d99c0305f94c6ed03c","row_sha256":"109d5f2888e3918274d98ae9f43e42980b477fcd20df8778fec8b0c450f1426d","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/1176","group_id":"emotion/test/1176","variant":"clean"}},"state_tokens":17,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":47,"state_len":17,"decide":46,"opts":[28,31,34,37,41,45],"row_ids_sha256":"5fcd9041ccfe073c5bd20ada874f2f5a60f0a1053abafa969075d211c0a60d89","oracle":{"probs":[0.003110571764409542,0.8625690937042236,0.024452341720461845,0.0060880668461322784,0.007011690642684698,0.09676820039749146],"logits_raw":[-12.397394180297852,1.1368980407714844,-7.436313629150391,-10.781673431396484,-10.44182300567627,-4.1265692710876465],"argmax":1,"top2_margin":0.7658008933067322,"near_tie":false},"graph":{"p":[0.003159084590151906,0.8609246015548706,0.02450433559715748,0.0061704316176474094,0.007127486169338226,0.09811406582593918],"max_abs_dp":0.0016444921493530273,"argmax_equal":true}}]},{"id":"tv4xh_emotion_09","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":109,"_meta":{"row":851,"text_sha256":"65d13295c3844d56ae0ad485e08f040ef338d406397893d51e5028943de7b6d7","row_sha256":"128280c0bbce14f63d49c543a4bd6585e6ac4f51b940f1ed10bd09ebc8c23b78","source":"emotion","repo":"dair-ai/emotion","revision":"cab853a1dbdf4c42c2b3ef2173804746df8825fe","split":"test","id":"emotion/test/851","group_id":"emotion/test/851","variant":"clean"}},"state_tokens":30,"questions":[{"q":0,"type":"choice","n_options":6,"row_len":60,"state_len":30,"decide":59,"opts":[41,44,47,50,54,58],"row_ids_sha256":"8e2fbb80b349e66b88c3e0ba84f9c3af4b535c63ce4cc1cb5b1f6cd01a7a4be6","oracle":{"probs":[0.7585453391075134,0.0082743214443326,0.005031976383179426,0.05150233209133148,0.10616350173950195,0.07048249989748001],"logits_raw":[-2.9437851905822754,-13.814909934997559,-15.011545181274414,-9.415519714355469,-7.675095081329346,-8.660651206970215],"argmax":0,"top2_margin":0.6523818373680115,"near_tie":false},"graph":{"p":[0.7581611275672913,0.008388466201722622,0.005132024642080069,0.051592037081718445,0.1058482676744461,0.0708780586719513],"max_abs_dp":0.00039555877447128296,"argmax_equal":true}}]},{"id":"tv4xh_tweet_offensive_00","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":180,"_meta":{"row":313,"text_sha256":"4204a4e5d11e8f6344574407c8d7d5ed4b9014b2773f4419288e7cf547416b6d","row_sha256":"0998a66e893bc5d8d83578fdfdd86b3c4e6f50071cc57339f3246fb9cfc78fe9","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/313","group_id":"tweet_offensive/test/313","variant":"clean"}},"state_tokens":51,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":82,"state_len":51,"decide":81,"opts":[62,80],"row_ids_sha256":"a90a90c295894ba76718003efa7ac30eef910ebf5302ddc5a15048c72ec933a9","oracle":{"probs":[0.9416719675064087,0.05832802876830101],"logits_raw":[-0.41902226209640503,-7.111629009246826],"argmax":0,"top2_margin":0.8833439350128174,"near_tie":false},"graph":{"p":[0.9415555000305176,0.058444488793611526],"max_abs_dp":0.00011646747589111328,"argmax_equal":true}}]},{"id":"tv4xh_tweet_offensive_01","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":181,"_meta":{"row":485,"text_sha256":"e629cdda69ff4d6a8cca379afb11cbf43cbe21a10a23943534acf4c372d6b5b0","row_sha256":"09bf5b835ab7df9c3e1fa8198199b2d8bb5d5aaf417b81e252d4dc43d76ca2c3","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/485","group_id":"tweet_offensive/test/485","variant":"clean"}},"state_tokens":37,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":68,"state_len":37,"decide":67,"opts":[48,66],"row_ids_sha256":"623f5e1af240294869866c964d969cfa84c6acdf2eb690a852c8682a383c49c7","oracle":{"probs":[0.5822824835777283,0.41771751642227173],"logits_raw":[-0.9034494161605835,-1.7026196718215942],"argmax":0,"top2_margin":0.16456496715545654,"near_tie":false},"graph":{"p":[0.5817045569419861,0.41829538345336914],"max_abs_dp":0.0005779266357421875,"argmax_equal":true}}]},{"id":"tv4xh_tweet_offensive_02","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":182,"_meta":{"row":48,"text_sha256":"801538222ea893f2f5fe0895a7cb28bb553409b93e1d4a86503b549fc6dea902","row_sha256":"09c17252edb29aa32749490427d39b1b36949e84f29a9a6a18b45d6f96bdab2b","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/48","group_id":"tweet_offensive/test/48","variant":"clean"}},"state_tokens":67,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":98,"state_len":67,"decide":97,"opts":[78,96],"row_ids_sha256":"1960fcbb602e90b80024e46fa5588fcace47adaff955b9fefb7ab680a73f8313","oracle":{"probs":[0.6563602089881897,0.3436397612094879],"logits_raw":[-1.9174383878707886,-3.4744315147399902],"argmax":0,"top2_margin":0.3127204477787018,"near_tie":false},"graph":{"p":[0.6559379696846008,0.34406203031539917],"max_abs_dp":0.0004222691059112549,"argmax_equal":true}}]},{"id":"tv4xh_tweet_offensive_03","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":183,"_meta":{"row":253,"text_sha256":"8852cfb2c98d2e4af8ffe17c897f6abe841ef18b58d139cfda231fac2e944f00","row_sha256":"0a0505b1788303b977c44d6fba918c41a7b10dc665995a2666b10aede2767d07","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/253","group_id":"tweet_offensive/test/253","variant":"clean"}},"state_tokens":53,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":84,"state_len":53,"decide":83,"opts":[64,82],"row_ids_sha256":"88b0d922ebd92b2613cf5e5b52f55f2e57da3917eecee61e7d9007c35ef5b052","oracle":{"probs":[0.48382195830345154,0.5161780118942261],"logits_raw":[-1.652477502822876,-1.49672269821167],"argmax":1,"top2_margin":0.032356053590774536,"near_tie":false},"graph":{"p":[0.48272645473480225,0.517273485660553],"max_abs_dp":0.001095503568649292,"argmax_equal":true}}]},{"id":"tv4xh_tweet_offensive_04","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":184,"_meta":{"row":376,"text_sha256":"8a32a5d72bafa33d70245fbebfe2de1b2567720cbba83e6ada59790b005f37ae","row_sha256":"0ad947d4e8cd30997a02e9257ac3bc097009f7cd4c9936c9ab017380c45e691d","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/376","group_id":"tweet_offensive/test/376","variant":"clean"}},"state_tokens":69,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":100,"state_len":69,"decide":99,"opts":[80,98],"row_ids_sha256":"64b6f4f6f01f70919aaaebc19c8f5bd4df724378d718f61ba1a915f77dfac8b0","oracle":{"probs":[0.4307321310043335,0.5692678689956665],"logits_raw":[-1.4674897193908691,-0.7965273857116699],"argmax":1,"top2_margin":0.138535737991333,"near_tie":false},"graph":{"p":[0.4326910972595215,0.5673089623451233],"max_abs_dp":0.0019589662551879883,"argmax_equal":true}}]},{"id":"tv4xh_tweet_offensive_05","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":185,"_meta":{"row":27,"text_sha256":"0cfb8a1d394ba1b15f499eac3eb06a69405c6d5604d51d24adc596dabdb61cc0","row_sha256":"0aedc5785019553d108d9c9792a9bd68893723d7677efa9f75dd1570bc6a4e20","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/27","group_id":"tweet_offensive/test/27","variant":"clean"}},"state_tokens":26,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":57,"state_len":26,"decide":56,"opts":[37,55],"row_ids_sha256":"479a06c04c8fb5a3dbfa882ca9caf64b9bdfe464bbfc6ee0bddb8478160b9a34","oracle":{"probs":[0.3505955934524536,0.6494044661521912],"logits_raw":[-1.2582781314849854,0.22486472129821777],"argmax":1,"top2_margin":0.29880887269973755,"near_tie":false},"graph":{"p":[0.35100987553596497,0.6489901542663574],"max_abs_dp":0.00041431188583374023,"argmax_equal":true}}]},{"id":"tv4xh_tweet_offensive_06","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":186,"_meta":{"row":283,"text_sha256":"04b1555a891521a455627758f5ebe0ca4c0cab351ed23652c3f949b246861424","row_sha256":"0b8fead701a10c9a25d7fed7dda6b968878feeb0c1514b9abc8f08d43c0154bc","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/283","group_id":"tweet_offensive/test/283","variant":"clean"}},"state_tokens":33,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":64,"state_len":33,"decide":63,"opts":[44,62],"row_ids_sha256":"7dcf4f664532339588849915636d9d8bea71a4ac32b630fb8a799413c2f5d2f8","oracle":{"probs":[0.6833937764167786,0.31660619378089905],"logits_raw":[-1.618958830833435,-3.4702038764953613],"argmax":0,"top2_margin":0.3667875826358795,"near_tie":false},"graph":{"p":[0.6831274628639221,0.3168725073337555],"max_abs_dp":0.0002663135528564453,"argmax_equal":true}}]},{"id":"tv4xh_tweet_offensive_07","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":187,"_meta":{"row":606,"text_sha256":"167a4237bdfbb0655f6edbeb93b448bfe79546711b0323a3ea1ca3e69f6f88bd","row_sha256":"0bce5a48f00f701e06ae3cdec7ed386930771ca79295e9aa94c4eacd6b992c50","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/606","group_id":"tweet_offensive/test/606","variant":"clean"}},"state_tokens":3,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":34,"state_len":3,"decide":33,"opts":[14,32],"row_ids_sha256":"7faf26b22fd4f168637acd77fbbfc522a6fca6a71e7983df9905d008200aca93","oracle":{"probs":[0.7425146102905273,0.25748535990715027],"logits_raw":[-1.0496981143951416,-3.5978965759277344],"argmax":0,"top2_margin":0.4850292503833771,"near_tie":false},"graph":{"p":[0.742087185382843,0.257912814617157],"max_abs_dp":0.00042745471000671387,"argmax_equal":true}}]},{"id":"tv4xh_tweet_offensive_08","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":188,"_meta":{"row":846,"text_sha256":"5f9d68b6a8d361748906cbf563c5074e4a44d79aef7436403ebdb53de685cff6","row_sha256":"0d0b7fb0772b88fb401ed1f014cf53d53fc50ae3331d3add9299bf301d45b548","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/846","group_id":"tweet_offensive/test/846","variant":"clean"}},"state_tokens":24,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":55,"state_len":24,"decide":54,"opts":[35,53],"row_ids_sha256":"58f1fa2894f01707fafe1a4504776309aed8eb138ef8dc1051e9d80298663daa","oracle":{"probs":[0.42474254965782166,0.5752574801445007],"logits_raw":[-2.0755884647369385,-1.345750331878662],"argmax":1,"top2_margin":0.15051493048667908,"near_tie":false},"graph":{"p":[0.42473065853118896,0.575269341468811],"max_abs_dp":1.189112663269043e-05,"argmax_equal":true}}]},{"id":"tv4xh_tweet_offensive_09","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":189,"_meta":{"row":653,"text_sha256":"707e8382da9966c04b4aba20589dae9963dfcc7edfe85303e7312282c5d8a5bc","row_sha256":"0d9ba78e56d04e64491bda789eae4ce795a9c8c59e912dbe0a37d94ff4e1d539","source":"tweet_offensive","repo":"cardiffnlp/tweet_eval","revision":"b3a375baf0f409c77e6bc7aa35102b7b3534f8be","split":"test","id":"tweet_offensive/test/653","group_id":"tweet_offensive/test/653","variant":"clean"}},"state_tokens":43,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":74,"state_len":43,"decide":73,"opts":[54,72],"row_ids_sha256":"6d560c13ab47b0e9c2c13ce76828cf3e802c3988c40407778a873ab960b4088d","oracle":{"probs":[0.4506243169307709,0.5493756532669067],"logits_raw":[-1.3256696462631226,-0.848914623260498],"argmax":1,"top2_margin":0.09875133633613586,"near_tie":false},"graph":{"p":[0.4518328011035919,0.5481671690940857],"max_abs_dp":0.001208484172821045,"argmax_equal":true}}]},{"id":"tv4xh_qnli_00","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":260,"_meta":{"row":2378,"text_sha256":"c6fe821d0c7c11b37b5f21a0c4954effcab910883eb1dbaa16c1ca8313ff491c","row_sha256":"06855f8415c6e0bfc6c726dde4f27b4bf77d91f43ced854a9aa1bbfd9bc64722","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/2378","group_id":"qnli/validation/2378","variant":"clean"}},"state_tokens":38,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":68,"state_len":38,"decide":67,"opts":[63,66],"row_ids_sha256":"99df0c06f0f9425e04d1fca380201d6e48f63c17ccd66b7df91cf923110fd97e","oracle":{"probs":[0.9952312111854553,0.004768781829625368],"logits_raw":[3.435715675354004,-9.414718627929688],"argmax":0,"top2_margin":0.9904624223709106,"near_tie":false},"graph":{"p":[0.9951614737510681,0.0048385607078671455],"max_abs_dp":6.977887824177742e-05,"argmax_equal":true}}]},{"id":"tv4xh_qnli_01","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":261,"_meta":{"row":3361,"text_sha256":"512d1eded0f043e83e2e66c0ed6900d213324728e1babc58b65849537f9718cd","row_sha256":"06c348434d5500b8032a2275f91cb9f1556d654a743f2446e97068a47f1238e5","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/3361","group_id":"qnli/validation/3361","variant":"clean"}},"state_tokens":28,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":59,"state_len":28,"decide":58,"opts":[54,57],"row_ids_sha256":"288962626037c9489d0dc39b039a0e1851491bad5b37f0c2316ceadbb5e2f50d","oracle":{"probs":[0.8558077812194824,0.14419223368167877],"logits_raw":[-0.46772128343582153,-4.752652168273926],"argmax":0,"top2_margin":0.7116155624389648,"near_tie":false},"graph":{"p":[0.8544808030128479,0.1455192118883133],"max_abs_dp":0.0013269782066345215,"argmax_equal":true}}]},{"id":"tv4xh_qnli_02","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":262,"_meta":{"row":407,"text_sha256":"fb2b941cbe4a24159c78a58b85908a39cb2999038a1dfd40d1e94f0af7cd07ad","row_sha256":"0759615d0bc41bd3973690f7d0dd7be2404699c5e316d67dc6e945a499b0a49f","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/407","group_id":"qnli/validation/407","variant":"clean"}},"state_tokens":25,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":58,"state_len":25,"decide":57,"opts":[53,56],"row_ids_sha256":"9e1624adcd32d912612142b93eb8e2d7b52d4d4843efff820067c423c7763ce7","oracle":{"probs":[0.09892620891332626,0.9010738134384155],"logits_raw":[-3.0568366050720215,2.2586402893066406],"argmax":1,"top2_margin":0.802147626876831,"near_tie":false},"graph":{"p":[0.10026466846466064,0.8997353911399841],"max_abs_dp":0.001338459551334381,"argmax_equal":true}}]},{"id":"tv4xh_qnli_03","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":263,"_meta":{"row":4731,"text_sha256":"1a6c7b51a51440bfb18df4d6f1776a43fc8f6e9dda43b9a58621967409c573bd","row_sha256":"08047603cb2b88d97bed089e0d47c928ac3e6d4d33a4d901dec8888a04dea1f9","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/4731","group_id":"qnli/validation/4731","variant":"clean"}},"state_tokens":35,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":65,"state_len":35,"decide":64,"opts":[60,63],"row_ids_sha256":"b96d3787f8a8f645e43340bfdb962022c1d805e562dfd721764f869a47fca0ff","oracle":{"probs":[0.06282120943069458,0.9371788501739502],"logits_raw":[-3.471147298812866,3.031398296356201],"argmax":1,"top2_margin":0.8743576407432556,"near_tie":false},"graph":{"p":[0.06205189600586891,0.9379480481147766],"max_abs_dp":0.0007693134248256683,"argmax_equal":true}}]},{"id":"tv4xh_qnli_04","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":264,"_meta":{"row":946,"text_sha256":"faabefe33b508352d8d589e298d323b2eb91c9ec9a4a45b18da5f2cdec71a911","row_sha256":"087b55e02c0d683adc4bc13b8a78cc17c1d364e14fbc406ce98772efa6b7db17","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/946","group_id":"qnli/validation/946","variant":"clean"}},"state_tokens":25,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":57,"state_len":25,"decide":56,"opts":[52,55],"row_ids_sha256":"78800ed336eaef84bef24270532cb75eff72328bfd821062ce1b13e56235247e","oracle":{"probs":[0.012402109801769257,0.9875978827476501],"logits_raw":[-5.340061187744141,5.192203521728516],"argmax":1,"top2_margin":0.9751957654953003,"near_tie":false},"graph":{"p":[0.012411726638674736,0.9875882267951965],"max_abs_dp":9.655952453613281e-06,"argmax_equal":true}}]},{"id":"tv4xh_qnli_05","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":265,"_meta":{"row":886,"text_sha256":"a2323b14a06d6655fa963b755d46c58d9dd666e2ff51cceea05537b44d09c4e7","row_sha256":"08d7c1a79b327fc0434770ddc94d5522c8f3a8f2b738f0092974a40ffc5c6dc5","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/886","group_id":"qnli/validation/886","variant":"clean"}},"state_tokens":37,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":69,"state_len":37,"decide":68,"opts":[64,67],"row_ids_sha256":"797d73f6b92f4c298f154af92dc34c22c0276cf14e8ffb5cfe021bcf8bcf3d02","oracle":{"probs":[0.9903318881988525,0.009668107144534588],"logits_raw":[3.5821847915649414,-7.555920600891113],"argmax":0,"top2_margin":0.9806637763977051,"near_tie":false},"graph":{"p":[0.9902538657188416,0.009746103547513485],"max_abs_dp":7.802248001098633e-05,"argmax_equal":true}}]},{"id":"tv4xh_qnli_06","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":266,"_meta":{"row":4136,"text_sha256":"e17111ca25b490ecb466646b5efc3cb6912875ca2143eba611fd180e3d1cf134","row_sha256":"08ee5449e51919b980722057d809f2a333e076f51977a606f9a8f48d65a5f36c","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/4136","group_id":"qnli/validation/4136","variant":"clean"}},"state_tokens":16,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":48,"state_len":16,"decide":47,"opts":[43,46],"row_ids_sha256":"9e16b1ea32214867d7ab4ebd2dc9b09c778b3a221aba0887e2e61f722dd92197","oracle":{"probs":[0.7781065702438354,0.22189347445964813],"logits_raw":[-0.39425909519195557,-3.413048267364502],"argmax":0,"top2_margin":0.5562130808830261,"near_tie":false},"graph":{"p":[0.7759284973144531,0.22407154738903046],"max_abs_dp":0.0021780729293823242,"argmax_equal":true}}]},{"id":"tv4xh_qnli_07","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":267,"_meta":{"row":3906,"text_sha256":"e5672b77c362c7b972fb1cfffcecb6024a6b23bb903a2e8b89bb46d8282a9218","row_sha256":"0901c7bb1e03299bb60f1e70a3de28c973907aae639550a523378e73ca1a3ab5","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/3906","group_id":"qnli/validation/3906","variant":"clean"}},"state_tokens":56,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":85,"state_len":56,"decide":84,"opts":[80,83],"row_ids_sha256":"fa6717cd2cadc595dbb2eeae0b60ca5d06df174a0f0d1f30c7a1191a66674b45","oracle":{"probs":[0.9859915375709534,0.014008446596562862],"logits_raw":[3.0387325286865234,-7.196573257446289],"argmax":0,"top2_margin":0.9719830751419067,"near_tie":false},"graph":{"p":[0.9859117865562439,0.014088197611272335],"max_abs_dp":7.975101470947266e-05,"argmax_equal":true}}]},{"id":"tv4xh_qnli_08","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":268,"_meta":{"row":798,"text_sha256":"33c94e5e38ca4e83a4a28cf20e08766628445271022a436d777e09586bea4c54","row_sha256":"090efeb48f5520325b2bffb843f851a59b96d5029285aa75059a92c3a94a1674","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/798","group_id":"qnli/validation/798","variant":"clean"}},"state_tokens":18,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":45,"state_len":18,"decide":44,"opts":[40,43],"row_ids_sha256":"ffaabd2ea2e5b48ed75bb56c9023dd2fddc372f829e1b8d78597b43816c3dd69","oracle":{"probs":[0.5620395541191101,0.4379604160785675],"logits_raw":[-1.8144114017486572,-2.4145853519439697],"argmax":0,"top2_margin":0.1240791380405426,"near_tie":false},"graph":{"p":[0.5594439506530762,0.44055604934692383],"max_abs_dp":0.0025956332683563232,"argmax_equal":true}}]},{"id":"tv4xh_qnli_09","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":269,"_meta":{"row":4307,"text_sha256":"8a67f1b71d6a48a812bf0d42fc6741acba8265638e28e476a3cf5e6c0fcd1ab0","row_sha256":"09abcf8960243b05d2b20ec2cdd1204138159cf01a9019d7827eb4398b9acce4","source":"qnli","repo":"nyu-mll/glue","revision":"bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c","split":"validation","id":"qnli/validation/4307","group_id":"qnli/validation/4307","variant":"clean"}},"state_tokens":25,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":53,"state_len":25,"decide":52,"opts":[48,51],"row_ids_sha256":"d6955329fa61b8e6d83c76741f7d440297b183812f7d3de58bbfbe0a376fb150","oracle":{"probs":[0.9831987023353577,0.016801290214061737],"logits_raw":[3.024991750717163,-6.766080856323242],"argmax":0,"top2_margin":0.9663974046707153,"near_tie":false},"graph":{"p":[0.9831418991088867,0.016858171671628952],"max_abs_dp":5.6881457567214966e-05,"argmax_equal":true}}]},{"id":"tv4xh_paws_00","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":340,"_meta":{"row":6859,"text_sha256":"d133a2515f30159c5dae04ad8d8ba56b6b9d753268e97060eb8ce7113ef08801","row_sha256":"08cea23d05d2955c437449fb24606ba4401ee55476205aa55a1633a3238df235","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/6859","group_id":"paws/test/6859","variant":"clean"}},"state_tokens":31,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":94,"state_len":31,"decide":93,"opts":[81,92],"row_ids_sha256":"b2faaa697c454d68a12b7cf68a376abf8d0aec0c5c28105dd805f0ccefc5a9c3","oracle":{"probs":[0.2217463105916977,0.7782536745071411],"logits_raw":[-3.4047627449035645,-0.38392239809036255],"argmax":1,"top2_margin":0.5565073490142822,"near_tie":false},"graph":{"p":[0.2212294489145279,0.7787705659866333],"max_abs_dp":0.0005168914794921875,"argmax_equal":true}}]},{"id":"tv4xh_paws_01","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":341,"_meta":{"row":1560,"text_sha256":"3f57aafe302c45afd11ba6814fe7261a1109baebb6c0a8671dd8e4af6e07bdb7","row_sha256":"08fec8b3386611646a7473ab9a745837c008760a4612800a8d1b2d0362fc9a1c","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/1560","group_id":"paws/test/1560","variant":"clean"}},"state_tokens":15,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":61,"state_len":15,"decide":60,"opts":[48,59],"row_ids_sha256":"26887ef3c6bb0a1ad37f92bfcc0c1185405422f33fff4c03fa99e32b166beef0","oracle":{"probs":[0.1901964396238327,0.8098034858703613],"logits_raw":[-4.0508832931518555,-0.5651565790176392],"argmax":1,"top2_margin":0.6196070313453674,"near_tie":false},"graph":{"p":[0.19081521034240723,0.8091848492622375],"max_abs_dp":0.0006187707185745239,"argmax_equal":true}}]},{"id":"tv4xh_paws_02","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":342,"_meta":{"row":7712,"text_sha256":"933523538c3eb6e0fd147ff96af499139a9c056a9e3345901f9e3ee457b1a48c","row_sha256":"0901ebdc21f24c42b0651b75962b879dd88d2f015bce4121c9c110027fd959a9","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/7712","group_id":"paws/test/7712","variant":"clean"}},"state_tokens":40,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":113,"state_len":40,"decide":112,"opts":[100,111],"row_ids_sha256":"8e2ffbd6813763390eab9983ab63d67cad75a58802e83851bfea7699c4a559cc","oracle":{"probs":[0.20196881890296936,0.7980311512947083],"logits_raw":[-3.8763391971588135,-0.5703438520431519],"argmax":1,"top2_margin":0.5960623025894165,"near_tie":false},"graph":{"p":[0.19994007050991058,0.800059974193573],"max_abs_dp":0.002028822898864746,"argmax_equal":true}}]},{"id":"tv4xh_paws_03","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":343,"_meta":{"row":3530,"text_sha256":"4da4508cc8c69c075e3f4f1aaa931b5b394a47ab9ea23efa69272f55dc44237a","row_sha256":"0954a85c777d1b7a2b28638281d2e238c4238ed6f881be0d21eda0b9e2209d63","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/3530","group_id":"paws/test/3530","variant":"clean"}},"state_tokens":34,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":94,"state_len":34,"decide":93,"opts":[81,92],"row_ids_sha256":"96c275bf730b4a7caae30908419730d3455c36e0df5a7f0bd2d65e9110748224","oracle":{"probs":[0.9690364003181458,0.030963608995079994],"logits_raw":[2.466215133666992,-5.818992614746094],"argmax":0,"top2_margin":0.9380728006362915,"near_tie":false},"graph":{"p":[0.9693445563316345,0.030655397102236748],"max_abs_dp":0.00030821189284324646,"argmax_equal":true}}]},{"id":"tv4xh_paws_04","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":344,"_meta":{"row":5349,"text_sha256":"a12f11b13b1aa82b9763c901029e81375771ec4343a069c040c560d8eb39dc1b","row_sha256":"09c772b82040c213ae17b52d3e35354212d0a2dc85d4c3c6e503a372db3b9091","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/5349","group_id":"paws/test/5349","variant":"clean"}},"state_tokens":21,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":76,"state_len":21,"decide":75,"opts":[63,74],"row_ids_sha256":"66701bd0891933de7ec25692ff5a15d48a50ef4e716da58f8ecea329da939f91","oracle":{"probs":[0.36472946405410767,0.6352705359458923],"logits_raw":[-3.7337610721588135,-2.398655891418457],"argmax":1,"top2_margin":0.27054107189178467,"near_tie":false},"graph":{"p":[0.36606353521347046,0.6339364051818848],"max_abs_dp":0.0013341307640075684,"argmax_equal":true}}]},{"id":"tv4xh_paws_05","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":345,"_meta":{"row":5078,"text_sha256":"84a9caa371f97bff330cfd3abcfa5c5c761640e50d7df225849cf33513014959","row_sha256":"0a78ee6268fc5390fd0f2de018ae91de423f3ec86b913a532d2d0d3ee0481a91","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/5078","group_id":"paws/test/5078","variant":"clean"}},"state_tokens":40,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":116,"state_len":40,"decide":115,"opts":[103,114],"row_ids_sha256":"5e4dd51bb93ffdb589082e50149de67c5254e1640584f563847aeb7c93b8a8a0","oracle":{"probs":[0.04360958933830261,0.956390380859375],"logits_raw":[-4.891183376312256,2.5384325981140137],"argmax":1,"top2_margin":0.91278076171875,"near_tie":false},"graph":{"p":[0.04358185455203056,0.9564181566238403],"max_abs_dp":2.777576446533203e-05,"argmax_equal":true}}]},{"id":"tv4xh_paws_06","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":346,"_meta":{"row":3539,"text_sha256":"44505af2003c8d22ac185056a2799a0936c7b7617c63754193dcbde1805528ed","row_sha256":"0aac5ddeb4adeec6d64faf8248c48f7f682c63b8864e2242df28c6051c66e5eb","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/3539","group_id":"paws/test/3539","variant":"clean"}},"state_tokens":28,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":91,"state_len":28,"decide":90,"opts":[78,89],"row_ids_sha256":"4d15b743ee30b1ea5b934cad151808e3e6cee661e3f800d929cf068f50d6bf18","oracle":{"probs":[0.9723531007766724,0.027646826580166817],"logits_raw":[3.229860305786133,-5.336178302764893],"argmax":0,"top2_margin":0.9447062611579895,"near_tie":false},"graph":{"p":[0.971889853477478,0.02811010181903839],"max_abs_dp":0.0004632752388715744,"argmax_equal":true}}]},{"id":"tv4xh_paws_07","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":347,"_meta":{"row":4865,"text_sha256":"0136388d20728451308640a9979e44675ccfbb86ecdca02bb0de1d097ff658c6","row_sha256":"0ae573d07f9666cb5a633e726f3cc5ffa13c09f558f67b229f54de306f4a0d20","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/4865","group_id":"paws/test/4865","variant":"clean"}},"state_tokens":25,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":82,"state_len":25,"decide":81,"opts":[69,80],"row_ids_sha256":"d3ef90e2a0cfbfbaf4eea90d6c6d2ba6be1d181767f3a0e1104ce07f2b96032a","oracle":{"probs":[0.8775542974472046,0.12244575470685959],"logits_raw":[2.3191027641296387,-2.419542074203491],"argmax":0,"top2_margin":0.7551085352897644,"near_tie":false},"graph":{"p":[0.8740140795707703,0.12598593533039093],"max_abs_dp":0.003540217876434326,"argmax_equal":true}}]},{"id":"tv4xh_paws_08","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":348,"_meta":{"row":1721,"text_sha256":"6df8a93c3d0dc1aab6e8823532dbad98697089398e8c3cf4eefa19162d57a3b7","row_sha256":"0afc01bb583b9ffd617e8c2c6d87d7fcc42ccc569b4a27fe5968dcb27f16e04e","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/1721","group_id":"paws/test/1721","variant":"clean"}},"state_tokens":32,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":96,"state_len":32,"decide":95,"opts":[83,94],"row_ids_sha256":"7501d10448d99d798cd0d302b2fa1acfc1ef0afd76c1c6d9241b22e185db0735","oracle":{"probs":[0.09633317589759827,0.9036667943000793],"logits_raw":[-4.1807122230529785,1.2055864334106445],"argmax":1,"top2_margin":0.8073335886001587,"near_tie":false},"graph":{"p":[0.09514207392930984,0.904857873916626],"max_abs_dp":0.0011911019682884216,"argmax_equal":true}}]},{"id":"tv4xh_paws_09","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":349,"_meta":{"row":2075,"text_sha256":"cd001b26bb35a17a3661e47c3e9c41997bfe37e8f0e0f9a34139c31a5c8f1256","row_sha256":"0b1e3adfbd74bad139926051a69f053eb8783b7cd3ba969f6c0bc64da3aff570","source":"paws","repo":"google-research-datasets/paws","revision":"161ece9501cf0a11f3e48bd356eaa82de46d6a09","split":"test","id":"paws/test/2075","group_id":"paws/test/2075","variant":"clean"}},"state_tokens":34,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":102,"state_len":34,"decide":101,"opts":[89,100],"row_ids_sha256":"f9b67a80544795f22f2a487fd2f0de1c2ee4af4f52f3c79be09c3b55f1f03774","oracle":{"probs":[0.9223020076751709,0.0776980072259903],"logits_raw":[0.6061607599258423,-5.346510887145996],"argmax":0,"top2_margin":0.8446040153503418,"near_tie":false},"graph":{"p":[0.9221404790878296,0.07785951346158981],"max_abs_dp":0.0001615285873413086,"argmax_equal":true}}]},{"id":"tv4xh_sciq_00","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":420,"_meta":{"row":788,"text_sha256":"170c9dc2f4e8bdae8fb55bc0daa860622ec13f12d01815fc5715ea18c6001d1b","row_sha256":"089f0fbe86676aa88f493fd5bbab51c01699d8a8cb18a43ab98acb9b950c1b65","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/788","group_id":"sciq/test/788","variant":"clean"}},"state_tokens":86,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":115,"state_len":86,"decide":114,"opts":[98,103,108,113],"row_ids_sha256":"85939644ce212df560c025116568edba94b8048a9a462524c5c1bd3f96d75921","oracle":{"probs":[0.001437272410839796,0.0020825190003961325,0.21221838891506195,0.7842618227005005],"logits_raw":[-9.818920135498047,-8.926681518554688,2.198984146118164,5.343997001647949],"argmax":3,"top2_margin":0.5720434188842773,"near_tie":false},"graph":{"p":[0.0014424541732296348,0.0020868072751909494,0.21179287135601044,0.7846778631210327],"max_abs_dp":0.00042551755905151367,"argmax_equal":true}}]},{"id":"tv4xh_sciq_01","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":421,"_meta":{"row":410,"text_sha256":"c59626ff7454f49e722fb8d510f3fc827cf740161419fbcdf03c89c27b9f2ac1","row_sha256":"08b076723422504a5a47b8ee01021ce401b7d69a47950f6a991f9e0fe637f4c1","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/410","group_id":"sciq/test/410","variant":"clean"}},"state_tokens":49,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":90,"state_len":49,"decide":89,"opts":[64,73,80,88],"row_ids_sha256":"311cfcae58b7f67d2d02f10355a5c42080c8b68121f77afc7d07b9a8fcf6e32a","oracle":{"probs":[0.0026132839266210794,0.9921098351478577,0.00244450313039124,0.0028323677834123373],"logits_raw":[-8.09910774230957,6.190968036651611,-8.259749412536621,-7.905406951904297],"argmax":1,"top2_margin":0.9892774820327759,"near_tie":false},"graph":{"p":[0.002638742094859481,0.9920186400413513,0.002473771572113037,0.0028687918093055487],"max_abs_dp":9.119510650634766e-05,"argmax_equal":true}}]},{"id":"tv4xh_sciq_02","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":422,"_meta":{"row":686,"text_sha256":"f95a6c8b8bac788a6a6d238326fda619d29715cda257c89621074d940c0cc7be","row_sha256":"08e084e01d87d9f2cbde9b49eabd9e58f7acf4450ead304dbb20da2dafaecdcb","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/686","group_id":"sciq/test/686","variant":"clean"}},"state_tokens":15,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":44,"state_len":15,"decide":43,"opts":[27,32,37,42],"row_ids_sha256":"1b344042b4333357fa43720411b8bc8d3b757730480cf5c1785bc0b252ecb87f","oracle":{"probs":[0.40391334891319275,0.025530511513352394,0.5522688031196594,0.018287353217601776],"logits_raw":[1.8897531032562256,-4.7541351318359375,2.642448663711548,-5.556949138641357],"argmax":2,"top2_margin":0.14835545420646667,"near_tie":false},"graph":{"p":[0.4035508632659912,0.02573258988559246,0.5523021817207336,0.018414320424199104],"max_abs_dp":0.0003624856472015381,"argmax_equal":true}}]},{"id":"tv4xh_sciq_03","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":423,"_meta":{"row":976,"text_sha256":"fded9b7a4f8b099c6859a4b62ebcd0138ad19b093deab16390cee8e81bdce783","row_sha256":"09199701412203509a8df1c8625f4915047a241dd6f600b4794dfd858a9fcd87","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/976","group_id":"sciq/test/976","variant":"clean"}},"state_tokens":92,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":124,"state_len":92,"decide":123,"opts":[104,110,117,122],"row_ids_sha256":"cdd07906e7793edbf272943915d3925b896c1dc6c4b50b7b2f467ee1fb0a1a25","oracle":{"probs":[0.9755893349647522,0.0044140941463410854,0.012646042741835117,0.007350550498813391],"logits_raw":[5.472659111022949,-7.515773773193359,-4.983305931091309,-6.288754463195801],"argmax":0,"top2_margin":0.9629433155059814,"near_tie":false},"graph":{"p":[0.9753456115722656,0.0044935159385204315,0.01272208709269762,0.007438824512064457],"max_abs_dp":0.00024372339248657227,"argmax_equal":true}}]},{"id":"tv4xh_sciq_04","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":424,"_meta":{"row":137,"text_sha256":"ba59711c47485dd22635a822488147114de4269bcd004b5f239a21eabba1b52f","row_sha256":"092761ded6b92eda54ab2be77c2961e46f118981511d4bbef3c6a6b4b0b2e173","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/137","group_id":"sciq/test/137","variant":"clean"}},"state_tokens":138,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":173,"state_len":138,"decide":172,"opts":[151,156,164,171],"row_ids_sha256":"5c848f8f41675a843585e6ba8aab55b344ed3d1e8f3b64c13ca4ee8e8a89611a","oracle":{"probs":[0.001599183538928628,0.0007019835757091641,0.995394766330719,0.002304028021171689],"logits_raw":[-10.091766357421875,-12.072760581970215,5.387907981872559,-9.213159561157227],"argmax":2,"top2_margin":0.9930907487869263,"near_tie":false},"graph":{"p":[0.0016178539954125881,0.0007166651193983853,0.9953323006629944,0.0023332317359745502],"max_abs_dp":6.246566772460938e-05,"argmax_equal":true}}]},{"id":"tv4xh_sciq_05","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":425,"_meta":{"row":35,"text_sha256":"47508d01df5c535f3993458b1da869515387561265ba901c455c9f88935cd443","row_sha256":"092c2204f4631862369ba3f128354dd476f8039320d2b30c5f0f4d747668d6c6","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/35","group_id":"sciq/test/35","variant":"clean"}},"state_tokens":17,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":51,"state_len":17,"decide":50,"opts":[30,37,43,49],"row_ids_sha256":"ded10c00e80ab96932a5c187b430c2d8023d8ff18bcc43fc79dd4a47e5f816ad","oracle":{"probs":[0.2448919713497162,0.6665546298027039,0.043333638459444046,0.04521973803639412],"logits_raw":[-2.0717663764953613,0.33742350339889526,-6.23877477645874,-6.136266708374023],"argmax":1,"top2_margin":0.42166265845298767,"near_tie":false},"graph":{"p":[0.244114950299263,0.6671082973480225,0.043177176266908646,0.04559963196516037],"max_abs_dp":0.000777021050453186,"argmax_equal":true}}]},{"id":"tv4xh_sciq_06","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":426,"_meta":{"row":147,"text_sha256":"1f357b2078baccd4b71f306ba81f4168d1efa1af175dc640dd49aaafe4ed5807","row_sha256":"092f40e1f611dfa93a75e886c3fc54805a4c9388ad4987b3f75104cd6dcb844f","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/147","group_id":"sciq/test/147","variant":"clean"}},"state_tokens":50,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":80,"state_len":50,"decide":79,"opts":[62,67,72,78],"row_ids_sha256":"d1dbf2d3b2b2924424f6e9087e4c4a04c4462a101103b3bdf28d2c1b6f3f0c52","oracle":{"probs":[0.00042063058936037123,0.005485112778842449,0.08442538231611252,0.9096688628196716],"logits_raw":[-11.906623840332031,-5.727797508239746,0.8499356508255005,6.56962776184082],"argmax":3,"top2_margin":0.8252434730529785,"near_tie":false},"graph":{"p":[0.0004260787391103804,0.005512352101504803,0.08417896181344986,0.9098826050758362],"max_abs_dp":0.0002464205026626587,"argmax_equal":true}}]},{"id":"tv4xh_sciq_07","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":427,"_meta":{"row":587,"text_sha256":"9dac3ba48027e8b7920c28a5d9235c43e6acc3f513b2c3c2ca1a8519e73533a9","row_sha256":"096911b23029621e070636b393a01e0af96ecf891faf1425a9696c0694fff9b4","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/587","group_id":"sciq/test/587","variant":"clean"}},"state_tokens":42,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":71,"state_len":42,"decide":70,"opts":[54,59,64,69],"row_ids_sha256":"7e26de724de7962a006100736ce583147b45968f2da2638ac147da438a58730c","oracle":{"probs":[0.001491582952439785,0.002996455878019333,0.9901238083839417,0.005388127639889717],"logits_raw":[-11.069089889526367,-9.390647888183594,4.565403938293457,-7.978855133056641],"argmax":2,"top2_margin":0.9847356677055359,"near_tie":false},"graph":{"p":[0.00151135865598917,0.0030223065987229347,0.9900552034378052,0.005411110818386078],"max_abs_dp":6.860494613647461e-05,"argmax_equal":true}}]},{"id":"tv4xh_sciq_08","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":428,"_meta":{"row":391,"text_sha256":"bb74034114f171d22ac664d0c93b509caf320f74c1165c459bf85f5ceee71fd9","row_sha256":"0980dc8fcead1e881d97e910a81689b6d90c7efc15f8794274102390abcbd81c","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/391","group_id":"sciq/test/391","variant":"clean"}},"state_tokens":80,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":112,"state_len":80,"decide":111,"opts":[94,99,104,110],"row_ids_sha256":"1f052fa986dc2ddf8db857d968542c8a0451b10164a4709b9b5534dd21a3bb1d","oracle":{"probs":[0.0033465949818491936,0.9719259142875671,0.013858530670404434,0.010868948884308338],"logits_raw":[-8.261676788330078,5.383841037750244,-4.842782020568848,-5.427430152893066],"argmax":1,"top2_margin":0.9580673575401306,"near_tie":false},"graph":{"p":[0.0034071882255375385,0.971771240234375,0.013889321126043797,0.01093230675905943],"max_abs_dp":0.00015467405319213867,"argmax_equal":true}}]},{"id":"tv4xh_sciq_09","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":429,"_meta":{"row":152,"text_sha256":"d802e064623d270f2c3f8943f3a50f5f20dd68352561f76f818dd74b7ed93274","row_sha256":"098e4320fc7563547edc0b02ef6fd8785d06e77c5f13e132ee92b91d4a7bffc9","source":"sciq","repo":"allenai/sciq","revision":"2c94ad3e1aafab77146f384e23536f97a4849815","split":"test","id":"sciq/test/152","group_id":"sciq/test/152","variant":"clean"}},"state_tokens":150,"questions":[{"q":0,"type":"choice","n_options":4,"row_len":185,"state_len":150,"decide":184,"opts":[163,169,175,183],"row_ids_sha256":"5e55e98e89bb67b9074379aa423e38bfe59162cf671ff7b0a73a78516b2d2cd6","oracle":{"probs":[0.0013307149056345224,0.9807592034339905,0.003617843147367239,0.014292181469500065],"logits_raw":[-11.39891242980957,4.487298965454102,-8.992473602294922,-5.686958312988281],"argmax":1,"top2_margin":0.966467022895813,"near_tie":false},"graph":{"p":[0.0013475031591951847,0.9805927276611328,0.0036648360546678305,0.014394883066415787],"max_abs_dp":0.00016647577285766602,"argmax_equal":true}}]},{"id":"tv4xh_legacy_holdout_00","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":608,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0010","pair_id":"v3-legacy-transfer-authorization-0010","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0010","id":"contrastive/authorization/v3-legacy-transfer-authorization-0010/a","text_sha256":"2ac33202a06425450b3a236002e2add85eca8f5bc6094f6260822e29b29b2074","row_sha256":"8f9e8f9d91b8b3c5ab87f3b8ed6882f368a521c502000b7c37b886935178a782","group_id":"v3-legacy-transfer-authorization-0010","variant":"clean"}},"state_tokens":62,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":75,"state_len":62,"decide":74,"opts":[70,73],"row_ids_sha256":"4ff936967498cf80fff7e0b829597f7550c95dcc3d59026b0ec0b7820c7bbd8c","oracle":{"probs":[0.005530292633920908,0.9944697618484497],"logits_raw":[-7.283004283905029,5.209131717681885],"argmax":1,"top2_margin":0.9889394640922546,"near_tie":false},"graph":{"p":[0.005602280143648386,0.994397759437561],"max_abs_dp":7.200241088867188e-05,"argmax_equal":true}}]},{"id":"tv4xh_legacy_holdout_01","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":609,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0010","pair_id":"v3-legacy-transfer-authorization-0010","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0010","id":"contrastive/authorization/v3-legacy-transfer-authorization-0010/b","text_sha256":"207a5a797dd41df412b0a35a3643f88083c8ba89c2eb3cf1f78538f13eef40d5","row_sha256":"8696645f99fda114766ee32c3b31ebf785c1aa93b1f8d63e2202149d54cc086e","group_id":"v3-legacy-transfer-authorization-0010","variant":"clean"}},"state_tokens":62,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":75,"state_len":62,"decide":74,"opts":[70,73],"row_ids_sha256":"772001f039c03433fd8f5f08d54a3fbde08f3ea3badc8742ab98f0034e5cbeda","oracle":{"probs":[0.9952870607376099,0.0047128633596003056],"logits_raw":[6.505980491638184,-6.372969150543213],"argmax":0,"top2_margin":0.9905741810798645,"near_tie":false},"graph":{"p":[0.9952261447906494,0.004773903638124466],"max_abs_dp":6.104027852416039e-05,"argmax_equal":true}}]},{"id":"tv4xh_legacy_holdout_02","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":610,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0011","pair_id":"v3-legacy-transfer-authorization-0011","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0011","id":"contrastive/authorization/v3-legacy-transfer-authorization-0011/a","text_sha256":"4c4b0487b6b6c9b022f3870d584814d80e4f40ac777172038bda483c7f8ab741","row_sha256":"d0a716aa1cb80425b7682ad7671a84ef7f8bf99aa7ad7a74b4e03349f414436f","group_id":"v3-legacy-transfer-authorization-0011","variant":"clean"}},"state_tokens":63,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":76,"state_len":63,"decide":75,"opts":[71,74],"row_ids_sha256":"2a12f9e6c9de2726c1d812d62ef37098768608d89aeaeaeb08e430498039b576","oracle":{"probs":[0.004478619899600744,0.9955214262008667],"logits_raw":[-7.559093475341797,5.443084716796875],"argmax":1,"top2_margin":0.9910427927970886,"near_tie":false},"graph":{"p":[0.004566629882901907,0.9954333901405334],"max_abs_dp":8.803606033325195e-05,"argmax_equal":true}}]},{"id":"tv4xh_legacy_holdout_03","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":611,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0011","pair_id":"v3-legacy-transfer-authorization-0011","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0011","id":"contrastive/authorization/v3-legacy-transfer-authorization-0011/b","text_sha256":"a3fbd4cfe8588ca42161d56c340dafd53e834a8d6b47507611f2f35553c60d2e","row_sha256":"8a00bdd178fc7ab32b2392906cfdd8bb74c04d23842506222d46e8c526de3274","group_id":"v3-legacy-transfer-authorization-0011","variant":"clean"}},"state_tokens":64,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":77,"state_len":64,"decide":76,"opts":[72,75],"row_ids_sha256":"d43a8095eeb2e0b712e648cbb2e9cc495e81cea1fb5a992c2690f4d17c615182","oracle":{"probs":[0.9965435862541199,0.0034564295783638954],"logits_raw":[6.4724273681640625,-7.155576705932617],"argmax":0,"top2_margin":0.9930871725082397,"near_tie":false},"graph":{"p":[0.9964989423751831,0.0035010352730751038],"max_abs_dp":4.464387893676758e-05,"argmax_equal":true}}]},{"id":"tv4xh_legacy_holdout_04","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":612,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0012","pair_id":"v3-legacy-transfer-authorization-0012","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0012","id":"contrastive/authorization/v3-legacy-transfer-authorization-0012/a","text_sha256":"70358674ea1eb7a2b4304de22754ec09d492d0f012fdeb25034914b0487b6de2","row_sha256":"5829235d159024fe1ee0c45ca901977dc2d6c392e53a9bdffe691c975ec41491","group_id":"v3-legacy-transfer-authorization-0012","variant":"clean"}},"state_tokens":64,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":77,"state_len":64,"decide":76,"opts":[72,75],"row_ids_sha256":"39b9d7256a98e6a942d9668feff453143d03b39ce00f73b257bd7a0ba2ccada3","oracle":{"probs":[0.00781749002635479,0.9921824336051941],"logits_raw":[-6.877280235290527,4.7765278816223145],"argmax":1,"top2_margin":0.984364926815033,"near_tie":false},"graph":{"p":[0.008004388771951199,0.9919956922531128],"max_abs_dp":0.00018689874559640884,"argmax_equal":true}}]},{"id":"tv4xh_legacy_holdout_05","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":613,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0012","pair_id":"v3-legacy-transfer-authorization-0012","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0012","id":"contrastive/authorization/v3-legacy-transfer-authorization-0012/b","text_sha256":"e66cbeedd72ceb3350f3dbc36b5eb1ce9df69e2a65fa177459b9c4d916eccb6d","row_sha256":"b1bd359c5413cad9411f83d4280580493a147db3903f6a4399aa9ab6fa35ad7c","group_id":"v3-legacy-transfer-authorization-0012","variant":"clean"}},"state_tokens":63,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":76,"state_len":63,"decide":75,"opts":[71,74],"row_ids_sha256":"c89bfd79ccf59ad139e97184231ace4eef9092680b54505fee7175135db4f8b4","oracle":{"probs":[0.9963489770889282,0.003650989383459091],"logits_raw":[6.348134994506836,-7.147637844085693],"argmax":0,"top2_margin":0.9926980137825012,"near_tie":false},"graph":{"p":[0.9962888956069946,0.003711126046255231],"max_abs_dp":6.013666279613972e-05,"argmax_equal":true}}]},{"id":"tv4xh_legacy_holdout_06","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":614,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0013","pair_id":"v3-legacy-transfer-authorization-0013","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0013","id":"contrastive/authorization/v3-legacy-transfer-authorization-0013/a","text_sha256":"9ea1528671ea68f95b395ebe9ac59201e76192ed3ae4a84f5f3731bfbc2d0bc5","row_sha256":"dab2995b774ef92b9a4625ab67555d2eecb87ac4a6165e59ae39b8bc91f13fc0","group_id":"v3-legacy-transfer-authorization-0013","variant":"clean"}},"state_tokens":63,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":76,"state_len":63,"decide":75,"opts":[71,74],"row_ids_sha256":"f3fba8eaad7037a67a95b3a89470564454716704411ddfc71de9b34739bf9ba4","oracle":{"probs":[0.005305086728185415,0.9946948885917664],"logits_raw":[-7.256594181060791,5.336117744445801],"argmax":1,"top2_margin":0.9893897771835327,"near_tie":false},"graph":{"p":[0.005391776096075773,0.9946081638336182],"max_abs_dp":8.672475814819336e-05,"argmax_equal":true}}]},{"id":"tv4xh_legacy_holdout_07","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":615,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0013","pair_id":"v3-legacy-transfer-authorization-0013","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0013","id":"contrastive/authorization/v3-legacy-transfer-authorization-0013/b","text_sha256":"d88b238a44fb7d026de87ab14cef084e9c629c16e73655a8e9c5ac2effe3d5cc","row_sha256":"a1cc230ce00109c7e4180f7929b9eb1eb919c39bda07e751562bbefacead253e","group_id":"v3-legacy-transfer-authorization-0013","variant":"clean"}},"state_tokens":64,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":77,"state_len":64,"decide":76,"opts":[72,75],"row_ids_sha256":"1b426e46574cdfb256b7a4a3f656a6a6c02d4ba83e78d33c3ccdb3ea73902c93","oracle":{"probs":[0.9961828589439392,0.0038170746993273497],"logits_raw":[6.647337436676025,-6.740999221801758],"argmax":0,"top2_margin":0.9923657774925232,"near_tie":false},"graph":{"p":[0.996119499206543,0.0038805005606263876],"max_abs_dp":6.342586129903793e-05,"argmax_equal":true}}]},{"id":"tv4xh_legacy_holdout_08","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":616,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0014","pair_id":"v3-legacy-transfer-authorization-0014","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0014","id":"contrastive/authorization/v3-legacy-transfer-authorization-0014/a","text_sha256":"5e5d94d8e1caaac4ccc98f427dcbf7b88d1ee6640c146ff35f3f4977b6e6735c","row_sha256":"21a16cb236c149612f5fd4a80e84b426b14f02f17665088c0fc8f181ec849e72","group_id":"v3-legacy-transfer-authorization-0014","variant":"clean"}},"state_tokens":64,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":77,"state_len":64,"decide":76,"opts":[72,75],"row_ids_sha256":"9d9ff763fd24fa5306a2c809e26ed959d44c73b33276fc506cba942ee346d175","oracle":{"probs":[0.01001625508069992,0.9899836778640747],"logits_raw":[-6.658547401428223,4.393592357635498],"argmax":1,"top2_margin":0.9799674153327942,"near_tie":false},"graph":{"p":[0.010189001448452473,0.9898110628128052],"max_abs_dp":0.00017274636775255203,"argmax_equal":true}}]},{"id":"tv4xh_legacy_holdout_09","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":617,"_meta":{"source":"legacy_holdout","family":"authorization","family_id":"authorization/v3-legacy-transfer-authorization-0014","pair_id":"v3-legacy-transfer-authorization-0014","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-authorization-0014","id":"contrastive/authorization/v3-legacy-transfer-authorization-0014/b","text_sha256":"804b414a261a737f6bab9cd62d420ad7f6aca02cab94b0f86cc27d1af97bf38c","row_sha256":"e51f048b2a4e12e4bceaa23b4edbb570287ebda28dcb7d662ea632e07f677208","group_id":"v3-legacy-transfer-authorization-0014","variant":"clean"}},"state_tokens":63,"questions":[{"q":0,"type":"noul","n_options":2,"row_len":76,"state_len":63,"decide":75,"opts":[71,74],"row_ids_sha256":"0b435357dea69ad230ad4beae130c9fbee5357f866ee2253e8791608844780cd","oracle":{"probs":[0.9925734996795654,0.007426523603498936],"logits_raw":[6.22642183303833,-5.5517778396606445],"argmax":0,"top2_margin":0.9851469993591309,"near_tie":false},"graph":{"p":[0.9925476908683777,0.0074523151852190495],"max_abs_dp":2.580881118774414e-05,"argmax_equal":true}}]},{"id":"tv4xh_composition_holdout_00","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":688,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/5/relevant/a","group_id":"composition/v3-composition-transfer/held_and_or/5","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/5/relevant","sibling":"a","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"251beac991d1b72ad9ca630d5713afaaca28c5546f2dc9d7174ce38417ccaf7b","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["member value"],"threshold":60},{"kind":"match","fields":["entry signer","entry designated approver"],"threshold":43},{"kind":"range","fields":["review value"],"threshold":27}],"facts":{"member value":60,"entry signer":"Elin","entry designated approver":"Noah","review value":26,"routing reference":604},"order":["member value","entry designated approver","entry signer","review value","routing reference"],"deciding_field":"review value","label":false}}},"state_tokens":116,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":145,"state_len":116,"decide":144,"opts":[134,143],"row_ids_sha256":"72d82d61c852c6ac2db1bfa47fc655eaeb0d040c01cd8f8d6756bf24cd143b56","oracle":{"probs":[0.9826298952102661,0.017370188608765602],"logits_raw":[4.424716949462891,-5.284842491149902],"argmax":0,"top2_margin":0.9652597308158875,"near_tie":false},"graph":{"p":[0.9826055765151978,0.01739446446299553],"max_abs_dp":2.4318695068359375e-05,"argmax_equal":true}}]},{"id":"tv4xh_composition_holdout_01","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":689,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/5/relevant/b","group_id":"composition/v3-composition-transfer/held_and_or/5","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/5/relevant","sibling":"b","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"5742c6c5c95abb772f41cf33b0ee4fcd42c492eafdac4ba407bc86336056f6db","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["member value"],"threshold":60},{"kind":"match","fields":["entry signer","entry designated approver"],"threshold":43},{"kind":"range","fields":["review value"],"threshold":27}],"facts":{"member value":60,"entry signer":"Elin","entry designated approver":"Noah","review value":27,"routing reference":604},"order":["member value","entry designated approver","entry signer","review value","routing reference"],"deciding_field":"review value","label":true}}},"state_tokens":116,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":145,"state_len":116,"decide":144,"opts":[134,143],"row_ids_sha256":"ddf3a10c7ae48d40b8bca252d1e16c9634b2cdc9d494312064d2f61ce1bf2e5b","oracle":{"probs":[0.02453497052192688,0.9754649996757507],"logits_raw":[-5.0374603271484375,3.823575973510742],"argmax":1,"top2_margin":0.9509299993515015,"near_tie":false},"graph":{"p":[0.02459212951362133,0.9754078388214111],"max_abs_dp":5.716085433959961e-05,"argmax_equal":true}}]},{"id":"tv4xh_composition_holdout_02","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":690,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/5/irrelevant/a","group_id":"composition/v3-composition-transfer/held_and_or/5","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/5/irrelevant","sibling":"a","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"251beac991d1b72ad9ca630d5713afaaca28c5546f2dc9d7174ce38417ccaf7b","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["member value"],"threshold":60},{"kind":"match","fields":["entry signer","entry designated approver"],"threshold":43},{"kind":"range","fields":["review value"],"threshold":27}],"facts":{"member value":60,"entry signer":"Elin","entry designated approver":"Noah","review value":26,"routing reference":604},"order":["member value","entry designated approver","entry signer","review value","routing reference"],"deciding_field":"review value","label":false}}},"state_tokens":116,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":145,"state_len":116,"decide":144,"opts":[134,143],"row_ids_sha256":"72d82d61c852c6ac2db1bfa47fc655eaeb0d040c01cd8f8d6756bf24cd143b56","oracle":{"probs":[0.9826298952102661,0.017370188608765602],"logits_raw":[4.424716949462891,-5.284842491149902],"argmax":0,"top2_margin":0.9652597308158875,"near_tie":false},"graph":{"p":[0.9826055765151978,0.01739446446299553],"max_abs_dp":2.4318695068359375e-05,"argmax_equal":true}}]},{"id":"tv4xh_composition_holdout_03","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":691,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/5/irrelevant/b","group_id":"composition/v3-composition-transfer/held_and_or/5","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/5/irrelevant","sibling":"b","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"e171f541fec96b43ea97de6218051909a0b44ac40e9031152a1bc858f901f36a","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"ge","fields":["member value"],"threshold":60},{"kind":"match","fields":["entry signer","entry designated approver"],"threshold":43},{"kind":"range","fields":["review value"],"threshold":27}],"facts":{"member value":60,"entry signer":"Elin","entry designated approver":"Noah","review value":26,"routing reference":293},"order":["member value","entry designated approver","entry signer","review value","routing reference"],"deciding_field":"review value","label":false}}},"state_tokens":116,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":145,"state_len":116,"decide":144,"opts":[134,143],"row_ids_sha256":"d2fceec35f219297d03be442c8699bfdc4e1f83f0fea9b828ce87032041596e2","oracle":{"probs":[0.982767641544342,0.017232324928045273],"logits_raw":[4.400705337524414,-5.32836389541626],"argmax":0,"top2_margin":0.9655353426933289,"near_tie":false},"graph":{"p":[0.982833981513977,0.017166046425700188],"max_abs_dp":6.633996963500977e-05,"argmax_equal":true}}]},{"id":"tv4xh_composition_holdout_04","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":692,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/6/relevant/a","group_id":"composition/v3-composition-transfer/held_and_or/6","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/6/relevant","sibling":"a","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"7353c7726b7121059309fd377f668e83744801d7610694938d98685c75d17ac6","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"eq","fields":["shipment value"],"threshold":25},{"kind":"gt","fields":["review value"],"threshold":19},{"kind":"flag","fields":["entry verified"],"threshold":28}],"facts":{"shipment value":36,"review value":20,"entry verified":false,"routing reference":171},"order":["entry verified","review value","routing reference","shipment value"],"deciding_field":"entry verified","label":false}}},"state_tokens":93,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":122,"state_len":93,"decide":121,"opts":[111,120],"row_ids_sha256":"44f5dcf6adc5a154147a0976a23f9596b25bbf74dda048646de7498aadb62a10","oracle":{"probs":[0.9914605617523193,0.00853948574513197],"logits_raw":[5.077467441558838,-6.362044811248779],"argmax":0,"top2_margin":0.9829210638999939,"near_tie":false},"graph":{"p":[0.9913631677627563,0.008636834099888802],"max_abs_dp":9.739398956298828e-05,"argmax_equal":true}}]},{"id":"tv4xh_composition_holdout_05","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":693,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/6/relevant/b","group_id":"composition/v3-composition-transfer/held_and_or/6","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/6/relevant","sibling":"b","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"bd8f6e378d492fa24aa8c05b8eb162a455fccaa1c2d63c0050d5b6948ce7f8a0","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"eq","fields":["shipment value"],"threshold":25},{"kind":"gt","fields":["review value"],"threshold":19},{"kind":"flag","fields":["entry verified"],"threshold":28}],"facts":{"shipment value":36,"review value":20,"entry verified":true,"routing reference":171},"order":["entry verified","review value","routing reference","shipment value"],"deciding_field":"entry verified","label":true}}},"state_tokens":93,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":122,"state_len":93,"decide":121,"opts":[111,120],"row_ids_sha256":"7feecd5056ddbbda8684c8699b5a7d77a45a2c46aa8b15961cdf48a4bb8868fc","oracle":{"probs":[0.015708433464169502,0.9842915534973145],"logits_raw":[-5.378754615783691,4.576816558837891],"argmax":1,"top2_margin":0.9685831069946289,"near_tie":false},"graph":{"p":[0.015670547261834145,0.9843294620513916],"max_abs_dp":3.790855407714844e-05,"argmax_equal":true}}]},{"id":"tv4xh_composition_holdout_06","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":694,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/6/irrelevant/a","group_id":"composition/v3-composition-transfer/held_and_or/6","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/6/irrelevant","sibling":"a","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"7353c7726b7121059309fd377f668e83744801d7610694938d98685c75d17ac6","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"eq","fields":["shipment value"],"threshold":25},{"kind":"gt","fields":["review value"],"threshold":19},{"kind":"flag","fields":["entry verified"],"threshold":28}],"facts":{"shipment value":36,"review value":20,"entry verified":false,"routing reference":171},"order":["entry verified","review value","routing reference","shipment value"],"deciding_field":"entry verified","label":false}}},"state_tokens":93,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":122,"state_len":93,"decide":121,"opts":[111,120],"row_ids_sha256":"44f5dcf6adc5a154147a0976a23f9596b25bbf74dda048646de7498aadb62a10","oracle":{"probs":[0.9914605617523193,0.00853948574513197],"logits_raw":[5.077467441558838,-6.362044811248779],"argmax":0,"top2_margin":0.9829210638999939,"near_tie":false},"graph":{"p":[0.9913631677627563,0.008636834099888802],"max_abs_dp":9.739398956298828e-05,"argmax_equal":true}}]},{"id":"tv4xh_composition_holdout_07","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":695,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/6/irrelevant/b","group_id":"composition/v3-composition-transfer/held_and_or/6","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/6/irrelevant","sibling":"b","pair_kind":"irrelevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"95ded910e2fd724e735da663b6f6c9e7e25ca07902beda1a9bf409f494b31402","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"eq","fields":["shipment value"],"threshold":25},{"kind":"gt","fields":["review value"],"threshold":19},{"kind":"flag","fields":["entry verified"],"threshold":28}],"facts":{"shipment value":36,"review value":20,"entry verified":false,"routing reference":692},"order":["entry verified","review value","routing reference","shipment value"],"deciding_field":"entry verified","label":false}}},"state_tokens":93,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":122,"state_len":93,"decide":121,"opts":[111,120],"row_ids_sha256":"5c983c072e3938d04d462852904e3afb46e3dbeebcb3f804bddbaf636123c76f","oracle":{"probs":[0.9913859367370605,0.00861404463648796],"logits_raw":[5.041863441467285,-6.376551628112793],"argmax":0,"top2_margin":0.9827718734741211,"near_tie":false},"graph":{"p":[0.9912922978401184,0.00870770774781704],"max_abs_dp":9.366311132907867e-05,"argmax_equal":true}}]},{"id":"tv4xh_composition_holdout_08","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":696,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/7/relevant/a","group_id":"composition/v3-composition-transfer/held_and_or/7","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/7/relevant","sibling":"a","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"11ebc2a7a60f00b085f98a334782aea0934153e1f3797e2ba71fd49ebb4e6e24","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"lt","fields":["request value"],"threshold":14},{"kind":"flag","fields":["package verified"],"threshold":5},{"kind":"ge","fields":["member value"],"threshold":39}],"facts":{"request value":15,"package verified":true,"member value":38,"routing reference":827},"order":["request value","routing reference","package verified","member value"],"deciding_field":"member value","label":false}}},"state_tokens":93,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":122,"state_len":93,"decide":121,"opts":[109,120],"row_ids_sha256":"fc1dcc418db4bcae38a43f6d136eab42f4325c574b67de52b55f99e2f204f172","oracle":{"probs":[0.008238524198532104,0.9917614459991455],"logits_raw":[-6.264581680297852,5.261988639831543],"argmax":1,"top2_margin":0.983522891998291,"near_tie":false},"graph":{"p":[0.00843882653862238,0.991561233997345],"max_abs_dp":0.0002003023400902748,"argmax_equal":true}}]},{"id":"tv4xh_composition_holdout_09","source":"transfer_v4_dev_sources_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":697,"_meta":{"id":"composition/v3-composition-transfer/held_and_or/7/relevant/b","group_id":"composition/v3-composition-transfer/held_and_or/7","source":"composition_holdout","variant":"clean","pair_id":"composition/v3-composition-transfer/held_and_or/7/relevant","sibling":"b","pair_kind":"relevant","family":"held_and_or","family_id":"held_and_or","render_style":1,"text_sha256":"47249d3f980ee4f26c4fdaf8f7f60d8479c35fc4f45d4ea9f0278492a6b4f528","certificate":{"tree":["and",["or",0,1],2],"atoms":[{"kind":"lt","fields":["request value"],"threshold":14},{"kind":"flag","fields":["package verified"],"threshold":5},{"kind":"ge","fields":["member value"],"threshold":39}],"facts":{"request value":15,"package verified":true,"member value":39,"routing reference":827},"order":["request value","routing reference","package verified","member value"],"deciding_field":"member value","label":true}}},"state_tokens":93,"questions":[{"q":0,"type":"choice","n_options":2,"row_len":122,"state_len":93,"decide":121,"opts":[109,120],"row_ids_sha256":"e5ffdf47181eb6f78638e8d18163a45e1526a20a6b0bfc10beb39cc8fa67763b","oracle":{"probs":[0.9925859570503235,0.007414059713482857],"logits_raw":[5.333069801330566,-6.4492011070251465],"argmax":0,"top2_margin":0.985171914100647,"near_tie":false},"graph":{"p":[0.9925002455711365,0.007499769330024719],"max_abs_dp":8.571147918701172e-05,"argmax_equal":true}}]},{"id":"tv4sh_00","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":648,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0010","pair_id":"v3-legacy-transfer-deadline-0010","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0010","id":"contrastive/deadline/v3-legacy-transfer-deadline-0010/a","text_sha256":"7b32f3b140896e6014ca68fe6005032694325a0fe586deedec53c90f3edfdf57","row_sha256":"abb6f317836f5b1c653110cd9a1a2bb6caa6ffde537b07e10eae0b242268a870","group_id":"v3-legacy-transfer-deadline-0010","variant":"clean"}},"state_tokens":77,"questions":[{"q":0,"type":"score","n_options":3,"row_len":98,"state_len":77,"decide":97,"opts":[87,92,96],"row_ids_sha256":"f72286f0e0cf1a42aaeccaf9abe10b80d9d89e8cb8ed90fddbbf5f4974ded440","oracle":{"probs":[0.9928833246231079,0.004162494093179703,0.002954128198325634],"logits_raw":[5.3532304763793945,-7.818686485290527,-8.643747329711914],"argmax":0,"top2_margin":0.9887208342552185,"near_tie":false},"graph":{"p":[0.9927782416343689,0.004213162697851658,0.0030085777398198843],"max_abs_dp":0.00010508298873901367,"argmax_equal":true}}]},{"id":"tv4sh_01","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":649,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0010","pair_id":"v3-legacy-transfer-deadline-0010","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0010","id":"contrastive/deadline/v3-legacy-transfer-deadline-0010/b","text_sha256":"3209194f4ecf26e0b1ee079354d40c7ca89a478a7a4ee6a7dba34ab145dcba2c","row_sha256":"12054c79dfc1493a2fdb659a1f1fa4a96959f13c98c6a42e94e5494c18b2aba6","group_id":"v3-legacy-transfer-deadline-0010","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"score","n_options":3,"row_len":99,"state_len":78,"decide":98,"opts":[88,93,97],"row_ids_sha256":"737034910388376c70929602c7cc2256165249bbcaf03dc694b23ddeb78d0148","oracle":{"probs":[0.001939888927154243,0.009620186872780323,0.9884399175643921],"logits_raw":[-10.777341842651367,-6.924695014953613,4.22076416015625],"argmax":2,"top2_margin":0.978819727897644,"near_tie":false},"graph":{"p":[0.001987962983548641,0.009829063899815083,0.9881829619407654],"max_abs_dp":0.000256955623626709,"argmax_equal":true}}]},{"id":"tv4sh_02","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":650,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0011","pair_id":"v3-legacy-transfer-deadline-0011","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0011","id":"contrastive/deadline/v3-legacy-transfer-deadline-0011/a","text_sha256":"caa7e35da81a3cb28a58aed070b8862a1b9d3af997e9743b76b3293e18974942","row_sha256":"858badab820e2dfef0501519d23eb6bce180ca7971473cfe705f673611d68915","group_id":"v3-legacy-transfer-deadline-0011","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"score","n_options":3,"row_len":99,"state_len":78,"decide":98,"opts":[88,93,97],"row_ids_sha256":"e871e137493d6de69e2c5cd2a6a0870feeb156ca21e880b729d3d80df85d010f","oracle":{"probs":[0.9238225221633911,0.03407422453165054,0.04210326820611954],"logits_raw":[1.9274169206619263,-6.0124969482421875,-5.503414630889893],"argmax":0,"top2_margin":0.8817192316055298,"near_tie":false},"graph":{"p":[0.9213100075721741,0.03508860245347023,0.04360131546854973],"max_abs_dp":0.002512514591217041,"argmax_equal":true}}]},{"id":"tv4sh_03","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":651,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0011","pair_id":"v3-legacy-transfer-deadline-0011","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0011","id":"contrastive/deadline/v3-legacy-transfer-deadline-0011/b","text_sha256":"5c5a0edb17d07ebd90c032acf716143b30671f5316b69e17723fb59ec9bd7c69","row_sha256":"f15a66a6b56cb9946b7968bdb77c53ae81e58f8d5825504b8b20ea286e8cc003","group_id":"v3-legacy-transfer-deadline-0011","variant":"clean"}},"state_tokens":77,"questions":[{"q":0,"type":"score","n_options":3,"row_len":98,"state_len":77,"decide":97,"opts":[87,92,96],"row_ids_sha256":"7ac7abf50367359ea506862658ca944c3e6e99ef12c61e4758ccb1b7345c12f3","oracle":{"probs":[0.007145531941205263,0.9859076142311096,0.0069467853754758835],"logits_raw":[-8.655280113220215,3.1995091438293457,-8.723151206970215],"argmax":1,"top2_margin":0.9787620902061462,"near_tie":false},"graph":{"p":[0.007198088336735964,0.9857282042503357,0.007073798216879368],"max_abs_dp":0.00017940998077392578,"argmax_equal":true}}]},{"id":"tv4sh_04","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":652,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0012","pair_id":"v3-legacy-transfer-deadline-0012","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0012","id":"contrastive/deadline/v3-legacy-transfer-deadline-0012/a","text_sha256":"a79e54bf80e987aa5dcbe6ac7adecb143981024bf41397e311360bec7a034e3c","row_sha256":"fc9f9d8c34210cdb9a943bfb17443651fe3bff5c41ab5462fd0e85b23ebb9bae","group_id":"v3-legacy-transfer-deadline-0012","variant":"clean"}},"state_tokens":77,"questions":[{"q":0,"type":"score","n_options":3,"row_len":98,"state_len":77,"decide":97,"opts":[87,92,96],"row_ids_sha256":"dbeb408023d6d9dce90f39198d7ed83966090c5a0bee3cd5d1963f76e7a1b4dc","oracle":{"probs":[0.9813706278800964,0.011255777440965176,0.00737359281629324],"logits_raw":[4.334125995635986,-6.416270732879639,-7.4339728355407715],"argmax":0,"top2_margin":0.9701148271560669,"near_tie":false},"graph":{"p":[0.980529248714447,0.011800856329500675,0.007670019753277302],"max_abs_dp":0.0008413791656494141,"argmax_equal":true}}]},{"id":"tv4sh_05","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":653,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0012","pair_id":"v3-legacy-transfer-deadline-0012","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0012","id":"contrastive/deadline/v3-legacy-transfer-deadline-0012/b","text_sha256":"a056b9ce266919cbb722895aa4d4432cc469b06fc27ebd0c677238d34200fb8e","row_sha256":"dfcb3fdbcf88be8c45cb2a71d511d19a58245a30c3e8fb7fd1bdc39e0f1fca92","group_id":"v3-legacy-transfer-deadline-0012","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"score","n_options":3,"row_len":99,"state_len":78,"decide":98,"opts":[88,93,97],"row_ids_sha256":"49ca4e78390fb7a24ddb9500c7d5ad9c9b844cdfeee77aeb5cac3439f8688a82","oracle":{"probs":[0.0022679311223328114,0.023175371810793877,0.9745566844940186],"logits_raw":[-11.398874282836914,-5.806680679321289,3.189281940460205],"argmax":2,"top2_margin":0.9513813257217407,"near_tie":false},"graph":{"p":[0.002298543695360422,0.023506276309490204,0.9741952419281006],"max_abs_dp":0.00036144256591796875,"argmax_equal":true}}]},{"id":"tv4sh_06","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":654,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0013","pair_id":"v3-legacy-transfer-deadline-0013","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0013","id":"contrastive/deadline/v3-legacy-transfer-deadline-0013/a","text_sha256":"1707ca65b05430e955da5abed9f42eb596b77c27a3e3f8a81d11d63a11235117","row_sha256":"ba18be7789b6139d2981db43e91aa5fa49ee0341070ef987283988963ebedc51","group_id":"v3-legacy-transfer-deadline-0013","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"score","n_options":3,"row_len":99,"state_len":78,"decide":98,"opts":[88,93,97],"row_ids_sha256":"320d1c99271903fc2301e6f2ad4b2a30b6afca89e958dd4b0e47e744f87979a6","oracle":{"probs":[0.3939671218395233,0.5534920692443848,0.05254078656435013],"logits_raw":[-2.9731130599975586,-2.155104160308838,-7.820528030395508],"argmax":1,"top2_margin":0.15952494740486145,"near_tie":false},"graph":{"p":[0.3932313323020935,0.5539569854736328,0.05281171575188637],"max_abs_dp":0.0007357895374298096,"argmax_equal":true}}]},{"id":"tv4sh_07","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":655,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0013","pair_id":"v3-legacy-transfer-deadline-0013","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0013","id":"contrastive/deadline/v3-legacy-transfer-deadline-0013/b","text_sha256":"f39da0354802898bbde918103b780c20c08561afa8524949e67d37c58f3aaffa","row_sha256":"6f39642be84beb8ea612a1be593599dfd6a81c4b0165a317b7f88580adb32118","group_id":"v3-legacy-transfer-deadline-0013","variant":"clean"}},"state_tokens":79,"questions":[{"q":0,"type":"score","n_options":3,"row_len":100,"state_len":79,"decide":99,"opts":[89,94,98],"row_ids_sha256":"a08a4b3c7af1acf85846ce5f0a75f41476d50fa5ff04c7a2e47c5e97c1d6e26e","oracle":{"probs":[0.012814337387681007,0.5985918641090393,0.38859376311302185],"logits_raw":[-11.269744873046875,-2.0208523273468018,-3.060375452041626],"argmax":1,"top2_margin":0.20999810099601746,"near_tie":false},"graph":{"p":[0.013051517307758331,0.5980434417724609,0.38890501856803894],"max_abs_dp":0.0005484223365783691,"argmax_equal":true}}]},{"id":"tv4sh_08","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":656,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0014","pair_id":"v3-legacy-transfer-deadline-0014","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0014","id":"contrastive/deadline/v3-legacy-transfer-deadline-0014/a","text_sha256":"fc9977e9dd9c23766d739280ab1085eaf5d91e7be1a3ec9e0092f7bfa4ebbf6c","row_sha256":"a4d8941e13b548ecf801d5ef8154ecb8944e0879b85db56e725f5c5df8d8ad2d","group_id":"v3-legacy-transfer-deadline-0014","variant":"clean"}},"state_tokens":76,"questions":[{"q":0,"type":"score","n_options":3,"row_len":97,"state_len":76,"decide":96,"opts":[86,91,95],"row_ids_sha256":"2cd6e39749616e04786b843e2aa24923d067c38348bc9ccef145523066d9bcc9","oracle":{"probs":[0.3177372217178345,0.6537267565727234,0.0285359974950552],"logits_raw":[-2.3441739082336426,-0.6082936525344849,-8.142894744873047],"argmax":1,"top2_margin":0.3359895348548889,"near_tie":false},"graph":{"p":[0.311102032661438,0.6602389812469482,0.028658954426646233],"max_abs_dp":0.006635189056396484,"argmax_equal":true}}]},{"id":"tv4sh_09","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":657,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0014","pair_id":"v3-legacy-transfer-deadline-0014","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0014","id":"contrastive/deadline/v3-legacy-transfer-deadline-0014/b","text_sha256":"e0777025a5b0c26270b3e5955629fbef0413904eeebd12540cde582b64449ec7","row_sha256":"5754f6c7211511e9a1a3dd1dcb09e6a325ed9facc94a9300fc4033c0bf4e37a1","group_id":"v3-legacy-transfer-deadline-0014","variant":"clean"}},"state_tokens":77,"questions":[{"q":0,"type":"score","n_options":3,"row_len":98,"state_len":77,"decide":97,"opts":[87,92,96],"row_ids_sha256":"220faf9d8a906f6b9216e914b5ef41c8865964b2cd5f684bd636509458b2543e","oracle":{"probs":[0.01322946883738041,0.3513040244579315,0.635466456413269],"logits_raw":[-11.438356399536133,-3.548424482345581,-2.1223411560058594],"argmax":2,"top2_margin":0.2841624319553375,"near_tie":false},"graph":{"p":[0.013326405547559261,0.352905809879303,0.6337677240371704],"max_abs_dp":0.0016987323760986328,"argmax_equal":true}}]},{"id":"tv4sh_10","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":658,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0015","pair_id":"v3-legacy-transfer-deadline-0015","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0015","id":"contrastive/deadline/v3-legacy-transfer-deadline-0015/a","text_sha256":"e698380e12ef3e684f4bbf2bfb49e21b88c13030cc4305f39172002bf75a2581","row_sha256":"ca1846312146ab4b7ede6c0348eb01ccae4a9e484bcd8c355a8f8497414e0479","group_id":"v3-legacy-transfer-deadline-0015","variant":"clean"}},"state_tokens":77,"questions":[{"q":0,"type":"score","n_options":3,"row_len":98,"state_len":77,"decide":97,"opts":[87,92,96],"row_ids_sha256":"f5c09da9e045ad3204189d11090ff985be4338e03f6f91b21dc0e843f1dca93e","oracle":{"probs":[0.9753585457801819,0.013239768333733082,0.011401768773794174],"logits_raw":[3.1373376846313477,-7.207667350769043,-7.567266941070557],"argmax":0,"top2_margin":0.9621188044548035,"near_tie":false},"graph":{"p":[0.9748657941818237,0.013459441252052784,0.011674805544316769],"max_abs_dp":0.0004927515983581543,"argmax_equal":true}}]},{"id":"tv4sh_11","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":659,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0015","pair_id":"v3-legacy-transfer-deadline-0015","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0015","id":"contrastive/deadline/v3-legacy-transfer-deadline-0015/b","text_sha256":"7d4d32bb0f70babb9e567683e98f6cfb068535585c37276488434bba5c61d5e1","row_sha256":"90ebc7661f64c68aef69fa8feba55e02ac0c081b22af11b0fb6a33f1770a81fe","group_id":"v3-legacy-transfer-deadline-0015","variant":"clean"}},"state_tokens":76,"questions":[{"q":0,"type":"score","n_options":3,"row_len":97,"state_len":76,"decide":96,"opts":[86,91,95],"row_ids_sha256":"6ff82e7b6fc4601c8a6b45019a9bbd941abfd1516a12e1a5e7e3d712977031f3","oracle":{"probs":[0.20461706817150116,0.7669795751571655,0.028403377160429955],"logits_raw":[-4.000775337219238,-0.821613609790802,-8.751838684082031],"argmax":1,"top2_margin":0.5623624920845032,"near_tie":false},"graph":{"p":[0.19525735080242157,0.7765898704528809,0.028152788057923317],"max_abs_dp":0.009610295295715332,"argmax_equal":true}}]},{"id":"tv4sh_12","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":660,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0016","pair_id":"v3-legacy-transfer-deadline-0016","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0016","id":"contrastive/deadline/v3-legacy-transfer-deadline-0016/a","text_sha256":"90947706e3d61df0db4c3c44469df55f90dd95289439b06a1b047f9ecb80d3c4","row_sha256":"4609e1f514173bb6cc53fcef82c8b16f934b5b6c8552661a2e01ebf1569a53ac","group_id":"v3-legacy-transfer-deadline-0016","variant":"clean"}},"state_tokens":77,"questions":[{"q":0,"type":"score","n_options":3,"row_len":98,"state_len":77,"decide":97,"opts":[87,92,96],"row_ids_sha256":"253fa4fe8c1aed51c73107c287c65c4e2b6d60d27989701a924235e0f93c2245","oracle":{"probs":[0.6259174346923828,0.3376424312591553,0.03644013777375221],"logits_raw":[-1.670121192932129,-3.155210018157959,-8.511838912963867],"argmax":0,"top2_margin":0.28827500343322754,"near_tie":false},"graph":{"p":[0.6197444796562195,0.34299543499946594,0.03726009279489517],"max_abs_dp":0.00617295503616333,"argmax_equal":true}}]},{"id":"tv4sh_13","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":661,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0016","pair_id":"v3-legacy-transfer-deadline-0016","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0016","id":"contrastive/deadline/v3-legacy-transfer-deadline-0016/b","text_sha256":"3345c00abfc381ed9e7d49dd4830b6b691edce70625996b261b66e517cac596f","row_sha256":"67d688120c333a429f9c98361e43b62a00c42acbc93dc99b9b2c76b598f999bf","group_id":"v3-legacy-transfer-deadline-0016","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"score","n_options":3,"row_len":99,"state_len":78,"decide":98,"opts":[88,93,97],"row_ids_sha256":"aed63548e31d0f571df9d2148582ab750aa32145ffc3757ae5532bc22f1dc33c","oracle":{"probs":[0.026563065126538277,0.9153996706008911,0.058037206530570984],"logits_raw":[-9.24885368347168,-0.7318239808082581,-7.368374824523926],"argmax":1,"top2_margin":0.8573624491691589,"near_tie":false},"graph":{"p":[0.027062930166721344,0.9147908091545105,0.05814623832702637],"max_abs_dp":0.0006088614463806152,"argmax_equal":true}}]},{"id":"tv4sh_14","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":662,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0017","pair_id":"v3-legacy-transfer-deadline-0017","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0017","id":"contrastive/deadline/v3-legacy-transfer-deadline-0017/a","text_sha256":"e6cf0c98a6e024fc739907f5cb3a3fca844d42b56ab463d46c2033ee7a7eb8a7","row_sha256":"cd426821d0d22f03f1de40e3dadb4d88c16109b010c7620270dd516f0bf0b93b","group_id":"v3-legacy-transfer-deadline-0017","variant":"clean"}},"state_tokens":77,"questions":[{"q":0,"type":"score","n_options":3,"row_len":98,"state_len":77,"decide":97,"opts":[87,92,96],"row_ids_sha256":"915b33ebcb051648cf137260eb7d18a82004ed145bb36bcab2275f1a4be55028","oracle":{"probs":[0.9944979548454285,0.0026517047081142664,0.0028503956273198128],"logits_raw":[4.620553016662598,-9.640191078186035,-9.466341018676758],"argmax":0,"top2_margin":0.9916475415229797,"near_tie":false},"graph":{"p":[0.9944317936897278,0.002676584990695119,0.0028915968723595142],"max_abs_dp":6.61611557006836e-05,"argmax_equal":true}}]},{"id":"tv4sh_15","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":663,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0017","pair_id":"v3-legacy-transfer-deadline-0017","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0017","id":"contrastive/deadline/v3-legacy-transfer-deadline-0017/b","text_sha256":"04ab254f4aeb306a27d84d8abc28714ad898019447b8e87ba41f63e5ac185ca7","row_sha256":"623b716f5719dd85aa3dbccab13695554b8131b3380aefb8c3511b23f26b244f","group_id":"v3-legacy-transfer-deadline-0017","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"score","n_options":3,"row_len":99,"state_len":78,"decide":98,"opts":[88,93,97],"row_ids_sha256":"05dbb5c9ea574ea7befbf93b6eda5b2572312dc41c052e328ed12af25ea6d9a6","oracle":{"probs":[0.023730741813778877,0.9597370028495789,0.01653231680393219],"logits_raw":[-7.434896469116211,1.4672186374664307,-8.304573059082031],"argmax":1,"top2_margin":0.9360062479972839,"near_tie":false},"graph":{"p":[0.02354281209409237,0.9596729874610901,0.016784265637397766],"max_abs_dp":0.00025194883346557617,"argmax_equal":true}}]},{"id":"tv4sh_16","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":664,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0018","pair_id":"v3-legacy-transfer-deadline-0018","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0018","id":"contrastive/deadline/v3-legacy-transfer-deadline-0018/a","text_sha256":"3159faf45f6f6cbd83e78bec26087c97cc2a53e96923f3e8543e0d16d8bea505","row_sha256":"648c0e4938dc08cf8bc819191e4105d3f39788b83d126e5b9ffe25c8e44634d7","group_id":"v3-legacy-transfer-deadline-0018","variant":"clean"}},"state_tokens":76,"questions":[{"q":0,"type":"score","n_options":3,"row_len":97,"state_len":76,"decide":96,"opts":[86,91,95],"row_ids_sha256":"21cf7c1dcc3e55b38ba9a555c69103b1521d68bdaca7ea6e0df864fd02e1e462","oracle":{"probs":[0.37730979919433594,0.552089512348175,0.07060069590806961],"logits_raw":[-3.6925230026245117,-2.7766757011413574,-7.725127220153809],"argmax":1,"top2_margin":0.1747797131538391,"near_tie":false},"graph":{"p":[0.3726317584514618,0.5559471249580383,0.07142119854688644],"max_abs_dp":0.0046780407428741455,"argmax_equal":true}}]},{"id":"tv4sh_17","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":665,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0018","pair_id":"v3-legacy-transfer-deadline-0018","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0018","id":"contrastive/deadline/v3-legacy-transfer-deadline-0018/b","text_sha256":"7e53a08ce33836290f3ad26ef97343372fcca8d9b3be8ad7fc494d59a4e8c384","row_sha256":"0b0627231d64cb4c8b0ac99d7806214c71ab8700ffc779ca420b6bff1b643064","group_id":"v3-legacy-transfer-deadline-0018","variant":"clean"}},"state_tokens":77,"questions":[{"q":0,"type":"score","n_options":3,"row_len":98,"state_len":77,"decide":97,"opts":[87,92,96],"row_ids_sha256":"5ae200e02f45ce81cbe843ecadf76266956b03d52b48428e294bbb01aa8f1489","oracle":{"probs":[0.01311638206243515,0.8726556301116943,0.11422805488109589],"logits_raw":[-10.516749382019043,-0.416924387216568,-5.30925178527832],"argmax":1,"top2_margin":0.7584275603294373,"near_tie":false},"graph":{"p":[0.013425428420305252,0.8717189431190491,0.1148555725812912],"max_abs_dp":0.0009366869926452637,"argmax_equal":true}}]},{"id":"tv4sh_18","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":666,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0019","pair_id":"v3-legacy-transfer-deadline-0019","sibling":"a","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0019","id":"contrastive/deadline/v3-legacy-transfer-deadline-0019/a","text_sha256":"498d90fc103eca25d611d80676bdcda336cd1ce79841f5d14213f7dacae8c71e","row_sha256":"3fd388a669406a32788e1db619b49c1bf72ebb426c0bb62cb61377d0db98092a","group_id":"v3-legacy-transfer-deadline-0019","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"score","n_options":3,"row_len":99,"state_len":78,"decide":98,"opts":[88,93,97],"row_ids_sha256":"196d39cd43b9b9fc49a158029b65f877089c29fb27359a465f79dc05f89dd05d","oracle":{"probs":[0.3676154613494873,0.5919274687767029,0.040457021445035934],"logits_raw":[-2.894029140472412,-1.747915267944336,-8.203693389892578],"argmax":1,"top2_margin":0.22431200742721558,"near_tie":false},"graph":{"p":[0.3658868670463562,0.5935103893280029,0.04060278460383415],"max_abs_dp":0.0017285943031311035,"argmax_equal":true}}]},{"id":"tv4sh_19","source":"transfer_v4_dev_score_heldout","reference":{"file":"evals/v4/transfer-v4/development.jsonl","line":667,"_meta":{"source":"legacy_holdout","family":"deadline","family_id":"deadline/v3-legacy-transfer-deadline-0019","pair_id":"v3-legacy-transfer-deadline-0019","sibling":"b","repo":null,"revision":null,"split":"generated","row":"v3-legacy-transfer-deadline-0019","id":"contrastive/deadline/v3-legacy-transfer-deadline-0019/b","text_sha256":"c9e31ed73d47b668f7343567a07269bca02606c680b8c6ebbbcb35055b8edfa7","row_sha256":"72b880321dae0e88d1ebc8287fa4a9aab2c344959fb6913f203eb27acacf8a60","group_id":"v3-legacy-transfer-deadline-0019","variant":"clean"}},"state_tokens":78,"questions":[{"q":0,"type":"score","n_options":3,"row_len":99,"state_len":78,"decide":98,"opts":[88,93,97],"row_ids_sha256":"52fc26f1039462f21a91f6d691760b6a017eac88287120127375544fae709af6","oracle":{"probs":[0.05134841054677963,0.5638529062271118,0.38479870557785034],"logits_raw":[-7.185816764831543,-1.4205377101898193,-2.339824676513672],"argmax":1,"top2_margin":0.17905420064926147,"near_tie":false},"graph":{"p":[0.053191009908914566,0.5632940530776978,0.3835149109363556],"max_abs_dp":0.0018425993621349335,"argmax_equal":true}}]}]},"summary":{"records":384,"questions":434,"by_source":{"transfer_v4_dev_head":{"records":60,"questions":60},"transfer_v4_dev_sources":{"records":140,"questions":140},"transfer_v4_dev_score":{"records":20,"questions":20},"semif_authored144":{"records":144,"questions":144},"own_text":{"records":11,"questions":40},"own_json":{"records":6,"questions":19},"own_long":{"records":3,"questions":11}},"near_ties":["tv4_021:q0","tv4_054:q0","tv4x_emotion_15:q0","tv4x_tweet_offensive_01:q0","tv4x_paws_01:q0","tv4s_11:q0","semif_ffee861474a9665a093a:q0","semif_2708193212d8a4a523c7:q0","semif_4089c199b8b796ec3ed8:q0","own_t01:q2","own_t02:q0","own_j05:q2"],"published_text_records":155,"withheld_records":9,"withheld_questions":28,"heldout_records":130,"heldout_questions":130,"heldout_near_ties":["tv4h_01:q0","tv4h_07:q0","tv4h_20:q0","tv4xh_emotion_02:q0"],"graph_gate":{"fixture":{"questions_non_near_tie":422,"argmax_equal_non_near_tie":422,"near_tie_questions":12,"argmax_equal_near_tie":10,"max_abs_dp":0.015306353569030762,"mean_of_run_mean_abs_dp":0.000768624044211272,"worst_run":{"id":"semif_1105577da4c8dad4609d","q":0,"max_abs_dp":0.015306353569030762}},"heldout":{"questions_non_near_tie":126,"argmax_equal_non_near_tie":126,"near_tie_questions":4,"argmax_equal_near_tie":3,"max_abs_dp":0.009610295295715332,"mean_of_run_mean_abs_dp":0.0006897504849677298,"worst_run":{"id":"tv4sh_11","q":0,"max_abs_dp":0.009610295295715332}}}},"trimmed":{"from_lane_files":{"fixtures/records.json":{"bytes":574316,"sha256":"a8ee72eb06fc7522b5f719d01b53e3686c902536f5bcda3071835dc9ac425969"},"fixtures/heldout.json":{"bytes":209207,"sha256":"8ec7ca4aa0e5be3a5261f03302d7e9df6cc78fae5ae368ca9db50b7b4921c6df"},"fixtures/LICENSE-SemIf":{"bytes":1270,"sha256":"010ae478823c7746be2c9659ef9fa7c98973f6632a4e8f6ecceb95abb6fb639d"},"oracle_4b/records_oracle.json":{"bytes":1376621,"sha256":"eb3031df83a75b746beec3480a4bbb47787e13f8a12412ff135feb9f609c70f9"},"oracle_4b/heldout/records_oracle.json":{"bytes":364887,"sha256":"dfa101a559450a8b75c140bafd120728561273908a527ad89ce44d5558d31c59"},"results/r12_fixture_K128_4b.json":{"bytes":669060,"sha256":"41dca9cbed8d58bf331480137dee62f212449769c2e723b2f7d79ca7f43ab6b4"},"results/r12_heldout_K128_4b.json":{"bytes":204280,"sha256":"96f097809ee73d5898cdd64614c05033c253a4012175e15a1d818046585d1738"},"exports/ship/kev_4b_decode_fp16_metal_pf128/metadata.json":{"bytes":8766,"sha256":"dcdd0ea1c4ea4369282176db2f937779aab00b944b4f26034e9314b192d88e87"}},"removed":["the transfer-v4 records' requests, rendered options, instructions and gold labels","every row's token ids (row_ids_sha256 = sha256 of the ids as little-endian int32 keeps them checkable)","the oracle's logits_T (= logits_raw / T) and the hidden-state dumps","the withheld records' text, keys, question ids and gold labels"],"serialization":"compact json (no indentation)"}}