{"schema":"coreai-clef-flash-fixtures/1","what":"The author's fp32 readout of Cloudflare/clef-flash (its own joint_schema_model.py: load_release_model() + systemone(), CPU, float32) on 216 records: 186 written for the port (round 1) and 30 written later as a held-out set. Every run carries its ids, spans, option logits and probabilities, the response, and the oracle's own checks. Images are named and hashed, not included.","source":{"hf_id":"Cloudflare/clef-flash","revision":"17f0b0ad64efb65d273590632833508766b2aae6","joint_schema_model_sha256":"0e304cf7c6500e8bb59bef7e2afd2c6373f82596dfb3b57d1aa93c175e2dc3a3","sha256":{"config.json":"66f87f6fb2616b46604daf2a9c67ddc87938296d07156efa34d59b5be49e3238","joint_head_config.json":"77efe959a38b5b17b241543e129e695f3c77465ece55a25985279bd8176279a0","processor_config.json":"d89ef49ce9cd37fbf510158e13c1ef063d9286411c1ec9049932dbe0487143b1","tokenizer.json":"06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523","tokenizer_config.json":"91a08f825d370d085d692e04cf117cdd7faad7bf18e996f1e6031b6dab03db72","model.safetensors.index.json":"941305ff9f77551e145a6cea976ef456cd5cb208cbece99f168376c752fcf96c"},"oracle":"joint_schema_model.load_release_model(device='cpu', dtype=float32) + systemone(), torch.inference_mode","versions":{"python":"3.12.11","torch":"2.9.0","torchvision":"0.24.0","transformers":"5.17.0","pillow":"12.3.0","numpy":"2.5.3","huggingface_hub":"1.33.0"},"device":"cpu","dtype":"float32","attn_implementation":{"text":"sdpa","vision":"sdpa"}},"contract":{"layout":"ids = prefix + [image block] + state + schema + suffix; every piece tokenized on its own without special tokens, then concatenated (the checkpoint's encode_record())","prefix":{"text":"<|im_start|>system\nRead the complete state and schema. Decide every field jointly. Each answer must be exactly one of that field's allowed options.<|im_end|>\n<|im_start|>user\nSTATE:\n","ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198],"n":36},"image_block":{"at_index":36,"form":"[vision_start 248053] + [image_pad 248056] * N + [vision_end 248054] + ['\\n' id]","note":"one image per record in this fixture; N = 64 at g256, 196 at g448, the processor's own count at native"},"state":"render(state): a string as is, anything else json.dumps(sort_keys=True, ensure_ascii=False, separators=(',', ':'))","schema_header":{"text":"\n\nSCHEMA FIELDS:\n","ids":[271,3423,34651,426,32349,25,198]},"field":"'\\nFIELD {k}\\nID: {id}\\nTYPE: {type}\\nINSTRUCTION: ' + render(instructions or id) [question span] + '\\nALLOWED OPTIONS:\\n' + per option j: 'OPTION {j}: ' + render({'option_id': id, 'description': desc}) [option span] + '\\n', then 'END FIELD\\n'","field_example":{"text":"\nFIELD 1\nID: department\nTYPE: choice\nINSTRUCTION: ","ids":[198,27806,220,16,198,887,25,9016,198,9222,25,5576,198,669,55530,25,220]},"suffix":{"text":"\n<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\nJOINT SCHEMA DECISIONS:","ids":[198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"n":18},"last_token":{"id":25,"text":":"},"special_ids":{"<|endoftext|>":248044,"<|im_start|>":248045,"<|im_end|>":248046,"<|vision_start|>":248053,"<|vision_end|>":248054,"<|image_pad|>":248056,"<|video_pad|>":248057,"<think>":248068,"</think>":248069},"spans":"question_span / option_spans are [start, end) token indices into ids (image block as N <|image_pad|>)","option_order":{"noul":["true","false"],"choice":"option ids sorted as strings","score":"levels 0..n-1"},"noul_default_criteria":{"true":"The proposition is true or the answer is yes.","false":"The proposition is false or the answer is no."},"softmax":"per question over that question's options, float32, temperature 1","response":"SystemOne-compatible: {model, answers: {question_id: answer}, usage: {input_tokens, output_tokens: 0}}; noul -> p(true); choice -> argmax id, its p, all p; score -> sum(level * p), max p, legend, all p; 4 decimals","mrope_closed_form":"P = len(prefix ids) = token_offset; i <= P -> (i,i,i); image token k at P+1+k -> (P+1, P+1+k//W, P+1+k%W); i >= P+N+1 -> i-(N-max(H,W)) on all planes; text-only -> i","near_tie_threshold":0.02},"arms":{"text":"no image","g256":"PIL BICUBIC resize to 256x256 before the processor","g448":"PIL BICUBIC resize to 448x448 before the processor","native":"the image as drawn","note":"the held-out set runs text, g256 and g448 only"},"processor":{"processor_class":"Qwen3VLProcessor","image_processor_class":"Qwen2VLImageProcessor","image_processor_mro":["Qwen2VLImageProcessor","TorchvisionBackend","BaseImageProcessor","ImageProcessingMixin"],"tokenizer_class":"Qwen2Tokenizer","resample":3,"patch_size":16,"merge_size":2,"temporal_patch_size":2,"image_mean":[0.5,0.5,0.5],"image_std":[0.5,0.5,0.5],"size":{"shortest_edge":65536,"longest_edge":16777216},"pad_token_id":248044},"sources":{"own_text":"written for this fixture (invented people, organisations, products and places)","own_text_long":"written / generated for this fixture; state about 1,800-2,200 tokens","own_json":"written for this fixture (invented)","own_image":"images drawn by make_fixtures.py (CC0-1.0); text written for this fixture","cc0_photo":"Wikimedia Commons photographs whose extmetadata.LicenseShortName is CC0 (checked at fetch time); 960 px thumbnails, sha256 pinned in PHOTOS; text written for this fixture","heldout_text":"written for the held-out set (invented people, organisations, products and places)","heldout_json":"written for the held-out set (invented)","heldout_text_long":"written for the held-out set; state 1,800-2,200 tokens","heldout_image":"images drawn by make_fixtures.py (CC0-1.0); text written for the held-out set","heldout_photo":"Wikimedia Commons photographs whose extmetadata.LicenseShortName is CC0 (checked at fetch time); 960 px thumbnails, sha256 pinned in HELDOUT_PHOTOS; text written for the held-out set","semif_authored144":{"upstream_repo":"github.com/TheoLeeCJ/SemIf","upstream_path":"benchmarks/data/authored144.jsonl","upstream_commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","rows":144,"mapping":"state -> state; question -> questions['decision'].instructions; options -> criteria {id: description}; type choice; gold = options[label].id","license":"MIT","license_file":"LICENSE at the same commit","license_file_sha256":"f765f2140f8507a8f0d81ec0fd2c4bd72fe6a066841ef27883ff876a76bf61be","license_notice":"MIT License\n\nCopyright (c) 2026 TheoLeeCJ\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n","what_is_redistributed":"the 144 rows' state, question and option texts, rewritten into the request shape above (records with source 'semif_authored144'); the MIT copyright and permission notice above goes with them"}},"images_license":"drawn images: CC0-1.0, regenerated byte for byte by conversion/clef_flash/make_fixtures.py; photographs: Wikimedia Commons files whose LicenseShortName is CC0 (file page, thumbnail URL and the thumbnail's sha256 per image)","images":[{"set":"fixture","name":"shapes_224","file":"shapes_224.png","size":[224,224],"png_sha256":"2c3a0fdc1f33cbff227afe6ad87506d16495bdcae46ed7993577eb4737506646","rgb_sha256":"9850fbbc51f5ffe0ecf9a55882797be8593c92c6ac9382d1a304c8b435f295fe","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"shapes_224","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py"}},{"set":"fixture","name":"bars_640x480","file":"bars_640x480.png","size":[640,480],"png_sha256":"cd9b0091dce5b4343e954eca551ba7d65a55e90ab501878f4a0e99f62e47748f","rgb_sha256":"4508cd2c2e41ef976db78a072243f2983cae5b1dc60d8bc5e144a63a2399e2a0","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"bars_640x480","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py"}},{"set":"fixture","name":"sevenseg_47_300x200","file":"sevenseg_47_300x200.png","size":[300,200],"png_sha256":"7cab564fbc9f3535713eb684340cfcd6085235205b8f294bde19df0c05dcfea4","rgb_sha256":"7ccf718af6607993557d277e738ee04a4a63275b190bea60ffeb4fd5ad6f0668","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"sevenseg_47_300x200","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py"}},{"set":"fixture","name":"receipt_1024x768","file":"receipt_1024x768.png","size":[1024,768],"png_sha256":"861051aabe8c4095dfad12de46f148e1846c1e5aac406fba8cff794c608d18b1","rgb_sha256":"75189abdc87409b306767827de9549528c4b1963d34900d13acf8d55e7e9e171","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"receipt_1024x768","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py"}},{"set":"fixture","name":"signal_green_256x240","file":"signal_green_256x240.png","size":[256,240],"png_sha256":"b9217027de1efc9745a5bb046d1119264fa7a7c45a0ebc5dbec9703c057024d7","rgb_sha256":"4c1f50c0b79d52bf68c896343c13ae67dc00ef8847fd7af979f98db6cbe1e7cb","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"signal_green_256x240","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py"}},{"set":"fixture","name":"gauge_300x200","file":"gauge_300x200.png","size":[300,200],"png_sha256":"417865041130f0ea34ab93dc9c92e5b14c5d8e3d28ff940d642ff50c50721b3c","rgb_sha256":"530538e733f26c9638f8d92f4c3e79576b2a3416fdbc7235eaa5ccff1ea6716a","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"gauge_300x200","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py"}},{"set":"fixture","name":"arrows_640x480","file":"arrows_640x480.png","size":[640,480],"png_sha256":"21ebbb411b11d64cdc7298abba3834535e39542c6a0c6c7e9a664e14dbc36e68","rgb_sha256":"544d801aae244a5bafd73e3fd7cef620e534d0adac02a5ffa9090ec683ed5db7","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"arrows_640x480","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py"}},{"set":"fixture","name":"floorplan_1024x768","file":"floorplan_1024x768.png","size":[1024,768],"png_sha256":"b7de171107d917969505cfc3bdd7980f87d9b6adc30cda281acc45e4c8e32b86","rgb_sha256":"2c0fc45eb1b83c911ac0d16eac97ddabd639432730ceef8c22ce66b77fb29627","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"floorplan_1024x768","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py"}},{"set":"fixture","name":"circles_640x480","file":"circles_640x480.png","size":[640,480],"png_sha256":"21a8ee718172007e86dec31aa4b6fa1bd1454413eb1472cd3477ab9f8acb99ae","rgb_sha256":"81350119b4dc8bb8f890d12417d1d77b42dd5537348e369a267b3000fdbd0a23","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"circles_640x480","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py"}},{"set":"fixture","name":"clock_3oclock_256x240","file":"clock_3oclock_256x240.png","size":[256,240],"png_sha256":"6b8b1ce2d8aaccf2684b2af0c6c4da5bc824194cc2ce94daf32bbb236eff2c86","rgb_sha256":"d9b8e9b9db338d04ad993ea11e437230fb9eda7199168fbc00c6a922dface8e3","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"clock_3oclock_256x240","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py"}},{"set":"fixture","name":"photo_fruit","file":"photo_fruit.png","size":[960,1280],"png_sha256":"f3cfb976fc6f10dff6fd94ef55efdb49596ee2a7bcba761f3d304488f6bc6bcb","rgb_sha256":"c3c4456ffed1d4cb16db626b669a5ba806be827cd11b50b22648778142fd6caf","license":"CC0-1.0","license_note":"Wikimedia Commons extmetadata.LicenseShortName = CC0","source":{"commons_title":"File:Fruits in bowl oranges lime apples (1).jpg","file_page":"https://commons.wikimedia.org/wiki/File:Fruits_in_bowl_oranges_lime_apples_(1).jpg","thumb_url":"https://thumb.wikimedia.org/wikipedia/commons/thumb/8/8f/Fruits_in_bowl_oranges_lime_apples_%281%29.jpg/960px-Fruits_in_bowl_oranges_lime_apples_%281%29.jpg","thumb_jpeg_sha256":"e781f1deefe098097204409fa0b97fe06da8dac96f403a1f6186a2fdddb92331","decode":"Pillow JPEG decode -> RGB -> PNG (the raw-RGB sha256 is the Pillow build's decode)"}},{"set":"fixture","name":"photo_cat","file":"photo_cat.png","size":[960,640],"png_sha256":"3b25c6ce53f98cb305bca403d53a6a38fe0be2feff70281047e231ffc4ae2351","rgb_sha256":"39270067b2a47a6585a72b13fb74435cdad28ec22bf9cb66e3c741f1ec57e328","license":"CC0-1.0","license_note":"Wikimedia Commons extmetadata.LicenseShortName = CC0","source":{"commons_title":"File:Tabby cat-3337027.jpg","file_page":"https://commons.wikimedia.org/wiki/File:Tabby_cat-3337027.jpg","thumb_url":"https://thumb.wikimedia.org/wikipedia/commons/thumb/9/9e/Tabby_cat-3337027.jpg/960px-Tabby_cat-3337027.jpg","thumb_jpeg_sha256":"242708bd1a489a885acebeec4a0f83d15d2551b9b210ddfd08daee11f82923bf","decode":"Pillow JPEG decode -> RGB -> PNG (the raw-RGB sha256 is the Pillow build's decode)"}},{"set":"fixture","name":"photo_lake","file":"photo_lake.png","size":[960,640],"png_sha256":"583071d0a921d93e49b46fc7fae1ae4a0600440ed5fa39231885783db891ea96","rgb_sha256":"1fd3457bf939842969277dd37065f57013f117c299ceb81fdbc1eeecd39fcd97","license":"CC0-1.0","license_note":"Wikimedia Commons extmetadata.LicenseShortName = CC0","source":{"commons_title":"File:Lake Mountain Landscape.jpg","file_page":"https://commons.wikimedia.org/wiki/File:Lake_Mountain_Landscape.jpg","thumb_url":"https://thumb.wikimedia.org/wikipedia/commons/thumb/7/75/Lake_Mountain_Landscape.jpg/960px-Lake_Mountain_Landscape.jpg","thumb_jpeg_sha256":"cf5e281fab8eb63ba4335067d8a69fe81d9217fac4a2365e9655146d7cfa35d4","decode":"Pillow JPEG decode -> RGB -> PNG (the raw-RGB sha256 is the Pillow build's decode)"}},{"set":"fixture","name":"photo_coffee","file":"photo_coffee.png","size":[960,635],"png_sha256":"20aab0bebbd2e2b438448e971a619c1cb9582ed515063f978258f2f0478455a0","rgb_sha256":"fddbe96e887af4ce3ee2100f9eceb2e27dbcf95a7910d628a9d9911cd88e3227","license":"CC0-1.0","license_note":"Wikimedia Commons extmetadata.LicenseShortName = CC0","source":{"commons_title":"File:Coffee cup and coffee bean.jpg","file_page":"https://commons.wikimedia.org/wiki/File:Coffee_cup_and_coffee_bean.jpg","thumb_url":"https://thumb.wikimedia.org/wikipedia/commons/thumb/3/33/Coffee_cup_and_coffee_bean.jpg/960px-Coffee_cup_and_coffee_bean.jpg","thumb_jpeg_sha256":"672b86815fae409f073df0ce6f0411f9a908e5251b01fa0c058c7b9c6eadb32c","decode":"Pillow JPEG decode -> RGB -> PNG (the raw-RGB sha256 is the Pillow build's decode)"}},{"set":"heldout","name":"ho_pie_720x480","file":"ho_pie_720x480.png","size":[720,480],"png_sha256":"66372ebfd2b973d77b5d4d4ffe40b202193add2da4c1fc9180839e7086308bf8","rgb_sha256":"5a673ca6f9868bd19fc0c25a668e3b217b99cf3c08768c4bed4c689e423ce7e5","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"ho_pie_720x480","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py --heldout"}},{"set":"heldout","name":"ho_table_1000x600","file":"ho_table_1000x600.png","size":[1000,600],"png_sha256":"1a4b8afdd5f1f0a1d1f731b6cba6e5687c3f2e3463bab02f2ed8a5e8a641dca4","rgb_sha256":"db3f1f258048316796f4cea238822dc019c0761d5dee3ebdfb34a9d5b4dd16a2","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"ho_table_1000x600","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py --heldout"}},{"set":"heldout","name":"ho_sign_200x150","file":"ho_sign_200x150.png","size":[200,150],"png_sha256":"bd4976cb0d9e7b55c01fcccf8133d363789d78bab64387e6e0e707093a68739f","rgb_sha256":"3b3f1103f20b17cf5672147ea8eb84feba6142509b9ea114d2c7c7f561056412","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"ho_sign_200x150","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py --heldout"}},{"set":"heldout","name":"ho_thermometer_160x400","file":"ho_thermometer_160x400.png","size":[160,400],"png_sha256":"a98841445775dec79f7a5c5d9b26adfc4b860ebfbb27a35ef7e99ea3e8158af5","rgb_sha256":"5d7ef8fefa38f572826b76b71199a6439d47c067d89fe2a22643883a5d33f331","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"ho_thermometer_160x400","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py --heldout"}},{"set":"heldout","name":"ho_calendar_760x700","file":"ho_calendar_760x700.png","size":[760,700],"png_sha256":"848c45a664e2ec1b85c79839dbdd8e169fa5ac97f59995326709e49a271aeffa","rgb_sha256":"3901e1a9f9e56e8e4578f5fff9cb9c907b6be5dfba00bda013d583b7e2e9fcb5","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"ho_calendar_760x700","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py --heldout"}},{"set":"heldout","name":"ho_digits_960x240","file":"ho_digits_960x240.png","size":[960,240],"png_sha256":"678381b01d5c9078aa84d00d9bf1a1ea01993893e592f26ee9414c526564df4e","rgb_sha256":"e1a514f4f61dc13749437a90d09f51ff92e229e0f302dc612ac271cb3eb104ec","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"ho_digits_960x240","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py --heldout"}},{"set":"heldout","name":"ho_receipt_480x840","file":"ho_receipt_480x840.png","size":[480,840],"png_sha256":"f133cf246c3ac79b4c9cac3a2c47ef7539cdf7493db0e71557cc8625e59b6cd8","rgb_sha256":"0d53e251ec2daa5e9de988488e60e7be9267790e3d11aa2584ba0c922e1a915e","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"ho_receipt_480x840","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py --heldout"}},{"set":"heldout","name":"ho_floorplan_1200x900","file":"ho_floorplan_1200x900.png","size":[1200,900],"png_sha256":"c54ffd41a76d476b237e15fe08c8c64b664c68ce46e2c93b1ea497a7a72cbba1","rgb_sha256":"3b8e33ad51486772916b4310362891c385f3e6b6a75783b9cc7c6c71663759fd","license":"CC0-1.0","license_note":"self-made: drawn by conversion/clef_flash/make_fixtures.py with Pillow primitives","generator":{"script":"conversion/clef_flash/make_fixtures.py","function":"ho_floorplan_1200x900","args":{},"how":"python3 conversion/clef_flash/make_fixtures.py --heldout"}},{"set":"heldout","name":"ho_photo_lighthouse","file":"ho_photo_lighthouse.png","size":[960,960],"png_sha256":"0a9f1baab17840622b823f700a3e538333f66c22e90c1148044d302a5ddbaf72","rgb_sha256":"388cf96d5b6a65a6dccc66c1f1c71d883582dae17c7f713522081375db82ac4c","license":"CC0-1.0","license_note":"Wikimedia Commons extmetadata.LicenseShortName = CC0","source":{"commons_title":"File:Cap de Barbaria lighthouse.jpg","file_page":"https://commons.wikimedia.org/wiki/File:Cap_de_Barbaria_lighthouse.jpg","thumb_url":"https://thumb.wikimedia.org/wikipedia/commons/thumb/a/aa/Cap_de_Barbaria_lighthouse.jpg/960px-Cap_de_Barbaria_lighthouse.jpg","thumb_jpeg_sha256":"6daf0cb176e6150a97ba138b0374c282df8ad0d15e80a47431a8f1287e8b0253","decode":"Pillow JPEG decode -> RGB -> PNG (the raw-RGB sha256 is the Pillow build's decode)"}},{"set":"heldout","name":"ho_photo_sunflowers","file":"ho_photo_sunflowers.png","size":[960,1280],"png_sha256":"ac2994ef2cf71a6175a03b1c9d82ddf3dd1f1f1b99e5ec156f299f8b2a9977df","rgb_sha256":"cfc036b97dbf7464d13502af44841e7923d670dbdb11ce9ca19d6c1fbce9e6a3","license":"CC0-1.0","license_note":"Wikimedia Commons extmetadata.LicenseShortName = CC0","source":{"commons_title":"File:Helianthus × laetiflorus flower (05).jpeg","file_page":"https://commons.wikimedia.org/wiki/File:Helianthus_%C3%97_laetiflorus_flower_(05).jpeg","thumb_url":"https://thumb.wikimedia.org/wikipedia/commons/thumb/1/18/Helianthus_%C3%97_laetiflorus_flower_%2805%29.jpeg/960px-Helianthus_%C3%97_laetiflorus_flower_%2805%29.jpeg","thumb_jpeg_sha256":"83ad983d60ace81885d370ee84fc820bb771d1124d5340dc0563443e0bfd88dd","decode":"Pillow JPEG decode -> RGB -> PNG (the raw-RGB sha256 is the Pillow build's decode)"}}],"names_screen":{"fixture":{"method":"dig +short <stem>.<tld> A for tld in com, net, io, co.uk, app; a stem is used only when none resolves","date":"2026-10-03","stems_used":["ashmerrow","brindlecourt","cradlewick","fennistone","harrowfen","ostlebury","pennimere","thistlecombe","wendlecote"],"stems_rejected_resolving":["quillmoor","pellwick","orrinvale","saltreach","tarnwick","wrenmoor","ferrylane","marrowgate","wimbleford","larkhollow","sorrelby","dunmarra","velloway","kettlemoor","oakhatch","tamberlyn","grisdale"],"checked":"checked 2026-10-03 by the supervisor after publication: exact-phrase web search (35 stems, 0 hits), DNS A records (.com/.net/.io/.co.uk/.app, none resolving), App Store search API (0), Companies House register (0); record: lane results/name_screen_appstore_companieshouse.json"},"heldout":{"method":"dig +short <stem>.<tld> A for tld in com, net, io, co.uk, app; a stem is used only when none resolves","date":"2026-10-03","stems_used":["bramblequist","brackwenna","calderwisp","cobbleswick","corvassen","drossingham","ellerwyke","emberdowne","fernacombe","glenvarrow","grallowmere","harlowick","hollindrake","kinnerwold","lindenquay","ostrevale","ottervane","pendrithy","pottersmire","quorrendale","sallowmead","skelmardine","tolverack","tumbrelby","vintlecombe","yarrowdeep"],"stems_rejected_resolving":["quibbleton"],"checked":"checked 2026-10-03 by the supervisor after publication: exact-phrase web search (35 stems, 0 hits), DNS A records (.com/.net/.io/.co.uk/.app, none resolving), App Store search API (0), Companies House register (0); record: lane results/name_screen_appstore_companieshouse.json"},"note":"invented names only; the DNS screen is what was run"},"heldout":{"description":"Held-out confirmation set, written on 2026-10-03 after round 4 by a separate step, to re-check a quantization choice that was tuned on the round-1 fixtures (fixtures/records.json). No earlier step has seen it: every text, JSON state, person and organisation name, drawn image and photograph is new, and novelty_check asserts that no record id, image name or pixel hash, person name, invented name stem, identical state or instruction, or 8-word shingle is shared with the round-1 files (see `novelty`). Use it only to confirm a decision already made: anything tuned or selected on it spends it, and a new held-out set is then needed.","novelty":{"compared_with":{"records_json":"fixtures/records.json","records_json_sha256":"84e24684b5ce29e8da7582a23acf83c923a02fc874681c9247c735ddaaadecbc","images_meta_json":"fixtures/images_meta.json","images_meta_json_sha256":"e630446ee9564b1ed76f43cb6be5d5680c482a925748f8b489cf633352bd5bbb","records":186,"images":14},"method":{"text":"per record: the state (a string as is, anything else compact JSON with sorted keys), each instructions and each option description or score level, as separate pieces","words":"lowercased runs of letters and digits; punctuation, underscores and spaces separate words","shingle_words":8,"people":"every part of every HELDOUT_PEOPLE name, whole word, case-insensitive, in the round-1 records.json","stems":"HELDOUT_NAMES_SCREEN stems as substrings of the round-1 records.json and of its names screen; round-1 stems (used and rejected) as substrings of the held-out requests","statements":"identical states or instructions after the word rule, any length"},"heldout":{"records":30,"images":10,"shingles":4716,"people":27,"stems":26},"round1_shingles":6152,"overlaps":{"record_ids":0,"image_names":0,"image_png_sha256":0,"image_rgb_sha256":0,"person_names":0,"stems":0,"identical_states_or_instructions":0,"shingles_8_words":0}}},"records":[{"id":"own_t01","set":"fixture","source":"own_text","request":{"model":"clef-flash","state":"Ticket 4471 from Mara Ellison: Since this morning the DL-40 desk lamp I bought from Wendlecote Home will not turn on. I tried two different outlets and a different cable. It is still within the 90-day warranty. I would like it repaired or replaced, not refunded.","questions":{"department":{"type":"choice","instructions":"Which team should handle this ticket?","criteria":{"billing":"Payments, invoices, charges and refunds","technical":"A product or service does not work: bugs, faults, outages","shipping":"Delivery, tracking, lost or damaged parcels","account":"Sign-in, profile and password problems"}},"wants_refund":{"type":"noul","instructions":"The customer asks for a refund."},"urgency":{"type":"score","instructions":"How urgent is the ticket?","criteria":["Can wait","This week","Today","Immediately"]}}},"gold":{"department":"technical","wants_refund":"false","urgency":null},"note":"support ticket","state_tokens":64},{"id":"own_t02","set":"fixture","source":"own_text","request":{"model":"clef-flash","state":"Incident 2026-117. At 02:14 the checkout service of the Ashmerrow web store began returning HTTP 503 for about 40% of requests. The on-call engineer, Ravi Mendes, rolled back release 8.4.1 at 02:39, and the error rate was back to normal by 02:46. No data loss was found. Root cause: the database connection-pool limit had been lowered from 200 to 20 in the new release.","questions":{"severity":{"type":"score","instructions":"How severe was the incident?","criteria":["SEV5 cosmetic","SEV4 minor","SEV3 moderate","SEV2 major","SEV1 critical"]},"resolved":{"type":"noul","instructions":"The incident is resolved."},"root_cause":{"type":"choice","instructions":"What caused the incident?","criteria":{"config_change":"A configuration value changed in a release","hardware":"A hardware failure","third_party":"An outage at an external provider","attack":"Malicious traffic","capacity":"Organic traffic exceeded capacity"}},"data_loss":{"type":"noul","instructions":"Was any data lost?","criteria":{"true":"Some customer or order data was lost or corrupted.","false":"No data was lost or corrupted."}}}},"gold":{"severity":null,"resolved":"true","root_cause":"config_change","data_loss":"false"},"note":"incident report; noul criteria override (both keys)","state_tokens":119},{"id":"own_t03","set":"fixture","source":"own_text","request":{"model":"clef-flash","state":"From: Ines Valcourt\nTo: Theo Brandt\nSubject: Re: Thursday site visit\n\nTheo, the inspector from Fennistone Testing can no longer come on Thursday. She offered Monday at 9:00 or Tuesday at 14:00 instead. Our loading bay is closed all day on Monday for resurfacing, so she could not get the equipment in. Can you confirm which slot works for you?\n\n> From: Theo Brandt\n> Ines, please confirm the Thursday 10:00 inspection is still on.","questions":{"new_slot":{"type":"choice","instructions":"Which slot should be booked?","criteria":{"mon_0900":"Monday at 9:00","tue_1400":"Tuesday at 14:00","thu_1000":"Thursday at 10:00, as originally planned"}},"thursday_still_on":{"type":"noul","instructions":"The Thursday inspection still takes place."},"reply_needed":{"type":"noul"}}},"gold":{"new_slot":"tue_1400","thursday_still_on":"false","reply_needed":"true"},"note":"email thread; one question without instructions","state_tokens":117},{"id":"own_t04","set":"fixture","source":"own_text","request":{"model":"clef-flash","state":"Weekly sync, design team. Attendees: Noor, Felix, Abebe. Decisions: ship the dark-mode toggle in v2.3; postpone the onboarding redesign to the first quarter. Action items: Felix drafts the release note by Friday; Abebe fixes the contrast bug on the settings page. Open question: whether to keep the beta label on the toggle.","questions":{"dark_mode":{"type":"choice","instructions":"What was decided about the dark-mode toggle?","criteria":{"ship_v23":"Ship it in v2.3","postpone":"Postpone it","cancel":"Cancel it"}},"release_note_owner":{"type":"choice","instructions":"Who drafts the release note?","criteria":{"noor":"Noor","felix":"Felix","abebe":"Abebe"}},"open_items":{"type":"score","instructions":"How many questions are still open?","criteria":["None","One","Several"]},"beta_label_decided":{"type":"noul","instructions":"The team decided whether to keep the beta label."}}},"gold":{"dark_mode":"ship_v23","release_note_owner":"felix","open_items":"1","beta_label_decided":"false"},"note":"meeting notes","state_tokens":76},{"id":"own_t05","set":"fixture","source":"own_text","request":{"model":"clef-flash","state":"Rated 2 out of 5. The KT-3 kettle looks great and boils fast, but after three weeks the lid hinge cracked, and once the automatic shut-off did not trigger, which is a real safety problem. Customer support replied within a day and offered a replacement.","questions":{"sentiment":{"type":"score","instructions":"What is the overall sentiment of the review?","criteria":["very negative","negative","mixed","positive","very positive"]},"safety_issue":{"type":"noul","instructions":"The review reports a safety problem."},"main_complaint":{"type":"choice","instructions":"Which aspect is the main complaint?","criteria":{"durability":{"covers":"parts breaking or wearing out","example":"a hinge snapped after a month"},"speed":{"covers":"how fast the product works","example":"boils in two minutes"},"design":{"covers":"looks and form","example":"a matte finish that shows fingerprints"},"support":{"covers":"the customer service experience","example":"an agent who never replied"}}},"recommend":{"type":"noul","instructions":"Would the reviewer recommend the kettle?","criteria":{"true":"The reviewer would recommend the product to others."}}}},"gold":{"sentiment":"1","safety_issue":"true","main_complaint":"durability","recommend":"false"},"note":"product review; option descriptions are JSON objects; noul override of 'true' only","state_tokens":56},{"id":"own_t06","set":"fixture","source":"own_text","request":{"model":"clef-flash","state":"[09:02] Pia: is the staging database down for anyone else?\n[09:03] Omar: yes, timeouts since about 08:50\n[09:05] Pia: production looks fine\n[09:07] Omar: restarted the staging replica, it's back up\n[09:08] Pia: thanks! running the migration test again","questions":{"environment":{"type":"choice","instructions":"Which environment was affected?","criteria":{"production":"Production only","staging":"Staging only","both":"Production and staging","neither":"No environment"}},"fixed":{"type":"noul","instructions":"The problem was fixed during the conversation."},"urgency":{"type":"score","criteria":["low","medium","high"]}}},"gold":{"environment":"staging","fixed":"true","urgency":null},"note":"chat log; score question without instructions","state_tokens":88},{"id":"own_t07","set":"fixture","source":"own_text","request":{"model":"clef-flash","state":"Can we find 30 minutes next week for the budget review? I'm out on Monday and on Wednesday afternoon. Jonas can only do mornings. Priya is travelling on Thursday and Friday. - Lena","questions":{"best_slot":{"type":"choice","instructions":"Which slot works for everyone?","criteria":{"mon_am":"Monday morning","tue_am":"Tuesday morning","tue_pm":"Tuesday afternoon","wed_pm":"Wednesday afternoon","thu_am":"Thursday morning","fri_am":"Friday morning"}},"length":{"type":"choice","instructions":"How long should the meeting be?","criteria":{"15":"15 minutes","30":"30 minutes","60":"one hour","90":"90 minutes"}},"needs_travel":{"type":"noul","instructions":"Someone has to travel to attend the meeting."}}},"gold":{"best_slot":"tue_am","length":"30","needs_travel":"false"},"note":"scheduling","state_tokens":41},{"id":"own_t08","set":"fixture","source":"own_text","request":{"model":"clef-flash","state":"Overnight oats for two: 1 cup rolled oats, 1 cup milk, half a cup of yogurt, 2 tablespoons chia seeds, 1 tablespoon honey. Stir, cover and refrigerate for at least 6 hours. Top with berries before serving. Contains dairy. Can be made vegan with oat milk, soy yogurt and maple syrup.","questions":{"diet":{"type":"choice","instructions":"As written, which diet does the recipe fit?","criteria":{"vegan":"Vegan: no animal products","vegetarian":"Vegetarian: no meat or fish, may contain dairy or honey","contains_meat":"Contains meat or fish"}},"needs_cooking":{"type":"noul","instructions":"The dish needs heat to prepare."},"effort":{"type":"score","instructions":"How much preparation effort does it take?","criteria":["minimal","some","a lot"]},"servings":{"type":"choice","instructions":"How many servings does the recipe make?","criteria":{"1":"one","2":"two","3":"three","4":"four","5":"five","6":"six","7":"seven","8":"eight","9":"nine","10":"ten"}}}},"gold":{"diet":"vegetarian","needs_cooking":"false","effort":"0","servings":"2"},"note":"recipe; a 10-option choice (ids '1'..'10' sort as strings)","state_tokens":70},{"id":"own_t09","set":"fixture","source":"own_text","request":{"model":"clef-flash","state":"Hi, I was charged twice for my March subscription: two charges of 14.99 on March 3. Please refund the duplicate charge. This is the second time this has happened and I'm starting to lose patience. Thanks, Ari Feldmann","questions":{"department":{"type":"choice","instructions":"Which team should handle this message?","criteria":{"billing":"Payments, invoices, charges and refunds","technical":"A product or service does not work: bugs, faults, outages","shipping":"Delivery, tracking, lost or damaged parcels","account":"Sign-in, profile and password problems"}},"duplicate_charge":{"type":"noul","instructions":"The customer was charged more than once for the same thing."},"refund_amount":{"type":"choice","instructions":"How much should be refunded?","criteria":{"0":"nothing","14.99":"one charge of 14.99","29.98":"both charges"}},"frustration":{"type":"score","instructions":"How frustrated is the customer?","criteria":["calm","slightly annoyed","annoyed","frustrated","very frustrated","angry","furious"]}}},"gold":{"department":"billing","duplicate_charge":"true","refund_amount":"14.99","frustration":null},"note":"billing ticket; a 7-level score","state_tokens":52},{"id":"own_t10","set":"fixture","source":"own_text","request":{"model":"clef-flash","state":"Request from Malik Osei (warehouse, night shift): I need to swap my Saturday 22:00-06:00 shift on the 14th with Dara Lindqvist, who has agreed. We are both certified forklift operators. I am asking two weeks in advance, as the policy requires.","questions":{"meets_conditions":{"type":"noul","instructions":"The shift swap meets the stated conditions."},"policy_issue":{"type":"choice","instructions":"Is there a policy problem with the request?","criteria":{"none":"No policy problem","notice":"Not enough notice","certification":"A missing certification","consent":"The other worker has not agreed"}},"review":{"type":"score","instructions":"Does the request need a manager's review?","criteria":["can be approved","needs review"]}}},"gold":{"meets_conditions":"true","policy_issue":"none","review":"0"},"note":"shift request; a 2-level score","state_tokens":68},{"id":"own_t11","set":"fixture","source":"own_text","request":{"model":"clef-flash","state":"The vendor's security questionnaire says customer files are encrypted at rest with AES-256 and in transit with TLS 1.3, backups are kept for 35 days, and staff access requires hardware keys. Penetration tests are run yearly; the most recent report is 14 months old.","questions":{"encrypted_at_rest":{"type":"noul","instructions":"Customer files are encrypted at rest."},"pentest_current":{"type":"noul","instructions":"Is the penetration test current?","criteria":{"true":"A penetration test was done within the last 12 months.","false":"The last penetration test is older than 12 months, or its date is unknown."}},"risk":{"type":"score","instructions":"How risky is the vendor overall?","criteria":["low","moderate","elevated","high"]},"next_step":{"type":"choice","instructions":"What should happen next?","criteria":{"approve":"Approve the vendor as is","request_pentest":"Ask for a new penetration test before approving","reject":"Reject the vendor"}}}},"gold":{"encrypted_at_rest":"true","pentest_current":"false","risk":null,"next_step":"request_pentest"},"note":"security questionnaire summary; noul override (both keys)","state_tokens":62},{"id":"own_t12","set":"fixture","source":"own_text","request":{"model":"clef-flash","state":"Order note from table 12: two adults and one child (age 6). Adults: one vegetarian lasagne, and one grilled trout without butter (dairy allergy). Child: plain pasta, no sauce. Drinks: one sparkling water, one lemonade, one orange juice. They asked for the bill to be split in two and want the dessert menu after the mains. Please put a birthday candle on the child's dessert.","questions":{"dairy_allergy":{"type":"noul","instructions":"Someone at the table has a dairy allergy."},"guests":{"type":"choice","instructions":"How many people are at the table?","criteria":{"1":"one","2":"two","3":"three","4":"four","5":"five","6":"six"}},"split_bill":{"type":"noul","instructions":"The bill should be split."},"split_ways":{"type":"choice","instructions":"Into how many parts should the bill be split?","criteria":{"2":"two","3":"three","4":"four"}},"occasion":{"type":"choice","instructions":"What is the occasion?","criteria":{"none":"No special occasion","birthday":"A birthday","anniversary":"An anniversary","business":"A business meal","graduation":"A graduation"}},"kitchen_effort":{"type":"score","instructions":"How demanding is the order for the kitchen?","criteria":["trivial","easy","moderate","involved","complex"]},"dessert_planned":{"type":"noul","instructions":"The guests plan to order dessert."},"child_meal":{"type":"choice","instructions":"What does the child eat?","criteria":{"plain_pasta":"Plain pasta","lasagne":"Vegetarian lasagne","trout":"Grilled trout","burger":"A burger"}}}},"gold":{"dairy_allergy":"true","guests":"3","split_bill":"true","split_ways":"2","occasion":"birthday","kitchen_effort":null,"dessert_planned":"true","child_meal":"plain_pasta"},"note":"restaurant order; the 8-question record","state_tokens":88},{"id":"own_t13","set":"fixture","source":"own_text","request":{"model":"clef-flash","state":"Notice: the staff car park at Brindlecourt is closed for repairs from June 2 to June 6. Staff may park in the east lot; visitors should use the street. The badge readers at the side entrance will be offline on June 3.","questions":{"car_park_open_june_4":{"type":"noul"},"street_parking":{"type":"choice","instructions":"Who should park on the street?","criteria":{"staff":"Staff","visitors":"Visitors","everyone":"Everyone"}},"disruption":{"type":"score","instructions":"How disruptive is the closure?","criteria":["minor","moderate","major"]}}},"gold":{"car_park_open_june_4":"false","street_parking":"visitors","disruption":null},"note":"notice; noul without instructions","state_tokens":53},{"id":"own_t14","set":"fixture","source":"own_text_long","request":{"model":"clef-flash","state":"2026-09-18T03:00:04.094Z INFO  orders-api      request_id=r-10238 POST /orders status=201 latency_ms=77\n2026-09-18T03:01:06.185Z INFO  payments-gateway request_id=r-10245 POST /authorize status=200 latency_ms=177 pool_in_use=13\n2026-09-18T03:02:08.276Z INFO  orders-api      request_id=r-10252 POST /orders status=201 latency_ms=69\n2026-09-18T03:03:10.367Z INFO  inventory-svc   request_id=r-10259 GET /stock/BOLT-M6 status=200 latency_ms=41 cache=hit\n2026-09-18T03:04:12.458Z INFO  orders-api      request_id=r-10266 POST /orders status=201 latency_ms=78\n2026-09-18T03:05:14.549Z INFO  payments-gateway request_id=r-10273 POST /authorize status=200 latency_ms=214 pool_in_use=17\n2026-09-18T03:06:16.640Z INFO  orders-api      request_id=r-10280 POST /orders status=201 latency_ms=80\n2026-09-18T03:07:18.731Z INFO  inventory-svc   request_id=r-10287 GET /stock/BOLT-M6 status=200 latency_ms=50 cache=hit\n2026-09-18T03:09:02.410Z WARN  payments-gateway p95_latency_ms=1210 threshold_ms=800 pool_in_use=176 pool_size=200\n2026-09-18T03:09:30.822Z INFO  orders-api      request_id=r-10294 POST /orders status=201 latency_ms=1002\n2026-09-18T03:09:44.913Z INFO  orders-api      request_id=r-10301 POST /orders status=201 latency_ms=942\n2026-09-18T03:11:52.018Z WARN  payments-gateway connection pool saturated pool_in_use=200 pool_size=200 waiting=37\n2026-09-18T03:12:07.233Z ERROR payments-gateway upstream connect error: connection refused (card-processor pool exhausted) attempt=1/3\n2026-09-18T03:12:09.004Z ERROR orders-api      request_id=r-10308 POST /orders status=502 error=payment_authorization_failed upstream=payments-gateway\n2026-09-18T03:12:39.095Z ERROR payments-gateway request_id=r-10315 POST /authorize status=503 error=upstream_unavailable attempt=2/3\n2026-09-18T03:13:09.186Z WARN  orders-api      request_id=r-10322 retry scheduled backoff_ms=750 reason=payment_authorization_failed\n2026-09-18T03:13:39.277Z ERROR orders-api      request_id=r-10329 POST /orders status=502 error=payment_authorization_failed upstream=payments-gateway\n2026-09-18T03:14:09.368Z ERROR payments-gateway request_id=r-10336 POST /authorize status=503 error=upstream_unavailable attempt=2/3\n2026-09-18T03:14:10.005Z INFO  notify-worker   queued 41 customer e-mails template=payment_delayed\n2026-09-18T03:14:39.459Z WARN  orders-api      request_id=r-10343 retry scheduled backoff_ms=500 reason=payment_authorization_failed\n2026-09-18T03:15:09.550Z ERROR orders-api      request_id=r-10350 POST /orders status=502 error=payment_authorization_failed upstream=payments-gateway\n2026-09-18T03:15:10.077Z INFO  alerting        page sent to on-call rotation=payments severity=high\n2026-09-18T03:15:39.641Z ERROR payments-gateway request_id=r-10357 POST /authorize status=503 error=upstream_unavailable attempt=2/3\n2026-09-18T03:16:09.732Z WARN  orders-api      request_id=r-10364 retry scheduled backoff_ms=250 reason=payment_authorization_failed\n2026-09-18T03:16:39.823Z ERROR orders-api      request_id=r-10371 POST /orders status=502 error=payment_authorization_failed upstream=payments-gateway\n2026-09-18T03:17:30.440Z INFO  payments-gateway restart requested by on-call config_change=pool_size:200->400\n2026-09-18T03:18:41.902Z INFO  payments-gateway process started version=5.12.3 pool_size=400\n2026-09-18T03:19:02.116Z INFO  payments-gateway health check passed upstream=card-processor latency_ms=88\n2026-09-18T03:19:10.914Z INFO  orders-api      request_id=r-10378 POST /orders status=201 latency_ms=85\n2026-09-18T03:19:50.005Z INFO  orders-api      request_id=r-10385 POST /orders status=201 latency_ms=92\n2026-09-18T03:20:30.096Z INFO  payments-gateway request_id=r-10392 POST /authorize status=200 latency_ms=100 pool_in_use=32\n2026-09-18T03:21:10.187Z INFO  orders-api      request_id=r-10399 POST /orders status=201 latency_ms=40\n2026-09-18T03:21:50.278Z INFO  orders-api      request_id=r-10406 retry succeeded original_status=502 attempts=2\n2026-09-18T03:22:30.369Z INFO  orders-api      request_id=r-10413 POST /orders status=201 latency_ms=50\n2026-09-18T03:24:48.300Z INFO  orders-api      backlog drained: 312 orders re-authorized, 0 failed permanently\n2026-09-18T03:25:03.012Z INFO  notify-worker   queued 41 customer e-mails template=payment_completed","questions":{"first_failing_component":{"type":"choice","instructions":"Which component failed first?","criteria":{"auth":"The authentication service","database":"The primary database","payments":"The payments gateway","inventory":"The inventory service","notifications":"The notification worker"}},"recovered":{"type":"noul","instructions":"By the end of the log the failing component has recovered."},"severity":{"type":"score","instructions":"How severe was the event for customers?","criteria":["informational","minor","major","critical"]},"error_window":{"type":"choice","instructions":"How long did the errors last?","criteria":{"under_1m":"Under one minute","1_to_5m":"One to five minutes","5_to_15m":"Five to fifteen minutes","over_15m":"More than fifteen minutes"}}}},"gold":{"first_failing_component":"payments","recovered":"true","severity":null,"error_window":"5_to_15m"},"note":"long state: service log (generated deterministically)","state_tokens":1893},{"id":"own_t15","set":"fixture","source":"own_text_long","request":{"model":"clef-flash","state":"MASTER SERVICES AGREEMENT (excerpt: Sections 1 to 16)\n\nThis Master Services Agreement (the \"Agreement\") is entered into on 4 March 2026 (the \"Effective Date\") between Harrowfen Field Services Ltd. (\"Provider\") and Ostlebury Growers Cooperative (\"Client\"). Provider and Client are each a \"Party\" and together the \"Parties\".\n\n1. Definitions\n1.1 \"Business Day\" means a day other than a Saturday, a Sunday or a public holiday at the relevant Client site.\n1.2 \"Business Hours\" means 08:00 to 18:00 on a Business Day.\n1.3 \"Confidential Information\" means any information disclosed by one Party to the other that is marked as confidential or that a reasonable person would understand to be confidential.\n\n2. Services\n2.1 Provider will supply the irrigation monitoring services described in Schedule A (the \"Services\"), including the installation of soil-moisture sensors at the Client's sites, the operation of the hosted monitoring dashboard, and second-line technical support during Business Hours.\n2.2 Provider may change the way it delivers the Services, provided that the change does not materially reduce their functionality or security. Provider will give Client at least thirty (30) days' written notice of any such change.\n\n3. Term and Renewal\n3.1 The Agreement starts on the Effective Date and continues for an initial term of twenty-four (24) months (the \"Initial Term\").\n3.2 After the Initial Term, the Agreement renews automatically for successive periods of twelve (12) months (each a \"Renewal Term\") unless either Party gives written notice of non-renewal at least ninety (90) days before the end of the then-current term.\n3.3 Provider may increase the fees for a Renewal Term by no more than five percent (5%) over the fees for the preceding twelve months, provided Provider notifies Client of the increase at least one hundred and twenty (120) days before the Renewal Term begins.\n\n4. Fees and Payment\n4.1 Client will pay the monthly fees set out in Schedule B. Provider will invoice monthly in arrears.\n4.2 Invoices are payable within thirty (30) days of the invoice date. Amounts not paid when due bear interest at one and a half percent (1.5%) per month until paid.\n4.3 Client may dispute an invoice in good faith by written notice within fifteen (15) days of receipt, stating its reasons. The Parties will try to resolve the dispute within thirty (30) days. Undisputed portions remain payable on the original due date.\n4.4 If any undisputed amount remains unpaid for more than forty-five (45) days after its due date, Provider may suspend the Services after giving ten (10) days' written notice.\n\n5. Service Levels\n5.1 Provider will make the monitoring dashboard available 99.5% of the time in each calendar month, excluding scheduled maintenance that is announced at least five (5) days in advance and limited to eight (8) hours per month.\n5.2 If availability falls below 99.5% in a month, Client is entitled to a service credit of five percent (5%) of that month's fees for each full 0.5% below the target, up to a maximum of twenty-five percent (25%) of that month's fees.\n5.3 Service credits are Client's sole remedy for a failure to meet the availability target, except where availability falls below 95% in three consecutive months, in which case Client may terminate under Section 10.2.\n\n6. Client Responsibilities\n6.1 Client will give Provider's technicians safe access to each site during Business Hours, with at least two (2) Business Days' notice.\n6.2 Client is responsible for the accuracy of the site information it provides and for protecting the sensors from physical damage caused by its own staff or contractors.\n\n7. Confidentiality\n7.1 Each Party will keep the other Party's Confidential Information secret and use it only to perform or receive the Services.\n7.2 These obligations survive for three (3) years after the Agreement ends, except for trade secrets, which remain protected for as long as they remain trade secrets.\n\n8. Data\n8.1 Client owns all sensor readings and reports generated for its sites (\"Client Data\"). Provider may use Client Data only to provide the Services and, in an aggregated form that does not identify Client or any site, to improve its products.\n8.2 On termination, Provider will make Client Data available for export for sixty (60) days and will then delete it, unless the law requires Provider to keep it.\n\n9. Liability\n9.1 Neither Party is liable for indirect or consequential loss, including loss of crops, profit or revenue, even if that loss was foreseeable.\n9.2 Each Party's total liability arising out of or in connection with this Agreement in any twelve-month period is limited to the total fees paid or payable by Client in the twelve (12) months before the event giving rise to the claim.\n9.3 The limits in this Section do not apply to liability for death or personal injury caused by negligence, for fraud, for breach of Section 7 (Confidentiality), or for gross negligence or wilful misconduct.\n\n10. Termination\n10.1 Either Party may terminate the Agreement by written notice if the other Party commits a material breach and fails to remedy it within thirty (30) days after receiving written notice describing the breach.\n10.2 Client may terminate the Agreement with immediate effect by written notice in the circumstances described in Section 5.3.\n10.3 Either Party may terminate the Agreement immediately if the other Party becomes insolvent or ceases to carry on business.\n10.4 Termination does not affect fees already due, and Sections 7, 8.2, 9 and 12 survive termination.\n\n11. Assignment\n11.1 Neither Party may assign or transfer this Agreement without the prior written consent of the other Party, which may not be unreasonably withheld.\n11.2 Provider may, however, assign the Agreement to an affiliate without consent by giving written notice to Client, provided that the affiliate is able to perform the Services to the same standard.\n\n12. Notices and Governing Law\n12.1 Notices must be in writing and delivered by hand, by courier, or by e-mail to the addresses in Schedule C. A notice sent by e-mail is received on the next Business Day after it is sent.\n12.2 This Agreement is governed by the laws of the jurisdiction in which Provider is registered, and the courts of that jurisdiction have exclusive jurisdiction over any dispute.\n\n13. Insurance\n13.1 Provider will maintain public liability insurance of at least two million (2,000,000) in the currency of Schedule B and professional indemnity insurance of at least one million (1,000,000) for the duration of the Agreement and for one (1) year afterwards.\n13.2 On request, Provider will give Client a certificate of insurance showing that the cover required by this Section is in force.\n\n14. Force Majeure\n14.1 Neither Party is liable for a delay or failure to perform caused by events beyond its reasonable control, including flood, drought, fire, storm, epidemic, war, or a failure of public utilities, provided that it notifies the other Party promptly and uses reasonable efforts to resume performance.\n14.2 If such an event prevents performance for more than sixty (60) consecutive days, either Party may terminate the Agreement by written notice without liability.\n\n15. Subcontracting and Audit\n15.1 Provider may use subcontractors to install or maintain sensors, provided that Provider remains responsible for their work as if it were its own and that each subcontractor is bound by confidentiality obligations no less protective than Section 7.\n15.2 Once in each contract year, on at least twenty (20) Business Days' written notice, Client may audit Provider's records relating to the Services and to the handling of Client Data. Each Party bears its own costs of the audit, unless the audit reveals a material breach by Provider, in which case Provider will reimburse Client's reasonable audit costs.\n\n16. Entire Agreement\n16.1 This Agreement, together with its Schedules, is the entire agreement between the Parties about its subject matter and replaces all earlier proposals and understandings.\n16.2 No amendment is effective unless it is in writing and signed by authorised representatives of both Parties.","questions":{"non_renewal_notice":{"type":"choice","instructions":"How much notice is needed to stop the automatic renewal?","criteria":{"30":"30 days","60":"60 days","90":"90 days","120":"120 days","none":"No notice is needed"}},"auto_renews":{"type":"noul","instructions":"The agreement renews automatically after the initial term."},"liability_cap":{"type":"choice","instructions":"How is each party's liability capped?","criteria":{"unlimited":"There is no cap","fees_12m":"The fees paid or payable in the 12 months before the claim","fixed_amount":"A fixed amount stated in the agreement","insurance":"The amount of the provider's insurance cover"}},"assign_without_consent":{"type":"noul","instructions":"Either party may assign the agreement freely without the other's consent."},"client_risk":{"type":"score","instructions":"How favourable are the terms to the client?","criteria":["very unfavourable","unfavourable","balanced","favourable","very favourable"]},"uptime_target":{"type":"choice","instructions":"What availability does the provider commit to?","criteria":{"99.0":"99.0%","99.5":"99.5%","99.9":"99.9%","99.99":"99.99%"}}}},"gold":{"non_renewal_notice":"90","auto_renews":"true","liability_cap":"fees_12m","assign_without_consent":"false","client_risk":null,"uptime_target":"99.5"},"note":"long state: contract excerpt","state_tokens":1836},{"id":"own_t16","set":"fixture","source":"own_text_long","request":{"model":"clef-flash","state":"THISTLECOMBE ALLOTMENT ASSOCIATION\nMinutes of the Annual General Meeting, held on Saturday 20 September 2026 at 10:00 in the community hall.\n\nPresent: 34 members (the quorum is 25). Apologies: 6 members. In the chair: Margaret Ollerton. Minutes taken by Desmond Achebe-Lund (secretary).\n\n1. Welcome and apologies\nThe chair opened the meeting at 10:04, welcomed four new plot holders and read the apologies. She reminded members that only members whose fees for 2026 are paid may vote; the secretary confirmed that all 34 members present are entitled to vote.\n\n2. Minutes of the previous AGM\nThe minutes of the AGM of 14 September 2025 were approved without amendment (proposed by Ruth Penhaligon, seconded by Tomasz Wierzbicki).\n\n3. Treasurer's report\nThe treasurer, Yusuf Kemal, presented the accounts for the year to 31 August 2026. Income was £8,420 (plot fees £7,150, sales at the site shop £1,040, donations £230). Expenditure was £6,980 (water £2,310, insurance £1,120, repairs £1,890, skip hire £780, sundries £880). Reserves at the year end stood at £11,600.\nSeveral members asked why the water bill had risen by 18%. The treasurer explained that most of the increase came from leaks on the north path, where water ran for several days before each leak was found. Members also asked about the shop: sales were up because of the new seed potato order, and the shop committee will repeat it next spring.\nThe accounts were accepted (proposed by Ama Boateng, seconded by Graham Lowther), with one abstention.\n\n4. Water system\nThe main pipe along the north path leaked three times this year, and the standpipe at plot 41 failed in July. The committee presented two options:\n(a) repair the damaged sections of the north run, estimated at about £1,400, with a likely need for further repairs within two or three years;\n(b) replace the whole north run with new polyethylene pipe and add two standpipes near plots 12 and 30, estimated at about £5,900, paid from reserves.\nIn the discussion, some members felt that option (b) would leave the reserves too low if the hall roof needed work. Others pointed out that the repeated leaks were already costing money and that the new standpipes would help older members who now carry water a long way. The treasurer confirmed that reserves would remain above £5,000 after option (b).\nGraham Lowther asked whether the association could apply for a grant from the district council's green spaces fund. The secretary said that the fund reopens in January and usually covers up to half of the cost of capital works for community groups, but that a decision would not come before April, too late for the February work. Members agreed that the committee should still apply, and that any grant received would go back into the reserves.\nRuth Penhaligon asked who would carry out the work. The committee intends to hire a contractor for the trenching and connections, while volunteers will clear the path and reinstate the beds afterwards; plot holders along the north path will be told at least four weeks in advance.\nThe chair put option (b) to the vote: for 21, against 11, abstentions 2. The motion to replace the north run was carried. The work will be scheduled for February, when the water supply is switched off for the winter, and the committee will obtain at least two quotes.\n\n5. Plot fees\nThe committee proposed raising the annual fee for a full plot from £45 to £52, and for a half plot from £25 to £29, from January 2027, to rebuild the reserves after the water works.\nMembers argued that the increase was too large in a single year and that the shop surplus and donations should be used first. A suggestion to raise fees by a smaller amount was not put as a formal motion.\nVote on the committee's proposal: for 12, against 19, abstentions 3. The motion was defeated, and fees remain unchanged for 2027. The committee will instead review the insurance renewal, which may be cheaper with a different broker, and report back at the spring meeting.\n\n6. Waiting list\nThere are 23 people on the waiting list, and the average wait is now 14 months. The committee proposed that full plots which become vacant be split into two half plots where the layout allows, and that plots continue to be offered to the waiting list strictly in order of application. A member suggested giving priority to residents of the village; after discussion the meeting agreed to keep the order of application. Several members said that half plots suit newcomers better, because a full plot is a lot of work in the first year and several new holders gave up within twelve months. Tomasz Wierzbicki offered to mentor new plot holders during their first season, and three other members volunteered to join him. The committee will also ask people on the list once a year whether they still want a plot, so that the list stays accurate.\nThe proposal was agreed by a show of hands, with no votes against.\n\n7. Compost and bonfires\nThe committee proposed that the burning of green waste on site be banned from 1 November 2026, and that compost heaps within two metres of the boundary hedge be replaced by closed compost bins, following complaints from neighbours about smoke and rats. The association will buy ten bins in bulk and sell them to members at cost. Members who cannot compost woody prunings may leave them at the chipping area by the main gate, which will be emptied twice a year.\nThe proposal was agreed with 2 votes against.\n\n8. Election of officers\nMargaret Ollerton announced that she is stepping down as chair after six years. Two nominations had been received: Priyanka Somerfield (proposed by Ruth Penhaligon, seconded by Ama Boateng) and Callum Reyes (proposed by Graham Lowther, seconded by Tomasz Wierzbicki). Both candidates spoke briefly. A ballot was held: Priyanka Somerfield 22 votes, Callum Reyes 12 votes. Priyanka Somerfield was elected chair.\nYusuf Kemal was re-elected treasurer unopposed. Desmond Achebe-Lund was re-elected secretary unopposed. Callum Reyes agreed to join the committee as an ordinary member.\nThe meeting thanked Margaret Ollerton for her years in the chair; she will continue to run the plant sale.\n\n9. Any other business\n(a) Shed security: two sheds were broken into in August and tools were taken. Members were asked to mark their tools and not to leave fuel on site. The committee will price a new gate lock.\n(b) Open day: an open day for local families is planned for Saturday 13 June 2027. Volunteers should give their names to the secretary.\n(c) Plot inspections: the committee will carry out the usual inspections in May and August. Plot holders whose plots are less than half cultivated will receive a reminder letter, and a second letter after four weeks if nothing has changed. Members asked that the inspection criteria be printed on the notice board; the secretary agreed to do so.\n(d) Thanks were recorded to the volunteers who cleared the overgrown plots in the spring and to the members who staff the shop.\n\nAction items\n- Committee: obtain at least two quotes for replacing the north water run, by 30 November 2026.\n- Yusuf Kemal: compare insurance renewal quotes and report at the spring meeting.\n- Desmond Achebe-Lund: write to members on the waiting list about the half-plot scheme, by 31 October 2026.\n- Priyanka Somerfield: order ten closed compost bins, by 15 October 2026.\n- Committee: price a new gate lock, by the next committee meeting.\n\nThe meeting closed at 12:16. The next committee meeting will be held on Tuesday 7 October 2026 at 19:30.","questions":{"water_decision":{"type":"choice","instructions":"What was decided about the water system?","criteria":{"repair":"Repair the damaged sections","replace":"Replace the north run and add standpipes","postpone":"Postpone the decision to next year","no_decision":"No decision was made"}},"fees_increase":{"type":"noul","instructions":"Annual plot fees will increase in 2027."},"new_chair":{"type":"choice","instructions":"Who is the new chair?","criteria":{"ollerton":"Margaret Ollerton","somerfield":"Priyanka Somerfield","reyes":"Callum Reyes","kemal":"Yusuf Kemal"}},"water_vote_consensus":{"type":"score","instructions":"How much agreement was there in the water vote?","criteria":["deeply divided","split","clear majority","unanimous"]},"waiting_list_order":{"type":"choice","instructions":"How will vacant plots be offered?","criteria":{"application_order":"To the waiting list in order of application","lottery":"By drawing lots","local_priority":"With priority for residents of the village"}},"bonfire_ban":{"type":"noul","instructions":"Burning green waste on site will be banned."}}},"gold":{"water_decision":"replace","fees_increase":"false","new_chair":"somerfield","water_vote_consensus":"2","waiting_list_order":"application_order","bonfire_ban":"true"},"note":"long state: minutes of a members' meeting","state_tokens":1811},{"id":"own_j01","set":"fixture","source":"own_json","request":{"model":"clef-flash","state":{"invoice":{"number":"INV-20931","vendor":"Cradlewick Office Supply","issued":"2026-08-02","due":"2026-09-01","currency":"EUR","lines":[{"item":"A4 paper, 5 boxes","amount":112.5},{"item":"toner cartridge","amount":89.0}],"total":201.5,"paid":0.0,"status":"open"},"today":"2026-09-14"},"questions":{"status":{"type":"choice","instructions":"What is the invoice status?","criteria":{"paid":"Paid in full","overdue":"Past the due date and unpaid","open_not_due":"Unpaid but not yet due","draft":"Not sent yet"}},"large":{"type":"noul","instructions":"The invoice total is above 500."},"priority":{"type":"score","instructions":"How soon should it be paid?","criteria":["no rush","this month","this week","today"]}}},"gold":{"status":"overdue","large":"false","priority":null},"note":"invoice","state_tokens":117},{"id":"own_j02","set":"fixture","source":"own_json","request":{"model":"clef-flash","state":{"order_id":"B-55102","items":[{"sku":"MUG-11","qty":2},{"sku":"TEA-04","qty":1}],"shipping":{"method":"express","address_verified":false},"payment":{"status":"authorized"},"customer_note":"Gift - no prices in the box please"},"questions":{"can_ship":{"type":"noul","instructions":"The order can ship now."},"blocker":{"type":"choice","instructions":"What is blocking the order, if anything?","criteria":{"none":"Nothing","payment":"Payment","address":"The shipping address","stock":"Stock"}},"gift":{"type":"noul","instructions":"The order is a gift."},"priority":{"type":"score","instructions":"How should the order be prioritised?","criteria":["normal","high","urgent"]}}},"gold":{"can_ship":"false","blocker":"address","gift":"true","priority":null},"note":"order","state_tokens":68},{"id":"own_j03","set":"fixture","source":"own_json","request":{"model":"clef-flash","state":{"device":"greenhouse-3","unit":"celsius","readings":[{"t":"06:00","temp":14.2,"humidity":81},{"t":"09:00","temp":19.8,"humidity":70},{"t":"12:00","temp":31.5,"humidity":52},{"t":"15:00","temp":33.1,"humidity":47}],"limits":{"temp_max":30,"humidity_min":50}},"questions":{"over_limit":{"type":"noul","instructions":"A reading exceeded a limit."},"first_violation":{"type":"choice","instructions":"When was a limit first exceeded?","criteria":{"06:00":"06:00","09:00":"09:00","12:00":"12:00","15:00":"15:00","never":"Never"}},"trend":{"type":"choice","instructions":"How is the temperature changing?","criteria":{"rising":"Rising","falling":"Falling","flat":"Flat"}},"action":{"type":"score","instructions":"What response is appropriate?","criteria":["no action","keep monitoring","open the vents","emergency cooling"]}}},"gold":{"over_limit":"true","first_violation":"12:00","trend":"rising","action":null},"note":"sensor readings","state_tokens":111},{"id":"own_j04","set":"fixture","source":"own_json","request":{"model":"clef-flash","state":{"event":{"title":"Quarterly planning","start":"2026-10-12T15:00","end":"2026-10-12T16:30","timezone":"UTC+2","organizer":"Hana Ruiz","attendees":[{"name":"Hana Ruiz","response":"accepted"},{"name":"Elio Marsh","response":"declined"},{"name":"Sven Kaur","response":"tentative"},{"name":"Ada Nwosu","response":"no_response"}],"location":"Room 4B","recurrence":null}},"questions":{"duration":{"type":"choice","instructions":"How long is the event?","criteria":{"30":"30 minutes","60":"60 minutes","90":"90 minutes","120":"120 minutes"}},"recurring":{"type":"noul","instructions":"The event repeats."},"accepted":{"type":"choice","instructions":"How many attendees have accepted?","criteria":{"0":"zero","1":"one","2":"two","3":"three","4":"four"}},"attendance_risk":{"type":"score","instructions":"How likely is it that too few people attend?","criteria":["unlikely","possible","likely"]}}},"gold":{"duration":"90","recurring":"false","accepted":"1","attendance_risk":null},"note":"calendar event","state_tokens":121},{"id":"own_j05","set":"fixture","source":"own_json","request":{"model":"clef-flash","state":{"user":{"id":88213,"display_name":"quietfern","created":"2019-04-11","plan":"free","verified_email":true,"two_factor":false,"last_login":"2026-09-30","country":"PT","flags":["password_reused"]}},"questions":{"plan":{"type":"choice","instructions":"Which plan is the user on?","criteria":{"free":"Free","pro":"Pro","team":"Team","enterprise":"Enterprise"}},"two_factor":{"type":"noul","instructions":"Two-factor authentication is enabled."},"account_risk":{"type":"score","instructions":"How at risk is the account?","criteria":["very low","low","medium","high","very high"]},"action":{"type":"choice","instructions":"What should the security team do?","criteria":{"none":"Nothing","prompt_2fa":"Prompt the user to enable two-factor authentication","force_reset":"Force a password reset","suspend":"Suspend the account"}}}},"gold":{"plan":"free","two_factor":"false","account_risk":null,"action":null},"note":"user profile","state_tokens":68},{"id":"own_j06","set":"fixture","source":"own_json","request":{"model":"clef-flash","state":{"error":{"status":429,"code":"rate_limited","message":"Too many requests","retry_after_seconds":30,"request_id":"req_7f3a"},"client":{"requests_last_minute":312,"plan_limit_per_minute":300}},"questions":{"error_class":{"type":"choice","instructions":"What kind of error is this?","criteria":{"auth":"Authentication or permission","rate_limit":"Rate limiting","not_found":"Resource not found","server":"Server fault","validation":"Invalid request","timeout":"Timeout"}},"retryable":{"type":"noul","instructions":"The request can be retried later."},"wait_seconds":{"type":"choice","instructions":"How long should the client wait before retrying?","criteria":{"0":"No wait","30":"30 seconds","300":"5 minutes","3600":"1 hour"}},"client_at_fault":{"type":"noul","instructions":"The client caused the error."},"severity":{"type":"score","instructions":"How severe is the error?","criteria":["low","medium","high"]}}},"gold":{"error_class":"rate_limit","retryable":"true","wait_seconds":"30","client_at_fault":"true","severity":null},"note":"API error payload","state_tokens":57},{"id":"own_j07","set":"fixture","source":"own_json","request":{"model":"clef-flash","state":{"warehouse":"north","items":[{"sku":"BOLT-M6","on_hand":1200,"reorder_point":500},{"sku":"NUT-M6","on_hand":340,"reorder_point":500},{"sku":"WASHER-6","on_hand":0,"reorder_point":300},{"sku":"BRACKET-L","on_hand":75,"reorder_point":50}]},"questions":{"out_of_stock":{"type":"choice","instructions":"Which item is out of stock?","criteria":{"BOLT-M6":"BOLT-M6","NUT-M6":"NUT-M6","WASHER-6":"WASHER-6","BRACKET-L":"BRACKET-L","none":"No item"}},"reorder_count":{"type":"choice","instructions":"How many items are at or below their reorder point?","criteria":{"0":"0","1":"1","2":"2","3":"3","4":"4"}},"urgency":{"type":"score","instructions":"How urgent is restocking?","criteria":["not urgent","soon","urgent","critical"]},"any_below":{"type":"noul","instructions":"At least one item is below its reorder point."}}},"gold":{"out_of_stock":"WASHER-6","reorder_count":"2","urgency":null,"any_below":"true"},"note":"inventory; choice ids are SKUs (mixed case sort)","state_tokens":93},{"id":"own_j08","set":"fixture","source":"own_json","request":{"model":"clef-flash","state":{"service":"billing-api","environment":"production","change_window":false,"diff":[{"key":"replicas","old":6,"new":2},{"key":"log_level","old":"info","new":"debug"},{"key":"db_pool_size","old":50,"new":50},{"key":"feature_flags.new_invoice_pdf","old":false,"new":true}]},"questions":{"risky":{"type":"noul","instructions":"The change is risky to deploy now."},"riskiest_change":{"type":"choice","instructions":"Which change carries the most risk?","criteria":{"replicas":{"key":"replicas","effect":"serving capacity"},"log_level":{"key":"log_level","effect":"log volume and cost"},"db_pool_size":{"key":"db_pool_size","effect":"database connections"},"feature_flag":{"key":"feature_flags.new_invoice_pdf","effect":"a new code path for invoices"}}},"review":{"type":"score","instructions":"What review does the change need?","criteria":["auto-approve","peer review","senior review","block until a change window"]}}},"gold":{"risky":"true","riskiest_change":"replicas","review":null},"note":"config diff; option descriptions are JSON objects","state_tokens":76},{"id":"own_j09","set":"fixture","source":"own_json","request":{"model":"clef-flash","state":{"tracking":"TRK-4410-22","promised_delivery":"2026-09-30","events":[{"time":"2026-09-28 08:10","status":"picked_up","location":"Depot A"},{"time":"2026-09-29 19:40","status":"in_transit","location":"Hub 3"},{"time":"2026-09-30 07:15","status":"exception","detail":"address label unreadable","location":"Hub 3"}]},"questions":{"status":{"type":"choice","instructions":"What is the current status of the parcel?","criteria":{"picked_up":"Picked up","in_transit":"In transit","exception":"Held because of a problem","out_for_delivery":"Out for delivery","delivered":"Delivered"}},"late":{"type":"noul","instructions":"The parcel will miss the promised delivery date."},"customer_action":{"type":"noul","instructions":"The customer needs to do something."},"escalation":{"type":"score","instructions":"How far should this be escalated?","criteria":["no escalation","team lead","carrier liaison"]}}},"gold":{"status":"exception","late":"true","customer_action":null,"escalation":null},"note":"tracking history","state_tokens":129},{"id":"own_j10","set":"fixture","source":"own_json","request":{"model":"clef-flash","state":{"survey":"post-workshop","responses":[{"q":"content","score":5},{"q":"pace","score":2},{"q":"materials","score":4},{"q":"venue","score":3}],"comment":"Great examples, but we rushed through the last two sections."},"questions":{"weakest":{"type":"choice","instructions":"Which aspect scored worst?","criteria":{"content":"Content","pace":"Pace","materials":"Materials","venue":"Venue"}},"overall":{"type":"score","instructions":"How satisfied is the respondent overall?","criteria":["very dissatisfied","dissatisfied","neutral","satisfied","very satisfied"]},"satisfied":{"type":"noul","instructions":"The respondent is broadly satisfied."}}},"gold":{"weakest":"pace","overall":null,"satisfied":"true"},"note":"survey response","state_tokens":56},{"id":"own_j11","set":"fixture","source":"own_json","request":{"model":"clef-flash","state":[{"room":"A","capacity":4,"projector":true,"free":["10:00","14:00"]},{"room":"B","capacity":12,"projector":true,"free":["14:00"]},{"room":"C","capacity":8,"projector":false,"free":["10:00","11:00","14:00"]}],"questions":{"room":{"type":"choice","instructions":"Which room fits eight people with a projector at 14:00?","criteria":{"A":"Room A","B":"Room B","C":"Room C","none":"No room fits"}},"slot_10":{"type":"choice","instructions":"Which rooms are free at 10:00?","criteria":{"A_C":"Rooms A and C","B":"Room B only","all":"All rooms","none":"No room"}},"feasible":{"type":"noul","instructions":"A room with a projector for eight people is free at 10:00."}}},"gold":{"room":"B","slot_10":"A_C","feasible":"false"},"note":"state is a top-level JSON array","state_tokens":85},{"id":"own_j12","set":"fixture","source":"own_json","request":{"model":"clef-flash","state":{"customer":{"name":"Jürgen Faß","city":"São Paulo"},"basket":[{"item":"café crème","price_eur":3.8},{"item":"crème brûlée","price_eur":6.5},{"item":"Spätzle","price_eur":9.2}],"coupon":{"code":"AUTUMN10","percent":10,"valid_until":"2026-09-30"},"today":"2026-10-03"},"questions":{"coupon_valid":{"type":"noul","instructions":"The coupon can still be used today."},"total_band":{"type":"choice","instructions":"What is the basket total before any discount?","criteria":{"under_10":"Under 10 EUR","10_to_20":"10 to 20 EUR","20_to_30":"20 to 30 EUR","over_30":"Over 30 EUR"}},"dessert_items":{"type":"choice","instructions":"How many items are desserts?","criteria":{"0":"none","1":"one","2":"two","3":"three"}},"basket_size":{"type":"score","instructions":"How large is the basket?","criteria":["small","medium","large"]}}},"gold":{"coupon_valid":"false","total_band":"10_to_20","dessert_items":"1","basket_size":null},"note":"non-ASCII strings in the state (ensure_ascii=False rendering)","state_tokens":103},{"id":"img_01","set":"fixture","source":"own_image","request":{"model":"clef-flash","state":"Count the shapes in the attached picture.","questions":{"circles":{"type":"choice","instructions":"How many circles are there?","criteria":{"1":"one","2":"two","3":"three","4":"four","5":"five","6":"six"}},"more_circles":{"type":"noul","instructions":"There are more circles than squares."},"clutter":{"type":"score","instructions":"How crowded is the picture?","criteria":["sparse","moderate","crowded"]}}},"image_files":["shapes_224.png"],"gold":{"circles":"3","more_circles":"true","clutter":null},"note":"224x224","state_tokens":8},{"id":"img_02","set":"fixture","source":"own_image","request":{"model":"clef-flash","state":{"task":"Read the attached bar chart.","bars":"left to right: red, orange, yellow, green, blue"},"questions":{"tallest":{"type":"choice","instructions":"Which bar is the tallest?","criteria":{"red":"The red bar","orange":"The orange bar","yellow":"The yellow bar","green":"The green bar","blue":"The blue bar"}},"green_over_orange":{"type":"noul","instructions":"The green bar is taller than the orange bar."},"spread":{"type":"score","instructions":"How uneven are the bar heights?","criteria":["about equal","somewhat uneven","uneven","very uneven"]}}},"image_files":["bars_640x480.png"],"gold":{"tallest":"green","green_over_orange":"true","spread":null},"note":"640x480, JSON state","state_tokens":26},{"id":"img_03","set":"fixture","source":"own_image","request":{"model":"clef-flash","state":"The attached image shows a two-digit display.","questions":{"tens_digit":{"type":"choice","instructions":"What is the left digit?","criteria":{"0":"0","1":"1","2":"2","3":"3","4":"4","5":"5","6":"6","7":"7","8":"8","9":"9"}},"even":{"type":"noul","instructions":"The displayed number is even."},"readability":{"type":"score","instructions":"How easy is the display to read?","criteria":["hard","moderate","easy"]}}},"image_files":["sevenseg_47_300x200.png"],"gold":{"tens_digit":"4","even":"false","readability":null},"note":"300x200; a 10-option choice","state_tokens":9},{"id":"img_04","set":"fixture","source":"own_image","request":{"model":"clef-flash","state":{"task":"Read the attached receipt."},"questions":{"total":{"type":"choice","instructions":"What is the receipt total?","criteria":{"5.75":"5.75","6.57":"6.57","6.75":"6.75","7.65":"7.65"}},"paid_cash":{"type":"noul","instructions":"The purchase was paid in cash."},"items":{"type":"choice","instructions":"How many different items were bought?","criteria":{"1":"one","2":"two","3":"three","4":"four","5":"five","6":"six"}},"legibility":{"type":"score","instructions":"How legible is the receipt?","criteria":["illegible","partly legible","fully legible"]}}},"image_files":["receipt_1024x768.png"],"gold":{"total":"6.75","paid_cash":"true","items":"3","legibility":null},"note":"1024x768, dot-matrix lettering, invented store name","state_tokens":9},{"id":"img_05","set":"fixture","source":"own_image","request":{"model":"clef-flash","state":"You are driving and approach this signal.","questions":{"action":{"type":"choice","instructions":"What should the driver do?","criteria":{"stop":"Stop","go":"Proceed","slow":"Slow down and prepare to stop"}},"lit_color":{"type":"choice","instructions":"Which lamp is lit?","criteria":{"red":"Red","yellow":"Yellow","green":"Green"}},"may_proceed":{"type":"noul","instructions":"The driver may proceed."},"caution":{"type":"score","instructions":"How much caution is needed?","criteria":["little","some","a lot"]}}},"image_files":["signal_green_256x240.png"],"gold":{"action":"go","lit_color":"green","may_proceed":"true","caution":null},"note":"256x240","state_tokens":8},{"id":"img_06","set":"fixture","source":"own_image","request":{"model":"clef-flash","state":{"instrument":"boiler pressure gauge","red_zone_starts_at_percent":75},"questions":{"in_red_zone":{"type":"noul","instructions":"The needle is in the red zone."},"reading":{"type":"choice","instructions":"Where does the needle point?","criteria":{"0_25":"0 to 25 percent","25_50":"25 to 50 percent","50_75":"50 to 75 percent","75_100":"75 to 100 percent"}},"alarm":{"type":"score","instructions":"What alarm level fits the reading?","criteria":["none","advisory","warning","shutdown"]}}},"image_files":["gauge_300x200.png"],"gold":{"in_red_zone":"true","reading":"75_100","alarm":null},"note":"300x200, JSON state","state_tokens":17},{"id":"img_07","set":"fixture","source":"own_image","request":{"model":"clef-flash","state":"Find the odd one out among the arrows.","questions":{"odd_direction":{"type":"choice","instructions":"Which way does the odd arrow point?","criteria":{"up":"Up","down":"Down","left":"Left","right":"Right"}},"odd_position":{"type":"choice","instructions":"Which arrow is the odd one, counting from the left?","criteria":{"1":"the first","2":"the second","3":"the third","4":"the fourth"}},"all_same":{"type":"noul","instructions":"All arrows point the same way."}}},"image_files":["arrows_640x480.png"],"gold":{"odd_direction":"up","odd_position":"2","all_same":"false"},"note":"640x480","state_tokens":9},{"id":"img_08","set":"fixture","source":"own_image","request":{"model":"clef-flash","state":{"task":"Answer questions about the attached floor plan."},"questions":{"largest_room":{"type":"choice","instructions":"Which room is the largest?","criteria":{"living":"Living room","kitchen":"Kitchen","bed":"Bedroom","bath":"Bathroom"}},"rooms":{"type":"choice","instructions":"How many labelled spaces are there?","criteria":{"2":"two","3":"three","4":"four","5":"five","6":"six"}},"bath_next_to_bed":{"type":"noul","instructions":"The bathroom is next to the bedroom."},"complexity":{"type":"score","instructions":"How complex is the layout?","criteria":["simple","moderate","complex"]}}},"image_files":["floorplan_1024x768.png"],"gold":{"largest_room":"living","rooms":"5","bath_next_to_bed":"true","complexity":null},"note":"1024x768, JSON state","state_tokens":12},{"id":"img_09","set":"fixture","source":"own_image","request":{"model":"clef-flash","state":"Compare the circles.","questions":{"largest":{"type":"choice","instructions":"What color is the largest circle?","criteria":{"blue":"Blue","red":"Red","green":"Green"}},"smallest":{"type":"choice","instructions":"What color is the smallest circle?","criteria":{"blue":"Blue","red":"Red","green":"Green"}},"same_size":{"type":"noul","instructions":"All circles are the same size."},"spread":{"type":"score","instructions":"How different are the sizes?","criteria":["similar","somewhat different","very different"]}}},"image_files":["circles_640x480.png"],"gold":{"largest":"red","smallest":"blue","same_size":"false","spread":null},"note":"640x480","state_tokens":4},{"id":"img_10","set":"fixture","source":"own_image","request":{"model":"clef-flash","state":"The attached image shows a wall clock.","questions":{"hour":{"type":"choice","instructions":"What hour does the clock show?","criteria":{"12":"twelve","3":"three","6":"six","9":"nine"}},"minute_hand_up":{"type":"noul","instructions":"The minute hand points straight up."},"confidence":{"type":"score","instructions":"How clearly can the time be read?","criteria":["unclear","fairly clear","very clear"]}}},"image_files":["clock_3oclock_256x240.png"],"gold":{"hour":"3","minute_hand_up":"true","confidence":null},"note":"256x240","state_tokens":8},{"id":"photo_01","set":"fixture","source":"cc0_photo","request":{"model":"clef-flash","state":{"task":"Check the fruit bowl in the attached photo."},"questions":{"limes":{"type":"choice","instructions":"How many limes are in the bowl?","criteria":{"1":"one","2":"two","3":"three","4":"four","5":"five","6":"six"}},"has_banana":{"type":"noul","instructions":"There is a banana in the bowl."},"kinds":{"type":"choice","instructions":"How many different kinds of fruit are there?","criteria":{"2":"two","3":"three","4":"four","5":"five","6":"six"}},"freshness":{"type":"score","instructions":"How fresh does the fruit look?","criteria":["spoiled","acceptable","fresh"]}}},"image_files":["photo_fruit.png"],"gold":{"limes":"3","has_banana":"false","kinds":"4","freshness":null},"note":"CC0 photo, 960x1280","state_tokens":13},{"id":"photo_02","set":"fixture","source":"cc0_photo","request":{"model":"clef-flash","state":"Identify the animal in the attached photo.","questions":{"animal":{"type":"choice","instructions":"Which animal is shown?","criteria":{"cat":"A cat","dog":"A dog","rabbit":"A rabbit","fox":"A fox"}},"eye_color":{"type":"choice","instructions":"What color are its eyes?","criteria":{"blue":"Blue","green":"Green","yellow":"Yellow or amber","brown":"Brown"}},"background":{"type":"score","instructions":"How busy is the background?","criteria":["plain","somewhat busy","busy"]},"striped":{"type":"noul","instructions":"The animal's fur has stripes."}}},"image_files":["photo_cat.png"],"gold":{"animal":"cat","eye_color":"yellow","background":null,"striped":"true"},"note":"CC0 photo, 960x640","state_tokens":9},{"id":"photo_03","set":"fixture","source":"cc0_photo","request":{"model":"clef-flash","state":{"task":"Answer questions about the attached landscape photo."},"questions":{"snow_visible":{"type":"noul","instructions":"Snow is visible in the photo."},"reflection":{"type":"noul","instructions":"The water reflects the landscape."},"sky":{"type":"choice","instructions":"What is the sky like?","criteria":{"clear":"Clear and sunny","overcast":"Overcast","rain":"Raining","night":"Night"}},"wildness":{"type":"score","instructions":"How wild is the place?","criteria":["urban","suburban","rural","wilderness"]}}},"image_files":["photo_lake.png"],"gold":{"snow_visible":"true","reflection":"true","sky":"overcast","wildness":null},"note":"CC0 photo, 960x640","state_tokens":12},{"id":"photo_04","set":"fixture","source":"cc0_photo","request":{"model":"clef-flash","state":"What is shown in the attached photo?","questions":{"drink":{"type":"choice","instructions":"Which drink is in the cup?","criteria":{"tea":"Tea","coffee":"Coffee","milk":"Milk","juice":"Juice"}},"milk_added":{"type":"noul","instructions":"Milk has been added to the drink."},"cup_color":{"type":"choice","instructions":"What color is the cup?","criteria":{"white":"White","black":"Black","red":"Red","blue":"Blue"}},"beans":{"type":"score","instructions":"How many coffee beans are visible?","criteria":["none","a few","many"]}}},"image_files":["photo_coffee.png"],"gold":{"drink":"coffee","milk_added":"false","cup_color":"white","beans":"2"},"note":"CC0 photo, 960x635","state_tokens":8},{"id":"semif_a3f18f3a63d45345942b","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The optician ordered replacement lenses. The workshop confirms they have not yet been fitted to the customer's glasses.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the replacement lenses have been fitted.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=2fdaa8a61e6e989dd866","state_tokens":22,"semif":{"family":"evidence_interpretation","variant":"original","group_id":"2fdaa8a61e6e989dd866","label":2,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_f40beba9088c8db8bbd6","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The optician ordered replacement lenses. The workshop confirms they have not yet been fitted to the customer's glasses.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: replacement lenses were ordered.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=2fdaa8a61e6e989dd866","state_tokens":22,"semif":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"2fdaa8a61e6e989dd866","label":0,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_0b43ea8e24e74d621f1b","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The workshop confirms it fitted the replacement lenses to the customer's glasses.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the replacement lenses have been fitted.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=2fdaa8a61e6e989dd866","state_tokens":14,"semif":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"2fdaa8a61e6e989dd866","label":1,"option_order":["insufficient","supported","contradicted"]}},{"id":"semif_942ec65dac0bfb9874f9","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The optician records a glasses prescription but no order or fitting status.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the replacement lenses have been fitted.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=2fdaa8a61e6e989dd866","state_tokens":14,"semif":{"family":"evidence_interpretation","variant":"missing","group_id":"2fdaa8a61e6e989dd866","label":0,"option_order":["insufficient","supported","contradicted"]}},{"id":"semif_e0c140e9222d1bce5647","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The guest writes, \"The listing says towels are supplied. I agree: fresh towels were provided during my stay.\"","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the guest endorses the listing's towel statement.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=original group=f4bea3d3bcc01056740d","state_tokens":23,"semif":{"family":"evidence_interpretation","variant":"original","group_id":"f4bea3d3bcc01056740d","label":0,"option_order":["supported","contradicted","insufficient"]}},{"id":"semif_8e8c3804a3c15ebb31e3","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The guest writes, \"The listing says towels are supplied. I agree: fresh towels were provided during my stay.\"","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the guest disputes the towel statement.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=criterion_reversal group=f4bea3d3bcc01056740d","state_tokens":23,"semif":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"f4bea3d3bcc01056740d","label":2,"option_order":["insufficient","supported","contradicted"]}},{"id":"semif_56c419b12cd37cbe790c","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The guest writes, \"The listing says towels are supplied, but none were provided. I dispute that statement.\"","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the guest endorses the listing's towel statement.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=evidence_change group=f4bea3d3bcc01056740d","state_tokens":22,"semif":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"f4bea3d3bcc01056740d","label":0,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_75b470586834df8c568d","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The guest forwards the listing but provides no comment about towels or the stay.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the guest endorses the listing's towel statement.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=f4bea3d3bcc01056740d","state_tokens":15,"semif":{"family":"evidence_interpretation","variant":"missing","group_id":"f4bea3d3bcc01056740d","label":2,"option_order":["supported","contradicted","insufficient"]}},{"id":"semif_834f3b3e10cc61b33d25","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The technician says terminal Cedar is online. Terminal Maple is explicitly offline.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: terminal Maple is online.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=b51b300da2936decc6b7","state_tokens":14,"semif":{"family":"evidence_interpretation","variant":"original","group_id":"b51b300da2936decc6b7","label":2,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_76c7a79b1ff2363a60c5","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The technician says terminal Cedar is online. Terminal Maple is explicitly offline.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: terminal Cedar is online.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=b51b300da2936decc6b7","state_tokens":14,"semif":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"b51b300da2936decc6b7","label":0,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_36c71bb649b7d5eccb1f","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The technician confirms terminal Maple is online and terminal Cedar is offline.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: terminal Maple is online.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=b51b300da2936decc6b7","state_tokens":13,"semif":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"b51b300da2936decc6b7","label":0,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_eac5121a4a1bf391a08c","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The technician lists both terminals but omits their connection status.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: terminal Maple is online.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=b51b300da2936decc6b7","state_tokens":12,"semif":{"family":"evidence_interpretation","variant":"missing","group_id":"b51b300da2936decc6b7","label":1,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_44574c44e243ee1b2f52","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The catalogue originally dated the letter to June. Its author issued a correction stating July was correct and June was mistaken.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the corrected date is in July.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=original group=e4cfd694c80e08d3e6a5","state_tokens":23,"semif":{"family":"evidence_interpretation","variant":"original","group_id":"e4cfd694c80e08d3e6a5","label":2,"option_order":["insufficient","contradicted","supported"]}},{"id":"semif_2608370d01783735c3ea","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The catalogue originally dated the letter to June. Its author issued a correction stating July was correct and June was mistaken.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the corrected date is in June.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=criterion_reversal group=e4cfd694c80e08d3e6a5","state_tokens":23,"semif":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"e4cfd694c80e08d3e6a5","label":2,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_ec94ef909f7918695ab3","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The catalogue author corrects the date to June and expressly rejects the earlier July date.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the corrected date is in July.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=evidence_change group=e4cfd694c80e08d3e6a5","state_tokens":17,"semif":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"e4cfd694c80e08d3e6a5","label":2,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_03d919fd0e9b3a7cd1e6","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The catalogue mentions a date correction but gives no corrected date.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the corrected date is in July.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=e4cfd694c80e08d3e6a5","state_tokens":12,"semif":{"family":"evidence_interpretation","variant":"missing","group_id":"e4cfd694c80e08d3e6a5","label":0,"option_order":["insufficient","supported","contradicted"]}},{"id":"semif_86a15618ba4cb90a307e","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The intern attached instructions for weighing the parcel and explicitly says no weighing has been performed.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the parcel has been weighed.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=20460d4ae0db46091e84","state_tokens":17,"semif":{"family":"evidence_interpretation","variant":"original","group_id":"20460d4ae0db46091e84","label":0,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_3dd9c3ac98f8ddf7bdd3","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The intern attached instructions for weighing the parcel and explicitly says no weighing has been performed.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the document contains weighing instructions.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=20460d4ae0db46091e84","state_tokens":17,"semif":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"20460d4ae0db46091e84","label":1,"option_order":["contradicted","supported","insufficient"]}},{"id":"semif_36273dd4777876a26338","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The intern performed the weighing and recorded the parcel's measured weight.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the parcel has been weighed.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=20460d4ae0db46091e84","state_tokens":13,"semif":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"20460d4ae0db46091e84","label":1,"option_order":["contradicted","supported","insufficient"]}},{"id":"semif_07ee70cfe4b01330ce89","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The intern supplies a parcel identifier but no procedure or measurement information.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the parcel has been weighed.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=20460d4ae0db46091e84","state_tokens":13,"semif":{"family":"evidence_interpretation","variant":"missing","group_id":"20460d4ae0db46091e84","label":2,"option_order":["supported","contradicted","insufficient"]}},{"id":"semif_51ceab3ce622c7c759f1","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"During a puppet show, a character announces the discovery of a dragon egg. The production note states this is fictional and no real egg was discovered.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the egg discovery was fictional.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=original group=51b91ba31984c5939c1c","state_tokens":29,"semif":{"family":"evidence_interpretation","variant":"original","group_id":"51b91ba31984c5939c1c","label":2,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_b8091be929b1e489fcc4","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"During a puppet show, a character announces the discovery of a dragon egg. The production note states this is fictional and no real egg was discovered.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: a real egg was discovered.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=criterion_reversal group=51b91ba31984c5939c1c","state_tokens":29,"semif":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"51b91ba31984c5939c1c","label":1,"option_order":["supported","contradicted","insufficient"]}},{"id":"semif_c89b86e3f55123bfa506","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The note describes a real bird egg discovered outside a performance and expressly says the discovery was not fictional.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the egg discovery was fictional.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=evidence_change group=51b91ba31984c5939c1c","state_tokens":20,"semif":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"51b91ba31984c5939c1c","label":1,"option_order":["supported","contradicted","insufficient"]}},{"id":"semif_3b0958cabc339005cf66","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"A note mentions an egg without describing a discovery or whether the account is fictional.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the egg discovery was fictional.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=51b91ba31984c5939c1c","state_tokens":16,"semif":{"family":"evidence_interpretation","variant":"missing","group_id":"51b91ba31984c5939c1c","label":0,"option_order":["insufficient","supported","contradicted"]}},{"id":"semif_f2b4ec4930322fe45118","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The architect permits reproducing a plan in an internal training booklet and expressly forbids selling copies.","questions":{"decision":{"type":"choice","instructions":"Sell copies only with the architect's sales permission. May copies be sold?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=770c1607f0a11f88f81e","state_tokens":19,"semif":{"family":"rule_application","variant":"original","group_id":"770c1607f0a11f88f81e","label":1,"option_order":["insufficient","prohibited","permitted"]}},{"id":"semif_61ac20b07a5189231e15","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The architect permits reproducing a plan in an internal training booklet and expressly forbids selling copies.","questions":{"decision":{"type":"choice","instructions":"Internal training reproduction is allowed when the architect permits that use. May the plan appear in the training booklet?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=770c1607f0a11f88f81e","state_tokens":19,"semif":{"family":"rule_application","variant":"criterion_reversal","group_id":"770c1607f0a11f88f81e","label":1,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_4b3d794c004566610181","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The architect expressly permits selling copies of the plan.","questions":{"decision":{"type":"choice","instructions":"Sell copies only with the architect's sales permission. May copies be sold?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=770c1607f0a11f88f81e","state_tokens":10,"semif":{"family":"rule_application","variant":"evidence_change","group_id":"770c1607f0a11f88f81e","label":2,"option_order":["prohibited","insufficient","permitted"]}},{"id":"semif_b7c4e62d07986286fce0","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"An architect's permission letter exists, but the uses it allows are omitted.","questions":{"decision":{"type":"choice","instructions":"Sell copies only with the architect's sales permission. May copies be sold?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=770c1607f0a11f88f81e","state_tokens":15,"semif":{"family":"rule_application","variant":"missing","group_id":"770c1607f0a11f88f81e","label":0,"option_order":["insufficient","permitted","prohibited"]}},{"id":"semif_9473603b44af63beab79","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The guide completed first-aid training but not mountain-navigation training.","questions":{"decision":{"type":"choice","instructions":"The first-aid badge requires only first-aid training. May this guide receive it?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=original group=ffd081ada3bd9be9207a","state_tokens":13,"semif":{"family":"rule_application","variant":"original","group_id":"ffd081ada3bd9be9207a","label":0,"option_order":["permitted","insufficient","prohibited"]}},{"id":"semif_d41072dc35d8cd14c056","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The guide completed first-aid training but not mountain-navigation training.","questions":{"decision":{"type":"choice","instructions":"Under the alternative rule, the badge requires both first-aid and mountain-navigation training. May this guide receive it?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=criterion_reversal group=ffd081ada3bd9be9207a","state_tokens":13,"semif":{"family":"rule_application","variant":"criterion_reversal","group_id":"ffd081ada3bd9be9207a","label":1,"option_order":["permitted","prohibited","insufficient"]}},{"id":"semif_33d6e5da58f0fee2490d","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The guide completed mountain-navigation training but not first-aid training.","questions":{"decision":{"type":"choice","instructions":"The first-aid badge requires only first-aid training. May this guide receive it?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=evidence_change group=ffd081ada3bd9be9207a","state_tokens":13,"semif":{"family":"rule_application","variant":"evidence_change","group_id":"ffd081ada3bd9be9207a","label":0,"option_order":["prohibited","insufficient","permitted"]}},{"id":"semif_823bf736b92e9d8fa66d","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The guide registered for training, but completion information is not supplied.","questions":{"decision":{"type":"choice","instructions":"The first-aid badge requires only first-aid training. May this guide receive it?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=ffd081ada3bd9be9207a","state_tokens":13,"semif":{"family":"rule_application","variant":"missing","group_id":"ffd081ada3bd9be9207a","label":1,"option_order":["prohibited","insufficient","permitted"]}},{"id":"semif_8a4c3de28be95c105ca1","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The designer approves making a sample badge but expressly withholds approval for mass production.","questions":{"decision":{"type":"choice","instructions":"Mass production is allowed only after mass-production approval. May it begin?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=2c3a0cb61f7bae78739e","state_tokens":16,"semif":{"family":"rule_application","variant":"original","group_id":"2c3a0cb61f7bae78739e","label":2,"option_order":["permitted","insufficient","prohibited"]}},{"id":"semif_ea7ec66a5e8eac886548","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The designer approves making a sample badge but expressly withholds approval for mass production.","questions":{"decision":{"type":"choice","instructions":"A sample may be made when sample production is approved. May the sample be made?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=2c3a0cb61f7bae78739e","state_tokens":16,"semif":{"family":"rule_application","variant":"criterion_reversal","group_id":"2c3a0cb61f7bae78739e","label":0,"option_order":["permitted","insufficient","prohibited"]}},{"id":"semif_25172cd6ddc49ab0ca70","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The designer explicitly approves mass production of the badge.","questions":{"decision":{"type":"choice","instructions":"Mass production is allowed only after mass-production approval. May it begin?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=2c3a0cb61f7bae78739e","state_tokens":10,"semif":{"family":"rule_application","variant":"evidence_change","group_id":"2c3a0cb61f7bae78739e","label":2,"option_order":["prohibited","insufficient","permitted"]}},{"id":"semif_fb73a85d386cf59b0c51","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The designer issued a decision but its production scope is unavailable.","questions":{"decision":{"type":"choice","instructions":"Mass production is allowed only after mass-production approval. May it begin?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=2c3a0cb61f7bae78739e","state_tokens":12,"semif":{"family":"rule_application","variant":"missing","group_id":"2c3a0cb61f7bae78739e","label":0,"option_order":["insufficient","permitted","prohibited"]}},{"id":"semif_a02fd582d2881cc462b2","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The visitor's wristband covers the sculpture garden and expressly excludes the rooftop terrace.","questions":{"decision":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the sculpture garden?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=original group=42ed4cbfde7f2f64274b","state_tokens":16,"semif":{"family":"rule_application","variant":"original","group_id":"42ed4cbfde7f2f64274b","label":2,"option_order":["prohibited","insufficient","permitted"]}},{"id":"semif_19f37d5a97034bd7d04f","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The visitor's wristband covers the sculpture garden and expressly excludes the rooftop terrace.","questions":{"decision":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the rooftop terrace?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=criterion_reversal group=42ed4cbfde7f2f64274b","state_tokens":16,"semif":{"family":"rule_application","variant":"criterion_reversal","group_id":"42ed4cbfde7f2f64274b","label":1,"option_order":["insufficient","prohibited","permitted"]}},{"id":"semif_fda0ca94652c6d739ecf","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The wristband expressly excludes the sculpture garden and covers only the rooftop terrace.","questions":{"decision":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the sculpture garden?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=evidence_change group=42ed4cbfde7f2f64274b","state_tokens":15,"semif":{"family":"rule_application","variant":"evidence_change","group_id":"42ed4cbfde7f2f64274b","label":0,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_d0e36c4cb10a29b3191c","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The visitor has a wristband but its covered areas are not supplied.","questions":{"decision":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the sculpture garden?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=42ed4cbfde7f2f64274b","state_tokens":14,"semif":{"family":"rule_application","variant":"missing","group_id":"42ed4cbfde7f2f64274b","label":0,"option_order":["insufficient","permitted","prohibited"]}},{"id":"semif_a0a39ace7e5337d038a7","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The account owner permits changing the display name but expressly forbids changing the recovery email.","questions":{"decision":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the recovery email be changed?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=6dd193172b9240a04bf9","state_tokens":17,"semif":{"family":"rule_application","variant":"original","group_id":"6dd193172b9240a04bf9","label":0,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_eeae283841ce75dd25bf","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The account owner permits changing the display name but expressly forbids changing the recovery email.","questions":{"decision":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the display name be changed?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=6dd193172b9240a04bf9","state_tokens":17,"semif":{"family":"rule_application","variant":"criterion_reversal","group_id":"6dd193172b9240a04bf9","label":1,"option_order":["insufficient","permitted","prohibited"]}},{"id":"semif_00bb4cc043f7d2b288d4","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The owner expressly permits changing the recovery email.","questions":{"decision":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the recovery email be changed?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=6dd193172b9240a04bf9","state_tokens":9,"semif":{"family":"rule_application","variant":"evidence_change","group_id":"6dd193172b9240a04bf9","label":2,"option_order":["insufficient","prohibited","permitted"]}},{"id":"semif_1a81ead69e93e5d0c86a","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The owner requested an account update, but the fields to be changed are omitted.","questions":{"decision":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the recovery email be changed?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=6dd193172b9240a04bf9","state_tokens":16,"semif":{"family":"rule_application","variant":"missing","group_id":"6dd193172b9240a04bf9","label":2,"option_order":["permitted","prohibited","insufficient"]}},{"id":"semif_4444dad25e438f128673","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The applicant lives in the valley but works outside it.","questions":{"decision":{"type":"choice","instructions":"A resident's pass requires living in the valley and has no workplace restriction. May it be issued?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=original group=3200eb252d6737e03027","state_tokens":11,"semif":{"family":"rule_application","variant":"original","group_id":"3200eb252d6737e03027","label":1,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_3226202ba1ada2570e9f","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The applicant lives in the valley but works outside it.","questions":{"decision":{"type":"choice","instructions":"Under the alternative rule, a pass requires working in the valley; residence alone is insufficient. May it be issued?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=criterion_reversal group=3200eb252d6737e03027","state_tokens":11,"semif":{"family":"rule_application","variant":"criterion_reversal","group_id":"3200eb252d6737e03027","label":0,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_3f783eec5c90812479bc","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The applicant lives outside the valley and works inside it.","questions":{"decision":{"type":"choice","instructions":"A resident's pass requires living in the valley and has no workplace restriction. May it be issued?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=evidence_change group=3200eb252d6737e03027","state_tokens":11,"semif":{"family":"rule_application","variant":"evidence_change","group_id":"3200eb252d6737e03027","label":1,"option_order":["permitted","prohibited","insufficient"]}},{"id":"semif_ffee861474a9665a093a","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The applicant supplies a name but no home or work location.","questions":{"decision":{"type":"choice","instructions":"A resident's pass requires living in the valley and has no workplace restriction. May it be issued?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=3200eb252d6737e03027","state_tokens":12,"semif":{"family":"rule_application","variant":"missing","group_id":"3200eb252d6737e03027","label":2,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_8c992e17e4fb54b044f5","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The florist confirms that the wreath was delivered. Candidate B: The florist offers to deliver a wreath tomorrow.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting completed delivery.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=c0cbeb97315307af8111","state_tokens":28,"semif":{"family":"candidate_selection","variant":"original","group_id":"c0cbeb97315307af8111","label":1,"option_order":["B","A","insufficient"]}},{"id":"semif_695dd61195ca025c4f4c","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The florist confirms that the wreath was delivered. Candidate B: The florist offers to deliver a wreath tomorrow.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate offering a future delivery.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=c0cbeb97315307af8111","state_tokens":28,"semif":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"c0cbeb97315307af8111","label":0,"option_order":["B","insufficient","A"]}},{"id":"semif_e6155431ab7b0f8170d4","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: I can deliver a wreath tomorrow. Candidate B: The wreath was delivered.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting completed delivery.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=c0cbeb97315307af8111","state_tokens":20,"semif":{"family":"candidate_selection","variant":"evidence_change","group_id":"c0cbeb97315307af8111","label":1,"option_order":["insufficient","B","A"]}},{"id":"semif_b49ecda75086d8c68d2a","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The florist gives an address. Candidate B: The wreath contains lilies.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting completed delivery.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=c0cbeb97315307af8111","state_tokens":20,"semif":{"family":"candidate_selection","variant":"missing","group_id":"c0cbeb97315307af8111","label":2,"option_order":["B","A","insufficient"]}},{"id":"semif_587ced261c6092cd1325","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The customer asks how an engraving would look but explicitly declines having it engraved yet. Candidate B: The customer instructs the shop to engrave the ring.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate authorizing engraving.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=original group=428176cd8ad2c7a74eea","state_tokens":35,"semif":{"family":"candidate_selection","variant":"original","group_id":"428176cd8ad2c7a74eea","label":1,"option_order":["insufficient","B","A"]}},{"id":"semif_533d4423d311de82b27d","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The customer asks how an engraving would look but explicitly declines having it engraved yet. Candidate B: The customer instructs the shop to engrave the ring.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate asking only for information while withholding authorization.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=criterion_reversal group=428176cd8ad2c7a74eea","state_tokens":35,"semif":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"428176cd8ad2c7a74eea","label":1,"option_order":["insufficient","A","B"]}},{"id":"semif_2bf540d515a51755b25c","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: Please engrave the ring. Candidate B: Show me how it would look, but do not engrave it yet.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate authorizing engraving.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=evidence_change group=428176cd8ad2c7a74eea","state_tokens":27,"semif":{"family":"candidate_selection","variant":"evidence_change","group_id":"428176cd8ad2c7a74eea","label":2,"option_order":["insufficient","B","A"]}},{"id":"semif_fdb9fc214aea2710763a","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The customer gives a name. Candidate B: The ring is silver.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate authorizing engraving.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=428176cd8ad2c7a74eea","state_tokens":17,"semif":{"family":"candidate_selection","variant":"missing","group_id":"428176cd8ad2c7a74eea","label":2,"option_order":["B","A","insufficient"]}},{"id":"semif_f6d68126b43f8c9ed183","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The note records that the completed insulation test passed. Candidate B: The note lists insulation-test steps but says no test was run.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an observed test result.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=389e65bf1ba709b84b12","state_tokens":29,"semif":{"family":"candidate_selection","variant":"original","group_id":"389e65bf1ba709b84b12","label":1,"option_order":["insufficient","A","B"]}},{"id":"semif_44bae0c8a1615a096c43","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The note records that the completed insulation test passed. Candidate B: The note lists insulation-test steps but says no test was run.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate containing only an unexecuted test procedure.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=389e65bf1ba709b84b12","state_tokens":29,"semif":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"389e65bf1ba709b84b12","label":0,"option_order":["B","A","insufficient"]}},{"id":"semif_2175379f4c69e6207626","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: Here are insulation-test steps; no test was run. Candidate B: The completed insulation test passed.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an observed test result.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=389e65bf1ba709b84b12","state_tokens":23,"semif":{"family":"candidate_selection","variant":"evidence_change","group_id":"389e65bf1ba709b84b12","label":1,"option_order":["insufficient","B","A"]}},{"id":"semif_def9fe59d262357a673a","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The cable is red. Candidate B: The tester is in a drawer.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an observed test result.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=389e65bf1ba709b84b12","state_tokens":18,"semif":{"family":"candidate_selection","variant":"missing","group_id":"389e65bf1ba709b84b12","label":0,"option_order":["insufficient","A","B"]}},{"id":"semif_712f6ffb763eef7c9a12","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The editor quotes a proposal for a paywall only to argue against it. Candidate B: The editor endorses introducing a paywall.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate supporting a paywall.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=original group=45f8ead2522a14384cdc","state_tokens":30,"semif":{"family":"candidate_selection","variant":"original","group_id":"45f8ead2522a14384cdc","label":0,"option_order":["B","A","insufficient"]}},{"id":"semif_e1d610bd14e3d16c09b9","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The editor quotes a proposal for a paywall only to argue against it. Candidate B: The editor endorses introducing a paywall.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate opposing a paywall.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=criterion_reversal group=45f8ead2522a14384cdc","state_tokens":30,"semif":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"45f8ead2522a14384cdc","label":0,"option_order":["A","insufficient","B"]}},{"id":"semif_22ec57822db50ec121cf","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: I endorse introducing a paywall. Candidate B: I reject introducing a paywall.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate supporting a paywall.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=evidence_change group=45f8ead2522a14384cdc","state_tokens":20,"semif":{"family":"candidate_selection","variant":"evidence_change","group_id":"45f8ead2522a14384cdc","label":1,"option_order":["insufficient","A","B"]}},{"id":"semif_3e75f2b84e8e08e3d3f2","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The editor names the magazine. Candidate B: The website has a blue header.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate supporting a paywall.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=45f8ead2522a14384cdc","state_tokens":19,"semif":{"family":"candidate_selection","variant":"missing","group_id":"45f8ead2522a14384cdc","label":2,"option_order":["B","A","insufficient"]}},{"id":"semif_ebd74aa05324a1556288","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The owner asks to reseal the existing window while keeping its frame. Candidate B: The owner asks to remove the window and fit a complete replacement.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate asking to repair the existing window.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=3ecec8e2c166e397ad07","state_tokens":33,"semif":{"family":"candidate_selection","variant":"original","group_id":"3ecec8e2c166e397ad07","label":1,"option_order":["insufficient","A","B"]}},{"id":"semif_d7e84e4245a29817da1f","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The owner asks to reseal the existing window while keeping its frame. Candidate B: The owner asks to remove the window and fit a complete replacement.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate asking for complete replacement.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=3ecec8e2c166e397ad07","state_tokens":33,"semif":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"3ecec8e2c166e397ad07","label":2,"option_order":["insufficient","A","B"]}},{"id":"semif_f41ae66b3c09630c7a42","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: Remove the window and install a replacement. Candidate B: Reseal the existing window and keep its frame.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate asking to repair the existing window.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=3ecec8e2c166e397ad07","state_tokens":25,"semif":{"family":"candidate_selection","variant":"evidence_change","group_id":"3ecec8e2c166e397ad07","label":2,"option_order":["insufficient","A","B"]}},{"id":"semif_82c2cd97c160ad27951b","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The window faces a courtyard. Candidate B: The owner gives a postcode.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate asking to repair the existing window.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=3ecec8e2c166e397ad07","state_tokens":18,"semif":{"family":"candidate_selection","variant":"missing","group_id":"3ecec8e2c166e397ad07","label":0,"option_order":["insufficient","B","A"]}},{"id":"semif_7ada97b6f99f4788e5c7","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The sponsor confirms receipt of a grant application but explicitly says no award decision has been made. Candidate B: The sponsor confirms that the grant was awarded.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an awarded grant.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=original group=a71a17a287f3d30db2aa","state_tokens":33,"semif":{"family":"candidate_selection","variant":"original","group_id":"a71a17a287f3d30db2aa","label":0,"option_order":["B","A","insufficient"]}},{"id":"semif_4d7a003342db5b0a17e3","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The sponsor confirms receipt of a grant application but explicitly says no award decision has been made. Candidate B: The sponsor confirms that the grant was awarded.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting receipt with the award still undecided.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=criterion_reversal group=a71a17a287f3d30db2aa","state_tokens":33,"semif":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"a71a17a287f3d30db2aa","label":1,"option_order":["insufficient","A","B"]}},{"id":"semif_19d3d8d0bf773e88be7b","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The grant was awarded. Candidate B: The application was received; the award remains undecided.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an awarded grant.","criteria":{"A":"Candidate A","B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=evidence_change group=a71a17a287f3d30db2aa","state_tokens":21,"semif":{"family":"candidate_selection","variant":"evidence_change","group_id":"a71a17a287f3d30db2aa","label":0,"option_order":["A","B","insufficient"]}},{"id":"semif_d6de731cca52359b1c87","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The sponsor gives its name. Candidate B: The application has a reference number.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an awarded grant.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=a71a17a287f3d30db2aa","state_tokens":19,"semif":{"family":"candidate_selection","variant":"missing","group_id":"a71a17a287f3d30db2aa","label":0,"option_order":["insufficient","A","B"]}},{"id":"semif_b45c63fb92729af27f05","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"On Monday the owner authorized scanning her notebook. On Tuesday she explicitly withdrew that authorization. The record says withdrawals cancel earlier permission, and no later permission was given.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: permission to scan the notebook is currently valid.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=986d7eaba0c7e9738e79","state_tokens":32,"semif":{"family":"evidence_interpretation","variant":"original","group_id":"986d7eaba0c7e9738e79","label":0,"option_order":["contradicted","supported","insufficient"]}},{"id":"semif_46b7029b9a704138b77a","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"On Monday the owner authorized scanning her notebook. On Tuesday she explicitly withdrew that authorization. The record says withdrawals cancel earlier permission, and no later permission was given.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the owner gave permission on Monday.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=986d7eaba0c7e9738e79","state_tokens":32,"semif":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"986d7eaba0c7e9738e79","label":2,"option_order":["insufficient","contradicted","supported"]}},{"id":"semif_caf773b23934fa7362a1","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The owner authorized scanning the notebook on Monday. The complete permission record confirms that she has not withdrawn it and it remains valid.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: permission to scan the notebook is currently valid.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=986d7eaba0c7e9738e79","state_tokens":25,"semif":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"986d7eaba0c7e9738e79","label":0,"option_order":["supported","contradicted","insufficient"]}},{"id":"semif_f46f392ef9e9e9df564b","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The owner authorized scanning on Monday. The supplied record omits all later permission changes and does not establish current validity.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: permission to scan the notebook is currently valid.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=986d7eaba0c7e9738e79","state_tokens":23,"semif":{"family":"evidence_interpretation","variant":"missing","group_id":"986d7eaba0c7e9738e79","label":1,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_005dcf6d2c0f21d64796","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The homeowner first allowed a surveyor to enter the attic. Before the visit she explicitly withdrew that permission. No later decision exists.","questions":{"decision":{"type":"choice","instructions":"Use the homeowner's latest permission decision; a withdrawal cancels the earlier grant. May the surveyor enter the attic?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=ca2e0b043985fa49bcd2","state_tokens":26,"semif":{"family":"rule_application","variant":"original","group_id":"ca2e0b043985fa49bcd2","label":1,"option_order":["permitted","prohibited","insufficient"]}},{"id":"semif_edb708bd0fe5fd67d6ee","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The homeowner first allowed a surveyor to enter the attic. Before the visit she explicitly withdrew that permission. No later decision exists.","questions":{"decision":{"type":"choice","instructions":"For this fictional protocol, the first recorded decision is binding and later withdrawals are ignored. May the surveyor enter the attic?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=ca2e0b043985fa49bcd2","state_tokens":26,"semif":{"family":"rule_application","variant":"criterion_reversal","group_id":"ca2e0b043985fa49bcd2","label":0,"option_order":["permitted","prohibited","insufficient"]}},{"id":"semif_97a35499bd8717f3e786","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The homeowner first refused attic entry, then explicitly granted it before the visit. No later decision exists.","questions":{"decision":{"type":"choice","instructions":"Use the homeowner's latest permission decision; a withdrawal cancels the earlier grant. May the surveyor enter the attic?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=ca2e0b043985fa49bcd2","state_tokens":20,"semif":{"family":"rule_application","variant":"evidence_change","group_id":"ca2e0b043985fa49bcd2","label":2,"option_order":["prohibited","insufficient","permitted"]}},{"id":"semif_31eb4533d2c89959ec46","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The homeowner gave an early permission decision, but neither its content nor any later decisions are available.","questions":{"decision":{"type":"choice","instructions":"Use the homeowner's latest permission decision; a withdrawal cancels the earlier grant. May the surveyor enter the attic?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=ca2e0b043985fa49bcd2","state_tokens":19,"semif":{"family":"rule_application","variant":"missing","group_id":"ca2e0b043985fa49bcd2","label":0,"option_order":["insufficient","prohibited","permitted"]}},{"id":"semif_0aa24a6701ac9c56c4a0","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The speaker authorized streaming, then explicitly withdrew the authorization before the event. Candidate B: The speaker authorized streaming and the complete record confirms it remains in force.","questions":{"decision":{"type":"choice","instructions":"Select the candidate with currently valid streaming permission; a withdrawal cancels an earlier grant.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=original group=85051ba3c30277824ebe","state_tokens":34,"semif":{"family":"candidate_selection","variant":"original","group_id":"85051ba3c30277824ebe","label":1,"option_order":["insufficient","B","A"]}},{"id":"semif_796d5c0da6eaadae6a3f","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: The speaker authorized streaming, then explicitly withdrew the authorization before the event. Candidate B: The speaker authorized streaming and the complete record confirms it remains in force.","questions":{"decision":{"type":"choice","instructions":"Select the candidate documenting an authorization that was later withdrawn.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=criterion_reversal group=85051ba3c30277824ebe","state_tokens":34,"semif":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"85051ba3c30277824ebe","label":1,"option_order":["B","A","insufficient"]}},{"id":"semif_599b3092b6fd18312d23","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: Streaming was authorized and the complete record says permission remains in force. Candidate B: Streaming was authorized, then explicitly withdrawn.","questions":{"decision":{"type":"choice","instructions":"Select the candidate with currently valid streaming permission; a withdrawal cancels an earlier grant.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=evidence_change group=85051ba3c30277824ebe","state_tokens":27,"semif":{"family":"candidate_selection","variant":"evidence_change","group_id":"85051ba3c30277824ebe","label":2,"option_order":["B","insufficient","A"]}},{"id":"semif_960a4bcd27363e6db4b9","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A: A permission form is mentioned but its terms and later changes are missing. Candidate B: The event name is supplied without permission history.","questions":{"decision":{"type":"choice","instructions":"Select the candidate with currently valid streaming permission; a withdrawal cancels an earlier grant.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=85051ba3c30277824ebe","state_tokens":29,"semif":{"family":"candidate_selection","variant":"missing","group_id":"85051ba3c30277824ebe","label":2,"option_order":["B","A","insufficient"]}},{"id":"semif_c6a411e7f2bbdf377173","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The director gave Lina authority to approve exhibit labels and explicitly forbade further delegation. Lina then told Mo to approve them. The rule states an attempted forbidden delegation grants no authority; Mo has no other grant.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: Mo has authority to approve exhibit labels.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=e7798dab5f4f121cab77","state_tokens":43,"semif":{"family":"evidence_interpretation","variant":"original","group_id":"e7798dab5f4f121cab77","label":0,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_e332aa927d0c2ed044aa","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The director gave Lina authority to approve exhibit labels and explicitly forbade further delegation. Lina then told Mo to approve them. The rule states an attempted forbidden delegation grants no authority; Mo has no other grant.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: Lina has authority to approve exhibit labels.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=e7798dab5f4f121cab77","state_tokens":43,"semif":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"e7798dab5f4f121cab77","label":2,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_a0128e551d095f3449f8","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The director gives Mo direct authority to approve exhibit labels; the record confirms this grant is valid.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: Mo has authority to approve exhibit labels.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=e7798dab5f4f121cab77","state_tokens":19,"semif":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"e7798dab5f4f121cab77","label":0,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_46d6a0731bd288c370f0","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Mo is named in correspondence about labels, but no authority grants or delegation terms are supplied.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: Mo has authority to approve exhibit labels.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=e7798dab5f4f121cab77","state_tokens":18,"semif":{"family":"evidence_interpretation","variant":"missing","group_id":"e7798dab5f4f121cab77","label":0,"option_order":["insufficient","contradicted","supported"]}},{"id":"semif_21ca7a8b970e36b8e20f","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The producer authorizes the stage manager to approve props. The stage manager passes the task to an assistant, who has no other approval authority.","questions":{"decision":{"type":"choice","instructions":"The stage manager may approve props but may not delegate that authority. Only authorized approvers may approve. May the assistant approve a prop?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=928ebc7817a8f1cac3e4","state_tokens":28,"semif":{"family":"rule_application","variant":"original","group_id":"928ebc7817a8f1cac3e4","label":0,"option_order":["prohibited","insufficient","permitted"]}},{"id":"semif_4aa83af95bb69e90bc2d","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The producer authorizes the stage manager to approve props. The stage manager passes the task to an assistant, who has no other approval authority.","questions":{"decision":{"type":"choice","instructions":"The stage manager may delegate prop approval to an assistant, and doing so grants the assistant approval authority. May this assistant approve?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=928ebc7817a8f1cac3e4","state_tokens":28,"semif":{"family":"rule_application","variant":"criterion_reversal","group_id":"928ebc7817a8f1cac3e4","label":0,"option_order":["permitted","prohibited","insufficient"]}},{"id":"semif_cdfa9eed51fafcf09b72","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The producer directly authorizes the assistant to approve props.","questions":{"decision":{"type":"choice","instructions":"The stage manager may approve props but may not delegate that authority. Only authorized approvers may approve. May the assistant approve a prop?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=928ebc7817a8f1cac3e4","state_tokens":11,"semif":{"family":"rule_application","variant":"evidence_change","group_id":"928ebc7817a8f1cac3e4","label":2,"option_order":["insufficient","prohibited","permitted"]}},{"id":"semif_d51b05bfb9d6d7f09331","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"An assistant works on the production, but no approval grants or delegation history are provided.","questions":{"decision":{"type":"choice","instructions":"The stage manager may approve props but may not delegate that authority. Only authorized approvers may approve. May the assistant approve a prop?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=928ebc7817a8f1cac3e4","state_tokens":17,"semif":{"family":"rule_application","variant":"missing","group_id":"928ebc7817a8f1cac3e4","label":1,"option_order":["prohibited","insufficient","permitted"]}},{"id":"semif_fdd902cc6e65fad7cb4f","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The rule permits direct delegation by the director but forbids any further delegation. Candidate A: I received prop approval authority directly from the director. Candidate B: I received it only from A, and have no other grant.","questions":{"decision":{"type":"choice","instructions":"Select the candidate holding valid prop approval authority under the stated delegation rule.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=53b0770ec36835b81d3a","state_tokens":44,"semif":{"family":"candidate_selection","variant":"original","group_id":"53b0770ec36835b81d3a","label":1,"option_order":["B","A","insufficient"]}},{"id":"semif_42384af9ae56beb37ff5","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The rule permits direct delegation by the director but forbids any further delegation. Candidate A: I received prop approval authority directly from the director. Candidate B: I received it only from A, and have no other grant.","questions":{"decision":{"type":"choice","instructions":"Select the candidate whose claimed authority comes only through a prohibited second delegation.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=53b0770ec36835b81d3a","state_tokens":44,"semif":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"53b0770ec36835b81d3a","label":2,"option_order":["insufficient","A","B"]}},{"id":"semif_cdf7f428857e1b841cb3","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The rule permits direct delegation by the director but forbids further delegation. Candidate A: My only claimed authority came from B. Candidate B: The director directly granted me prop approval authority.","questions":{"decision":{"type":"choice","instructions":"Select the candidate holding valid prop approval authority under the stated delegation rule.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=53b0770ec36835b81d3a","state_tokens":37,"semif":{"family":"candidate_selection","variant":"evidence_change","group_id":"53b0770ec36835b81d3a","label":2,"option_order":["A","insufficient","B"]}},{"id":"semif_774f7f9245f83c796d44","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The director's delegation rule is supplied, but Candidate A and Candidate B provide only their names and no authority histories.","questions":{"decision":{"type":"choice","instructions":"Select the candidate holding valid prop approval authority under the stated delegation rule.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=53b0770ec36835b81d3a","state_tokens":23,"semif":{"family":"candidate_selection","variant":"missing","group_id":"53b0770ec36835b81d3a","label":2,"option_order":["B","A","insufficient"]}},{"id":"semif_92ec2ffa3dd46ada1335","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The return policy accepts faulty lamps despite opened packaging, but explicitly excludes water damage even when the lamp is faulty. This opened lamp is faulty because water entered it. No other cause is present.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: this lamp qualifies for return under the stated policy.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=056cc5b726d6e8300a77","state_tokens":38,"semif":{"family":"evidence_interpretation","variant":"original","group_id":"056cc5b726d6e8300a77","label":0,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_6149a17bc154f9c5b4a0","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The return policy accepts faulty lamps despite opened packaging, but explicitly excludes water damage even when the lamp is faulty. This opened lamp is faulty because water entered it. No other cause is present.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the lamp has a fault.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=056cc5b726d6e8300a77","state_tokens":38,"semif":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"056cc5b726d6e8300a77","label":0,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_be866e1dbec5d42d9423","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The return policy accepts faulty lamps despite opened packaging, but excludes water damage even when faulty. This opened lamp has a manufacturing wiring fault, with no water damage or other exclusion.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: this lamp qualifies for return under the stated policy.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=056cc5b726d6e8300a77","state_tokens":35,"semif":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"056cc5b726d6e8300a77","label":2,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_97d2ed2591fc7638806c","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The return policy accepts faulty lamps despite opened packaging but excludes water damage even when faulty. The lamp is faulty, but the cause and presence of water damage are omitted.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: this lamp qualifies for return under the stated policy.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=056cc5b726d6e8300a77","state_tokens":33,"semif":{"family":"evidence_interpretation","variant":"missing","group_id":"056cc5b726d6e8300a77","label":0,"option_order":["insufficient","supported","contradicted"]}},{"id":"semif_4533dd91267b4d815690","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The opened paint tin has a manufacturing defect and also contains added solvent. Both facts are confirmed.","questions":{"decision":{"type":"choice","instructions":"Normally reject opened paint. Accept it for a manufacturing defect, except that added solvent always makes it ineligible. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=e3ce8ec496dbc9974f94","state_tokens":19,"semif":{"family":"rule_application","variant":"original","group_id":"e3ce8ec496dbc9974f94","label":2,"option_order":["permitted","insufficient","prohibited"]}},{"id":"semif_d880e65fef5e2993fdd0","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The opened paint tin has a manufacturing defect and also contains added solvent. Both facts are confirmed.","questions":{"decision":{"type":"choice","instructions":"Accept defective paint even if opened or mixed with solvent; neither condition overrides the defect exception. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=e3ce8ec496dbc9974f94","state_tokens":19,"semif":{"family":"rule_application","variant":"criterion_reversal","group_id":"e3ce8ec496dbc9974f94","label":0,"option_order":["permitted","insufficient","prohibited"]}},{"id":"semif_90494c76078ffc3c717a","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The opened tin has a manufacturing defect and the inspection confirms no solvent was added.","questions":{"decision":{"type":"choice","instructions":"Normally reject opened paint. Accept it for a manufacturing defect, except that added solvent always makes it ineligible. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=e3ce8ec496dbc9974f94","state_tokens":16,"semif":{"family":"rule_application","variant":"evidence_change","group_id":"e3ce8ec496dbc9974f94","label":0,"option_order":["permitted","insufficient","prohibited"]}},{"id":"semif_9ac5cc82c4126c91174a","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The opened tin is defective, but whether solvent was added is not known.","questions":{"decision":{"type":"choice","instructions":"Normally reject opened paint. Accept it for a manufacturing defect, except that added solvent always makes it ineligible. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=e3ce8ec496dbc9974f94","state_tokens":15,"semif":{"family":"rule_application","variant":"missing","group_id":"e3ce8ec496dbc9974f94","label":1,"option_order":["permitted","insufficient","prohibited"]}},{"id":"semif_386e10b1c1ce906473a4","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Accept late submissions for documented outages, except that missing signatures always disqualify. Candidate A: My submission was late because of a documented outage and includes the required signature. Candidate B: Mine was late because of a documented outage but lacks the signature.","questions":{"decision":{"type":"choice","instructions":"Select the submission qualifying under the stated exception and exclusion.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=a5b234c2e4fb304f1071","state_tokens":51,"semif":{"family":"candidate_selection","variant":"original","group_id":"a5b234c2e4fb304f1071","label":1,"option_order":["B","A","insufficient"]}},{"id":"semif_4da6aedc1139224eeff2","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Accept late submissions for documented outages, except that missing signatures always disqualify. Candidate A: My submission was late because of a documented outage and includes the required signature. Candidate B: Mine was late because of a documented outage but lacks the signature.","questions":{"decision":{"type":"choice","instructions":"Select the submission disqualified specifically by the signature exclusion despite the outage exception.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=a5b234c2e4fb304f1071","state_tokens":51,"semif":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"a5b234c2e4fb304f1071","label":1,"option_order":["insufficient","B","A"]}},{"id":"semif_a22dab8ba073ff6cdb8d","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Accept late submissions for documented outages, except that missing signatures always disqualify. Candidate A: My late outage-related submission lacks the signature. Candidate B: My late outage-related submission has the signature. Both outages are documented.","questions":{"decision":{"type":"choice","instructions":"Select the submission qualifying under the stated exception and exclusion.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=a5b234c2e4fb304f1071","state_tokens":47,"semif":{"family":"candidate_selection","variant":"evidence_change","group_id":"a5b234c2e4fb304f1071","label":1,"option_order":["insufficient","B","A"]}},{"id":"semif_42c54fa2ad45743e24d8","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Accept late submissions for documented outages, except that missing signatures always disqualify. Candidate A and Candidate B report late submissions but provide neither outage documentation nor signature information.","questions":{"decision":{"type":"choice","instructions":"Select the submission qualifying under the stated exception and exclusion.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=a5b234c2e4fb304f1071","state_tokens":34,"semif":{"family":"candidate_selection","variant":"missing","group_id":"a5b234c2e4fb304f1071","label":1,"option_order":["A","insufficient","B"]}},{"id":"semif_f3d13cf184f916677def","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The desk log says the crate is in storage. The warehouse inventory says it has left storage. Both reports are current. The supplied protocol says the warehouse inventory controls location decisions whenever these two sources conflict.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: under the supplied protocol, the crate is currently in storage.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=d142f7fd0efa0eddc788","state_tokens":40,"semif":{"family":"evidence_interpretation","variant":"original","group_id":"d142f7fd0efa0eddc788","label":0,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_1105577da4c8dad4609d","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The desk log says the crate is in storage. The warehouse inventory says it has left storage. Both reports are current. The supplied protocol says the warehouse inventory controls location decisions whenever these two sources conflict.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the desk log reports the crate in storage.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=d142f7fd0efa0eddc788","state_tokens":40,"semif":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"d142f7fd0efa0eddc788","label":2,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_0ebd299739302d8b057b","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The desk log says the crate has left storage; the warehouse inventory says it is in storage. Both reports are current, and the inventory controls conflicts under the protocol.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: under the supplied protocol, the crate is currently in storage.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=d142f7fd0efa0eddc788","state_tokens":33,"semif":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"d142f7fd0efa0eddc788","label":2,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_eafc22c8c40df3932a8e","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The protocol gives the inventory priority, but the current inventory and desk-log entries for the crate are omitted.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: under the supplied protocol, the crate is currently in storage.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=d142f7fd0efa0eddc788","state_tokens":21,"semif":{"family":"evidence_interpretation","variant":"missing","group_id":"d142f7fd0efa0eddc788","label":2,"option_order":["supported","contradicted","insufficient"]}},{"id":"semif_0aa1447680e4f25862bb","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The desk clerk reports the equipment deposit as paid. The signed ledger reports it as unpaid. Both reports are current.","questions":{"decision":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; in conflicts, the signed ledger overrides the desk clerk. May it be released?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=3476d3f4f32290fe57c3","state_tokens":23,"semif":{"family":"rule_application","variant":"original","group_id":"3476d3f4f32290fe57c3","label":1,"option_order":["insufficient","prohibited","permitted"]}},{"id":"semif_980e76145c925747220c","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The desk clerk reports the equipment deposit as paid. The signed ledger reports it as unpaid. Both reports are current.","questions":{"decision":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; under this alternative protocol, the desk clerk overrides the ledger. May it be released?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=3476d3f4f32290fe57c3","state_tokens":23,"semif":{"family":"rule_application","variant":"criterion_reversal","group_id":"3476d3f4f32290fe57c3","label":2,"option_order":["insufficient","prohibited","permitted"]}},{"id":"semif_4aee37a6a155747766aa","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The desk clerk reports the deposit unpaid, while the signed ledger reports it paid. Both reports are current.","questions":{"decision":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; in conflicts, the signed ledger overrides the desk clerk. May it be released?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=3476d3f4f32290fe57c3","state_tokens":21,"semif":{"family":"rule_application","variant":"evidence_change","group_id":"3476d3f4f32290fe57c3","label":0,"option_order":["permitted","prohibited","insufficient"]}},{"id":"semif_6db17cc22bd656d78558","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The record mentions the desk clerk and ledger but supplies neither deposit status.","questions":{"decision":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; in conflicts, the signed ledger overrides the desk clerk. May it be released?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=3476d3f4f32290fe57c3","state_tokens":14,"semif":{"family":"rule_application","variant":"missing","group_id":"3476d3f4f32290fe57c3","label":2,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_ef3e0f8e657bf59f8a29","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"For this task, the current signed checklist controls readiness when it conflicts with chat. Candidate A: Chat says ready, but the signed checklist says not ready. Candidate B: Chat says not ready, but the signed checklist says ready.","questions":{"decision":{"type":"choice","instructions":"Select the candidate ready according to the controlling signed checklist.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=original group=c979db7416b206c1eb09","state_tokens":46,"semif":{"family":"candidate_selection","variant":"original","group_id":"c979db7416b206c1eb09","label":2,"option_order":["A","insufficient","B"]}},{"id":"semif_f756e1cb04e11766341e","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"For this task, the current signed checklist controls readiness when it conflicts with chat. Candidate A: Chat says ready, but the signed checklist says not ready. Candidate B: Chat says not ready, but the signed checklist says ready.","questions":{"decision":{"type":"choice","instructions":"Under the alternative protocol, chat controls conflicts. Select the candidate ready according to chat.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=criterion_reversal group=c979db7416b206c1eb09","state_tokens":46,"semif":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"c979db7416b206c1eb09","label":1,"option_order":["insufficient","A","B"]}},{"id":"semif_d167e51bb1f499099645","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The signed checklist controls conflicts. Candidate A: The checklist says ready and chat says not ready. Candidate B: The checklist says not ready and chat says ready.","questions":{"decision":{"type":"choice","instructions":"Select the candidate ready according to the controlling signed checklist.","criteria":{"A":"Candidate A","B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=evidence_change group=c979db7416b206c1eb09","state_tokens":32,"semif":{"family":"candidate_selection","variant":"evidence_change","group_id":"c979db7416b206c1eb09","label":0,"option_order":["A","B","insufficient"]}},{"id":"semif_674ca4e9346216255466","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The signed checklist controls conflicts, but Candidate A and Candidate B provide only project names without checklist or chat status.","questions":{"decision":{"type":"choice","instructions":"Select the candidate ready according to the controlling signed checklist.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=c979db7416b206c1eb09","state_tokens":22,"semif":{"family":"candidate_selection","variant":"missing","group_id":"c979db7416b206c1eb09","label":0,"option_order":["insufficient","B","A"]}},{"id":"semif_1df851a202994a07f123","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The agreed handover requires both the source files and the user guide. The delivery record confirms the source files arrived and explicitly says the guide was not delivered.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the agreed handover is complete.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=f7403e39d6796072491c","state_tokens":31,"semif":{"family":"evidence_interpretation","variant":"original","group_id":"f7403e39d6796072491c","label":0,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_b5e6788d95b8395d2888","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The agreed handover requires both the source files and the user guide. The delivery record confirms the source files arrived and explicitly says the guide was not delivered.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the source files were delivered.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=f7403e39d6796072491c","state_tokens":31,"semif":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"f7403e39d6796072491c","label":2,"option_order":["insufficient","contradicted","supported"]}},{"id":"semif_32f36cd0fbbfb3604586","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The agreed handover requires the source files and user guide. The delivery record confirms that both arrived.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the agreed handover is complete.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=f7403e39d6796072491c","state_tokens":20,"semif":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"f7403e39d6796072491c","label":0,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_843185706adff0c60f8b","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The agreement requires source files and a user guide, but the delivery record gives no information about either item.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the agreed handover is complete.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=f7403e39d6796072491c","state_tokens":21,"semif":{"family":"evidence_interpretation","variant":"missing","group_id":"f7403e39d6796072491c","label":2,"option_order":["contradicted","supported","insufficient"]}},{"id":"semif_b531dd603bf080dab820","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The contractor supplied the translated manual but explicitly did not supply the glossary.","questions":{"decision":{"type":"choice","instructions":"Approve final handover exactly when both the translated manual and glossary have been supplied. May final handover be approved?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=a060fd4621be415e9f1a","state_tokens":15,"semif":{"family":"rule_application","variant":"original","group_id":"a060fd4621be415e9f1a","label":0,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_be2dd67bc733c1213bef","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The contractor supplied the translated manual but explicitly did not supply the glossary.","questions":{"decision":{"type":"choice","instructions":"Approve the translation milestone when the manual has been supplied; the glossary is not required for this milestone. May this milestone be approved?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=a060fd4621be415e9f1a","state_tokens":15,"semif":{"family":"rule_application","variant":"criterion_reversal","group_id":"a060fd4621be415e9f1a","label":1,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_b8e2750307df9761e716","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The contractor supplied both the translated manual and the glossary.","questions":{"decision":{"type":"choice","instructions":"Approve final handover exactly when both the translated manual and glossary have been supplied. May final handover be approved?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=a060fd4621be415e9f1a","state_tokens":12,"semif":{"family":"rule_application","variant":"evidence_change","group_id":"a060fd4621be415e9f1a","label":0,"option_order":["permitted","prohibited","insufficient"]}},{"id":"semif_5d31b4bf2d0fc193f53b","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The contractor reports working on the handover but supplies no delivery record for either item.","questions":{"decision":{"type":"choice","instructions":"Approve final handover exactly when both the translated manual and glossary have been supplied. May final handover be approved?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=a060fd4621be415e9f1a","state_tokens":17,"semif":{"family":"rule_application","variant":"missing","group_id":"a060fd4621be415e9f1a","label":1,"option_order":["permitted","insufficient","prohibited"]}},{"id":"semif_a9230edfc6ec9e524af0","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"A complete incident package requires both a timeline and a remedy report. Candidate A: Both documents are attached. Candidate B: Only the timeline is attached; the remedy report is absent.","questions":{"decision":{"type":"choice","instructions":"Select the complete incident package.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=2b0f8eda8903e38a47de","state_tokens":36,"semif":{"family":"candidate_selection","variant":"original","group_id":"2b0f8eda8903e38a47de","label":2,"option_order":["B","insufficient","A"]}},{"id":"semif_c100e84585149a04be49","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"A complete incident package requires both a timeline and a remedy report. Candidate A: Both documents are attached. Candidate B: Only the timeline is attached; the remedy report is absent.","questions":{"decision":{"type":"choice","instructions":"Select the package containing the timeline but missing the remedy report.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=2b0f8eda8903e38a47de","state_tokens":36,"semif":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"2b0f8eda8903e38a47de","label":0,"option_order":["B","A","insufficient"]}},{"id":"semif_579beed184ffc1104aa9","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"A complete package requires both documents. Candidate A: The timeline is attached but the remedy report is absent. Candidate B: Both the timeline and remedy report are attached.","questions":{"decision":{"type":"choice","instructions":"Select the complete incident package.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=2b0f8eda8903e38a47de","state_tokens":33,"semif":{"family":"candidate_selection","variant":"evidence_change","group_id":"2b0f8eda8903e38a47de","label":0,"option_order":["B","A","insufficient"]}},{"id":"semif_c4e9bd463b6fde5be743","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"A complete package requires both documents. Candidate A and Candidate B give only package titles and no attachment information.","questions":{"decision":{"type":"choice","instructions":"Select the complete incident package.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=2b0f8eda8903e38a47de","state_tokens":21,"semif":{"family":"candidate_selection","variant":"missing","group_id":"2b0f8eda8903e38a47de","label":1,"option_order":["B","insufficient","A"]}},{"id":"semif_b863998d4be2721a5dfd","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The routing rule assigns repairs to the in-house workshop when it can perform the work; use the outside workshop only when the in-house workshop cannot. Both workshops confirm they can perform this repair.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the routing rule assigns this repair to the outside workshop.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=01020a1eaf46ec2fba5d","state_tokens":38,"semif":{"family":"evidence_interpretation","variant":"original","group_id":"01020a1eaf46ec2fba5d","label":2,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_37cebc3792e98049439f","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The routing rule assigns repairs to the in-house workshop when it can perform the work; use the outside workshop only when the in-house workshop cannot. Both workshops confirm they can perform this repair.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the outside workshop can perform this repair.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=01020a1eaf46ec2fba5d","state_tokens":38,"semif":{"family":"evidence_interpretation","variant":"criterion_reversal","group_id":"01020a1eaf46ec2fba5d","label":1,"option_order":["insufficient","supported","contradicted"]}},{"id":"semif_64d63f86e328aff1481c","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The routing rule assigns repairs to the in-house workshop when it can perform the work; use the outside workshop only when the in-house workshop cannot. The in-house workshop cannot perform this repair; the outside workshop can.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the routing rule assigns this repair to the outside workshop.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=01020a1eaf46ec2fba5d","state_tokens":43,"semif":{"family":"evidence_interpretation","variant":"evidence_change","group_id":"01020a1eaf46ec2fba5d","label":2,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_6d64a8459f64eac7f4cc","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The routing rule assigns repairs to the in-house workshop when it can perform the work; use the outside workshop only when the in-house workshop cannot. Neither workshop's ability to perform this repair is supplied.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the routing rule assigns this repair to the outside workshop.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=01020a1eaf46ec2fba5d","state_tokens":40,"semif":{"family":"evidence_interpretation","variant":"missing","group_id":"01020a1eaf46ec2fba5d","label":0,"option_order":["insufficient","contradicted","supported"]}},{"id":"semif_2708193212d8a4a523c7","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The town hall and school both confirm they are available and suitable for the meeting.","questions":{"decision":{"type":"choice","instructions":"Book the school only as a fallback when the town hall is unavailable or unsuitable; otherwise book the town hall. May the school be booked under this policy?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=6a06a2f278301c1412ee","state_tokens":16,"semif":{"family":"rule_application","variant":"original","group_id":"6a06a2f278301c1412ee","label":0,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_5a37d1dfcad20810a76a","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The town hall and school both confirm they are available and suitable for the meeting.","questions":{"decision":{"type":"choice","instructions":"Under the alternative policy, either suitable available venue may be booked without priority. May the school be booked?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=6a06a2f278301c1412ee","state_tokens":16,"semif":{"family":"rule_application","variant":"criterion_reversal","group_id":"6a06a2f278301c1412ee","label":1,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_acacd8b46acd25cb954b","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The town hall is confirmed unavailable, while the school is available and suitable.","questions":{"decision":{"type":"choice","instructions":"Book the school only as a fallback when the town hall is unavailable or unsuitable; otherwise book the town hall. May the school be booked under this policy?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=6a06a2f278301c1412ee","state_tokens":15,"semif":{"family":"rule_application","variant":"evidence_change","group_id":"6a06a2f278301c1412ee","label":1,"option_order":["insufficient","permitted","prohibited"]}},{"id":"semif_4089c199b8b796ec3ed8","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"The two venues are named, but neither availability nor suitability is provided.","questions":{"decision":{"type":"choice","instructions":"Book the school only as a fallback when the town hall is unavailable or unsuitable; otherwise book the town hall. May the school be booked under this policy?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=6a06a2f278301c1412ee","state_tokens":14,"semif":{"family":"rule_application","variant":"missing","group_id":"6a06a2f278301c1412ee","label":0,"option_order":["insufficient","prohibited","permitted"]}},{"id":"semif_02f2b95a5c5a4777a45c","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A is the preferred translator and is confirmed able to handle this language. Candidate B is the fallback translator and is also able to handle it.","questions":{"decision":{"type":"choice","instructions":"Choose A whenever A is able; choose B only when A is unable and B is able.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=2797f8c23d05c0ee06b3","state_tokens":29,"semif":{"family":"candidate_selection","variant":"original","group_id":"2797f8c23d05c0ee06b3","label":1,"option_order":["B","A","insufficient"]}},{"id":"semif_ebea4681e3dcc7d92ca2","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A is the preferred translator and is confirmed able to handle this language. Candidate B is the fallback translator and is also able to handle it.","questions":{"decision":{"type":"choice","instructions":"Under the alternative priority, choose B whenever B is able; choose A only when B is unable and A is able.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=2797f8c23d05c0ee06b3","state_tokens":29,"semif":{"family":"candidate_selection","variant":"criterion_reversal","group_id":"2797f8c23d05c0ee06b3","label":1,"option_order":["insufficient","B","A"]}},{"id":"semif_b8c5b9b285cbdddc9f8d","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A is the preferred translator but is confirmed unable to handle this language. Candidate B is the fallback and is confirmed able.","questions":{"decision":{"type":"choice","instructions":"Choose A whenever A is able; choose B only when A is unable and B is able.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=2797f8c23d05c0ee06b3","state_tokens":25,"semif":{"family":"candidate_selection","variant":"evidence_change","group_id":"2797f8c23d05c0ee06b3","label":2,"option_order":["A","insufficient","B"]}},{"id":"semif_7ed75d854d99b3a81916","set":"fixture","source":"semif_authored144","request":{"model":"clef-flash","state":"Candidate A is preferred and Candidate B is the fallback, but neither candidate's ability to handle the language is supplied.","questions":{"decision":{"type":"choice","instructions":"Choose A whenever A is able; choose B only when A is unable and B is able.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=2797f8c23d05c0ee06b3","state_tokens":23,"semif":{"family":"candidate_selection","variant":"missing","group_id":"2797f8c23d05c0ee06b3","label":0,"option_order":["insufficient","B","A"]}},{"id":"ho_t01","set":"heldout","source":"heldout_text","request":{"model":"clef-flash","state":"Hi Bertrand, I would like to book annual leave from Monday 9 November to Friday 13 November, five working days, to attend my sister's wedding abroad. I still have 12 days of leave for this year, so 7 would remain afterwards. Wiebke Antonelli has agreed to take over the weekly supplier calls while I am away, and I will hand over the open purchase orders to her on Friday 6 November. I can be reached by phone in a real emergency, but I would rather not check messages. Could you let me know by the end of next week, so that I can book my flights? Many thanks, Tamsin Okonkwo","questions":{"leave_days":{"type":"choice","instructions":"How many working days of leave are requested?","criteria":{"3":"three","4":"four","5":"five","6":"six"}},"cover_arranged":{"type":"noul","instructions":"Someone has agreed to look after the requester's duties during the leave."},"days_left_after":{"type":"choice","instructions":"How many leave days will the requester have left afterwards?","criteria":{"5":"5 days","7":"7 days","12":"12 days","17":"17 days"}},"reason":{"type":"choice","instructions":"Why is the leave requested?","criteria":{"family_event":"To attend a family event","medical":"For medical treatment or illness","study":"To sit an exam or attend a course","moving":"To move house"}},"ease_of_approval":{"type":"score","instructions":"How easy would it be for a manager to approve this request?","criteria":["hard","needs a conversation","straightforward"]}}},"gold":{"leave_days":"5","cover_arranged":"true","days_left_after":"7","reason":"family_event","ease_of_approval":null},"note":"leave request","state_tokens":139},{"id":"ho_t02","set":"heldout","source":"heldout_text","request":{"model":"clef-flash","state":"Warranty claim W-58213, received 28 September 2026. Customer: Ilse Barrowclough. Product: Tolverack TV-210 cordless vacuum cleaner, bought new on 2 February 2025 with a two-year manufacturer's warranty. Reported fault: the battery no longer holds its charge; after a whole night on the dock the cleaner runs for less than four minutes, against the forty minutes stated in the manual. She has kept the original receipt and quotes serial number TV210-77K-0419. She mentions that the cleaner fell down a flight of stairs last summer, but says it kept working normally for months after that. She asks for a replacement battery to be posted to her home address, because she no longer has a box to send the whole cleaner back in.","questions":{"in_warranty":{"type":"noul","instructions":"On the date the claim was received, the product is still inside its warranty period."},"faulty_part":{"type":"choice","instructions":"Which part does the customer say is faulty?","criteria":{"battery":"The battery","motor":"The motor","brush":"The brush roll","dock":"The charging dock","filter":"The dust filter"}},"possible_damage":{"type":"noul","instructions":"Could the product have been damaged by the customer?","criteria":{"true":"The customer mentions an accident such as a fall, a spill or a knock.","false":"No accident or rough handling is mentioned."}},"remedy":{"type":"choice","instructions":"What does the customer ask for?","criteria":{"refund":"Her money back","part_by_post":"A replacement part posted to her","repair":"A repair after sending the product in","new_unit":"A complete new product"}},"claim_strength":{"type":"score","instructions":"How strong is the claim?","criteria":["weak","uncertain","strong"]}}},"gold":{"in_warranty":"true","faulty_part":"battery","possible_damage":"true","remedy":"part_by_post","claim_strength":null},"note":"warranty claim; noul criteria override (both keys)","state_tokens":179},{"id":"ho_t03","set":"heldout","source":"heldout_text","request":{"model":"clef-flash","state":"Dear Corvassen Lettings, I saw your advert for the two-bedroom flat on the third floor of Ellerwyke House (reference CL-2207) and would like to arrange a viewing. I work night shifts as a nurse, so weekday mornings after 10:00 suit me best. I would be moving in with my partner and our elderly greyhound. The advert says small pets only, so please tell me whether a calm 30 kg dog would be accepted. We do not own a car, so the lack of a parking space is no problem for us. Could you also confirm whether the rent of 1,450 a month includes heating, and how large the deposit is? We hope to move in on 1 December. Kind regards, Jovan Petrakis","questions":{"asks_about_pet":{"type":"noul","instructions":"The writer asks whether an animal would be allowed in the flat."},"bedrooms":{"type":"choice","criteria":{"1":"one bedroom","2":"two bedrooms","3":"three bedrooms","studio":"a studio"}},"needs_parking":{"type":"noul","instructions":"The writer needs a parking space."},"viewing_time":{"type":"choice","instructions":"When would the writer like to view the flat?","criteria":{"weekday_morning":"On a weekday morning","weekday_evening":"On a weekday evening","weekend":"At the weekend","any":"No preference is given"}},"other_question":{"type":"choice","instructions":"Apart from pets, what else does the writer ask about?","criteria":{"rent_deposit":"What the rent covers and the size of the deposit","garden":"Use of a garden","lease_length":"How long the tenancy lasts","internet":"The internet connection"}},"match":{"type":"score","instructions":"How well does the flat seem to suit the writer?","criteria":["poorly","partly","well","very well"]}}},"gold":{"asks_about_pet":"true","bedrooms":"2","needs_parking":"false","viewing_time":"weekday_morning","other_question":"rent_deposit","match":null},"note":"rental enquiry; a choice without instructions","state_tokens":164},{"id":"ho_t04","set":"heldout","source":"heldout_text","request":{"model":"clef-flash","state":"Feedback on the Grallowmere Food Fair, Saturday. The tasting tents were the best part of the day: the cheese stall and the smoked fish stall both sold out by early afternoon. Parking was a mess. The overflow field opened an hour late and the queue of cars stretched back to the main road, so we spent forty minutes just getting in. Entry was 8 per adult, which felt fair, but charging children over five the full adult price did not. The cookery demonstrations started on time, although the speakers in the demo tent kept cutting out and we missed half of the talk on fermenting vegetables. The toilets stayed clean all day. We would come back next year if the parking is sorted out. Rosalind Achterberg","questions":{"biggest_problem":{"type":"choice","instructions":"What was the biggest problem for this visitor?","criteria":{"parking":"Parking and getting in","food":"The quality of the food","price":"The entry price","toilets":"The toilets","weather":"The weather"}},"would_return":{"type":"noul","instructions":"Would the visitor come again?","criteria":{"true":"The visitor would attend again, possibly on a condition.","false":"The visitor would not attend again."}},"food_left":{"type":"noul","instructions":"Every stall still had food at the end of the day."},"tone":{"type":"score","instructions":"How favourable is the feedback overall?","criteria":["hostile","very critical","critical","mixed","favourable","very favourable","glowing"]}}},"gold":{"biggest_problem":"parking","would_return":"true","food_left":"false","tone":null},"note":"event feedback; noul criteria override (both keys); a 7-level score","state_tokens":149},{"id":"ho_t05","set":"heldout","source":"heldout_text","request":{"model":"clef-flash","state":"Bug report, Calderwisp Notes 3.8.2 on a tablet with the latest system update. Steps: open a notebook that has more than 200 pages, switch to the outline view, then turn the tablet to landscape. Expected: the outline stays open on the same page. Actual: the app freezes for about ten seconds and then closes, and the last two minutes of typing are gone. It happens every time with large notebooks and never with small ones. Version 3.7.9 did not behave like this. Workaround: lock the screen rotation before opening the outline. Reported by Ezinne Valdés-Holt.","questions":{"defect_kind":{"type":"choice","instructions":"What kind of defect is this?","criteria":{"crash":{"symptom":"the app stops responding or closes","usual_cause":"an unhandled error"},"display":{"symptom":"something is drawn in the wrong place","usual_cause":"a layout mistake"},"sync":{"symptom":"changes do not reach other devices","usual_cause":"a server or network fault"},"slowness":{"symptom":"the app is slow but keeps working","usual_cause":"inefficient processing"}}},"regression":{"type":"noul","instructions":"The problem did not exist in an earlier version."},"loses_work":{"type":"noul","instructions":"Some of the user's work is lost when the problem happens."},"small_notebooks":{"type":"noul","instructions":"The problem also occurs with small notebooks."},"priority":{"type":"score","instructions":"What priority should the fix get?","criteria":["P4 backlog","P3 normal","P2 high","P1 urgent"]}}},"gold":{"defect_kind":"crash","regression":"true","loses_work":"true","small_notebooks":"false","priority":null},"note":"bug report; option descriptions are JSON objects","state_tokens":131},{"id":"ho_t06","set":"heldout","source":"heldout_text","request":{"model":"clef-flash","state":"Order 66-1904, sent with Pendrithy Parcels for next-day delivery. The tracking page says 'delivered, left in a safe place' at 14:52 yesterday, but there was nothing at my door, in the porch or with the neighbours on either side. I was working from home all afternoon and nobody rang the bell. The parcel holds a replacement phone screen worth 89, and I need it for a repair I have booked on Thursday. The caretaker of our building has not received anything either. Please investigate, and either find the parcel or send a new one before Thursday. I do not want my money back. Augustin Lemaire-Kovac","questions":{"problem":{"type":"choice","instructions":"What went wrong?","criteria":{"late":"The parcel is late","damaged":"The parcel arrived damaged","missing_after_scan":"Tracking says delivered but nothing arrived","wrong_item":"The wrong item arrived","redirected":"The parcel was sent to another address on request"}},"was_home":{"type":"noul","instructions":"The customer was at home when the delivery was recorded."},"outcome_wanted":{"type":"choice","instructions":"What outcome does the customer want?","criteria":{"refund":"A refund","resend":"The parcel found or sent again","compensation":"Extra compensation","cancel":"The order cancelled"}},"time_sensitive":{"type":"noul"},"mood":{"type":"score","instructions":"How upset does the customer sound?","criteria":["calm","concerned","upset","furious"]}}},"gold":{"problem":"missing_after_scan","was_home":"true","outcome_wanted":"resend","time_sensitive":"true","mood":null},"note":"delivery problem; a noul without instructions","state_tokens":146},{"id":"ho_t07","set":"heldout","source":"heldout_text","request":{"model":"clef-flash","state":"I am scaling up my grandmother's flapjack recipe for a school bake sale. The original makes one tray of 12 squares: 250 g oats, 125 g butter, 100 g brown sugar, 4 tablespoons of golden syrup and a pinch of salt, baked at 180 °C for 25 minutes. I need 36 squares, which means three trays. My kitchen scales only go up to 500 g and I have lost my measuring spoons, but I know that one tablespoon of syrup weighs about 20 g. Should the baking time change if I bake the trays one after another in the same oven? And can I replace half of the butter with sunflower oil without the bars falling apart?","questions":{"factor":{"type":"choice","instructions":"By what factor must the quantities be multiplied?","criteria":{"1.5":"1.5","2":"2","3":"3","4":"4","6":"6"}},"oats_total":{"type":"choice","instructions":"How much oats is needed in total?","criteria":{"500":"500 g","750":"750 g","1000":"1,000 g","1250":"1,250 g"}},"syrup_total":{"type":"choice","instructions":"How many grams of golden syrup are needed in total?","criteria":{"60":"60 g","80":"80 g","240":"240 g","320":"320 g"}},"oats_one_weighing":{"type":"noul","instructions":"All of the oats can be weighed in one go on the cook's scales."},"asks_swap":{"type":"noul","instructions":"The cook asks about replacing one ingredient with another."},"oven_temperature":{"type":"choice","instructions":"At what temperature are the trays baked?","criteria":{"160":"160 °C","170":"170 °C","180":"180 °C","200":"200 °C"}},"questions_asked":{"type":"choice","instructions":"How many questions does the message ask?","criteria":{"1":"one","2":"two","3":"three","4":"four"}},"difficulty":{"type":"score","instructions":"How tricky is this scaling job?","criteria":["trivial","easy","moderate","tricky"]}}},"gold":{"factor":"3","oats_total":"750","syrup_total":"240","oats_one_weighing":"false","asks_swap":"true","oven_temperature":"180","questions_asked":"2","difficulty":null},"note":"recipe scaling; the 8-question record","state_tokens":159},{"id":"ho_t08","set":"heldout","source":"heldout_text","request":{"model":"clef-flash","state":"Policy SM-447120, Sallowmead Travel Cover, single trip from 3 to 17 August. My connecting flight home was cancelled because of a strike, and I had to stay two extra nights in a hotel before the airline could rebook me. I paid 236 for the hotel and 41 for meals and I have every receipt. The airline refused to pay, saying a strike is outside its control. My policy wording covers 'travel delay of more than 12 hours' up to 300, with an excess of 50 per claim, and excludes 'costs that another company has agreed to refund'. My suitcase also arrived a day late, but nothing in it was lost. What can I claim, and do I need a written statement from the airline? Florentyna Abernathy","questions":{"policy_section":{"type":"choice","instructions":"Which section of the policy fits the main claim?","criteria":{"medical":"Emergency medical costs","delay":"Travel delay","baggage":"Lost or stolen baggage","cancellation":"Cancelling the trip before leaving","liability":"Damage to other people or their property"}},"costs_total":{"type":"choice","instructions":"What do the hotel and meal costs add up to?","criteria":{"236":"236","277":"277","286":"286","300":"300"}},"payout":{"type":"choice","instructions":"If the claim is accepted in full, how much will the insurer pay?","criteria":{"227":"227","250":"250","277":"277","300":"300"}},"airline_paid":{"type":"noul","instructions":"The airline has agreed to refund the costs."},"baggage_lost":{"type":"noul","instructions":"Some of the traveller's baggage was lost for good."},"clarity":{"type":"score","criteria":["unclear","mostly clear","very clear"]}}},"gold":{"policy_section":"delay","costs_total":"277","payout":"227","airline_paid":"false","baggage_lost":"false","clarity":null},"note":"travel insurance question; a score without instructions","state_tokens":175},{"id":"ho_t09","set":"heldout","source":"heldout_text","request":{"model":"clef-flash","state":"October timetable, Kinnerwold Learning Centre. Ottilie Varga teaches maths on Mondays and Wednesdays from 16:00 to 17:30 in room 2. Casimir Oyelaran teaches chemistry on Tuesdays from 17:00 to 18:00 in room 5, and gives the same lesson online on Thursdays at 18:00. Each pupil may book at most three sessions a week. Bartholomew (Year 10) has football training every Tuesday from 16:30 to 18:30 and wants two maths sessions and one chemistry session each week. Ingrid (Year 12) is free every day after 15:30 but has no laptop at home. The centre is closed on Wednesday 21 October for staff training; the lessons of that day move to Friday 23 October at the same times.","questions":{"bartholomew_chemistry":{"type":"choice","instructions":"Which chemistry session can Bartholomew attend?","criteria":{"tuesday":"Tuesday in room 5","thursday":"Thursday online","neither":"Neither"}},"ingrid_laptop":{"type":"noul","instructions":"Ingrid has a laptop at home."},"within_limit":{"type":"noul","instructions":"Bartholomew's wished-for sessions stay within the weekly booking limit."},"moved_lesson":{"type":"choice","instructions":"What happens to the maths lesson of Wednesday 21 October?","criteria":{"cancelled":"It is cancelled","fri_23":"It moves to Friday 23 October, same time","thu_22":"It moves to Thursday 22 October","online":"It is held online that day"}},"maths_length":{"type":"choice","instructions":"How long is one maths session?","criteria":{"60":"60 minutes","75":"75 minutes","90":"90 minutes","120":"120 minutes"}},"planning_effort":{"type":"score","instructions":"How hard is this timetable to plan?","criteria":["simple","manageable","complicated"]}}},"gold":{"bartholomew_chemistry":"thursday","ingrid_laptop":"false","within_limit":"true","moved_lesson":"fri_23","maths_length":"90","planning_effort":null},"note":"tutoring schedule","state_tokens":197},{"id":"ho_t10","set":"heldout","source":"heldout_text","request":{"model":"clef-flash","state":"Replies to the September issue of the Yarrowdeep Bulletin.\nCosima: Loved the article about the old mill. Please print more local history.\nBarnaby: Far too long this month. I gave up after page six.\nYevgenia: Thanks for including the new bin collection days. Two of the crossword answers were wrong, though.\nLaszlo: More local history, please! The photographs were lovely too.\nHenrike: Could you stop posting it to me on paper? I would rather read it on my screen.\nCosima: I agree with Henrike, the paper copies get soaked in my letterbox.","questions":{"most_wanted":{"type":"choice","instructions":"Which topic do readers ask to see more of?","criteria":{"local_history":"Local history","recipes":"Recipes","sport":"Sport","puzzles":"Puzzles","council":"Council news"}},"mistake_reported":{"type":"noul","instructions":"A reader points out an error in the issue."},"all_positive":{"type":"noul","instructions":"Every reply is positive."},"digital_readers":{"type":"choice","instructions":"How many different readers want a digital copy instead of paper?","criteria":{"0":"none","1":"one","2":"two","3":"three"}},"reply_count":{"type":"score","instructions":"How many replies are listed?","criteria":["one or two","three or four","five or six","more than six"]},"reception":{"type":"score","instructions":"How well was the issue received overall?","criteria":["badly","with mixed feelings","well","very well"]}}},"gold":{"most_wanted":"local_history","mistake_reported":"true","all_positive":"false","digital_readers":"2","reply_count":"2","reception":null},"note":"newsletter replies","state_tokens":134},{"id":"ho_j01","set":"heldout","source":"heldout_json","request":{"model":"clef-flash","state":{"timesheet":{"employee":"Ludovica Ferreira","week_starting":"2026-09-28","contract_hours":37.5,"days":[{"date":"2026-09-28","in":"08:58","out":"17:30","break_min":30},{"date":"2026-09-29","in":"09:05","out":"17:35","break_min":30},{"date":"2026-09-30","status":"sick"},{"date":"2026-10-01","in":"08:45","out":"19:15","break_min":45},{"date":"2026-10-02","in":"09:00","out":"13:00","break_min":0,"note":"half-day leave"}],"approved_by":null}},"questions":{"sick_days":{"type":"choice","instructions":"How many sick days are recorded?","criteria":{"0":"none","1":"one","2":"two","3":"three"}},"longest_day":{"type":"choice","instructions":"On which date was the longest shift worked?","criteria":{"2026-09-28":"Monday","2026-09-29":"Tuesday","2026-10-01":"Thursday","2026-10-02":"Friday"}},"approved":{"type":"noul","instructions":"Someone has approved the timesheet."},"late_start":{"type":"noul","instructions":"Did the employee start late on any day?","criteria":{"true":"On at least one day the clock-in time is later than 09:00.","false":"Every clock-in time is 09:00 or earlier."}},"hours_concern":{"type":"score","instructions":"How concerning are the hours worked this week?","criteria":["no concern","minor concern","serious concern"]}}},"gold":{"sick_days":"1","longest_day":"2026-10-01","approved":"false","late_start":"true","hours_concern":null},"note":"timesheet; noul criteria override (both keys)","state_tokens":206},{"id":"ho_j02","set":"heldout","source":"heldout_json","request":{"model":"clef-flash","state":{"stock_count":{"store":"Hollindrake Hardware","counted_on":"2026-09-27","tolerance_units":3,"lines":[{"sku":"HH-HAMMER-16","expected":42,"counted":42},{"sku":"HH-TAPE-5M","expected":120,"counted":113},{"sku":"HH-GLOVES-L","expected":60,"counted":66},{"sku":"HH-DRILLBITS","expected":18,"counted":9},{"sku":"HH-SEALANT-W","expected":35,"counted":35}]}},"questions":{"beyond_tolerance":{"type":"choice","instructions":"How many lines differ from the expected count by more than the tolerance?","criteria":{"0":"none","1":"one","2":"two","3":"three","4":"four","5":"five"}}}},"gold":{"beyond_tolerance":"3"},"note":"inventory difference (expected vs counted); the 1-question record","state_tokens":137},{"id":"ho_j03","set":"heldout","source":"heldout_json","request":{"model":"clef-flash","state":{"log":"vaccine fridge B","unit":"°C","allowed_range":[2,8],"entries":[{"time":"07:30","reading":4.1,"by":"TO"},{"time":"11:30","reading":5.0,"by":"TO"},{"time":"15:30","reading":8.9,"by":"RA","comment":"door found ajar after delivery"},{"time":"16:00","reading":7.2,"by":"RA"},{"time":"19:30","reading":3.6,"by":"RA"}],"since_last_reset":{"max":9.4,"min":3.1}},"questions":{"excursion":{"type":"noul","instructions":"A reading outside the allowed range was logged."},"first_excursion":{"type":"choice","instructions":"At which logged time was the range first exceeded?","criteria":{"07:30":"07:30","11:30":"11:30","15:30":"15:30","16:00":"16:00","19:30":"19:30","none":"never"}},"cause":{"type":"choice","instructions":"What does the log suggest as the cause?","criteria":{"door":"The door was not shut properly","power":"A power cut","defrost":"A defrost cycle","sensor":"A faulty sensor"}},"recovered":{"type":"noul","instructions":"By the last entry the temperature is back inside the allowed range."},"risk":{"type":"score","instructions":"How serious is this for the stored vaccines?","criteria":["negligible","low","moderate","high","severe"]}}},"gold":{"excursion":"true","first_excursion":"15:30","cause":"door","recovered":"true","risk":null},"note":"temperature log","state_tokens":137},{"id":"ho_j04","set":"heldout","source":"heldout_json","request":{"model":"clef-flash","state":{"reservation":{"ref":"OV-3318","venue":"Ottervane Bistro","date":"2026-10-17","time":"19:30","party_size":6,"requirements":[{"guest":2,"need":"nut allergy"},{"guest":5,"need":"step-free access"}],"deposit":{"required":true,"amount":60,"paid":true},"status":"confirmed","cancellation":"free until 48 hours before the booking; the deposit is kept after that","notes":"anniversary dinner, a quiet corner if possible"},"now":"2026-10-16T12:00"},"questions":{"free_cancellation":{"type":"noul","instructions":"Cancelling now would cost nothing."},"party":{"type":"choice","instructions":"How many people is the table for?","criteria":{"2":"two","4":"four","6":"six","8":"eight"}},"kitchen_need":{"type":"choice","instructions":"Which requirement must the kitchen deal with?","criteria":{"allergy":{"handled_by":"kitchen","detail":"an allergy"},"access":{"handled_by":"front of house","detail":"step-free seating"},"high_chair":{"handled_by":"front of house","detail":"a seat for a small child"},"vegan":{"handled_by":"kitchen","detail":"a plant-based menu"}}},"special_occasion":{"type":"noul"}}},"gold":{"free_cancellation":"false","party":"6","kitchen_need":"allergy","special_occasion":"true"},"note":"reservation; option descriptions are JSON objects; a noul without instructions","state_tokens":138},{"id":"ho_j05","set":"heldout","source":"heldout_json","request":{"model":"clef-flash","state":{"member":20417,"currency":"EUR","plan":{"name":"Brackwenna Gym monthly","fee":39.0,"due_day":1},"payments":[{"month":"2026-06","paid_on":"2026-06-01","amount":39.0},{"month":"2026-07","paid_on":"2026-07-09","amount":39.0},{"month":"2026-08","paid_on":null,"amount":0.0},{"month":"2026-09","paid_on":"2026-09-02","amount":78.0}],"late_fee_rule":"5.00 for each payment made more than 5 days after the due day"},"questions":{"missed_month":{"type":"choice","instructions":"Which month has no payment recorded?","criteria":{"2026-06":"June","2026-07":"July","2026-08":"August","2026-09":"September","none":"none"}},"july_late":{"type":"noul","instructions":"The July payment was made more than five days after the due day."},"double_payment":{"type":"noul","instructions":"The September payment equals two monthly fees."},"owed":{"type":"choice","instructions":"Ignoring late fees, how much is still owed for June to September?","criteria":{"0":"nothing","39":"39.00","78":"78.00","117":"117.00"}},"reliability":{"type":"score","instructions":"How reliable a payer is this member?","criteria":["unreliable","somewhat reliable","reliable","very reliable"]}}},"gold":{"missed_month":"2026-08","july_late":"true","double_payment":"true","owed":"0","reliability":null},"note":"payment history","state_tokens":175},{"id":"ho_j06","set":"heldout","source":"heldout_json","request":{"model":"clef-flash","state":{"app":"Fernacombe Run","version":"5.2.0","settings":{"units":"imperial","auto_pause":true,"voice_cues":{"enabled":true,"every_km":1,"language":"it"},"privacy":{"share_routes":"followers","hide_start_end":false,"public_profile":true},"notifications":{"weekly_summary":true,"challenges":false,"marketing":false},"theme":"system","heart_rate_zones":[120,140,160,175]}},"questions":{"hides_endpoints":{"type":"noul","instructions":"The start and end points of routes are hidden."},"route_audience":{"type":"choice","instructions":"Who can see the user's routes?","criteria":{"everyone":"Anyone","followers":"Followers only","nobody":"Only the user"}},"marketing":{"type":"noul","instructions":"Marketing notifications are switched on.","criteria":{"false":"Marketing notifications are off or not set."}},"zone_boundaries":{"type":"choice","instructions":"How many heart-rate zone boundaries are set?","criteria":{"3":"3","4":"4","5":"5","6":"6"}},"unit_mismatch":{"type":"noul","instructions":"The spoken cues use a different unit system from the one shown on screen."},"privacy_level":{"type":"score","instructions":"How privacy-conscious are these settings?","criteria":["very open","open","moderate","cautious","very cautious"]}}},"gold":{"hides_endpoints":"false","route_audience":"followers","marketing":"false","zone_boundaries":"4","unit_mismatch":"true","privacy_level":null},"note":"app settings; noul criteria override ('false' only)","state_tokens":104},{"id":"ho_j07","set":"heldout","source":"heldout_json","request":{"model":"clef-flash","state":[{"id":"VR-1101","product":"Vintlecombe router X2","opened":"2026-09-29","priority":"high","status":"open","subject":"No internet after firmware update","assignee":null},{"id":"VR-1102","product":"Vintlecombe mesh point","opened":"2026-09-30","priority":"low","status":"waiting_on_customer","subject":"Mesh light blinks orange","assignee":"Ruairi Esposito"},{"id":"VR-1103","product":"Vintlecombe router X2","opened":"2026-10-01","priority":"high","status":"open","subject":"Firmware update stuck at 80%","assignee":"Ruairi Esposito"},{"id":"VR-1104","product":"Vintlecombe router X1","opened":"2026-10-02","priority":"medium","status":"closed","subject":"Parental controls schedule ignored","assignee":"Saoirse Lindahl"}],"questions":{"unassigned":{"type":"choice","instructions":"Which ticket has nobody assigned?","criteria":{"VR-1101":"VR-1101","VR-1102":"VR-1102","VR-1103":"VR-1103","VR-1104":"VR-1104","none":"every ticket is assigned"}},"open_high":{"type":"choice","instructions":"How many tickets are open with high priority?","criteria":{"0":"0","1":"1","2":"2","3":"3","4":"4"}},"shared_theme":{"type":"choice","instructions":"What do the high-priority tickets have in common?","criteria":{"firmware":"A firmware update","parental":"Parental controls","mesh":"Mesh points","billing":"Billing"}},"all_closed":{"type":"noul","instructions":"All of the tickets are closed."},"wider_incident":{"type":"score","instructions":"Could a firmware release be behind a wider incident?","criteria":["unlikely","likely"]}}},"gold":{"unassigned":"VR-1101","open_high":"2","shared_theme":"firmware","all_closed":"false","wider_incident":null},"note":"support tickets; state is a top-level JSON array; a 2-level score","state_tokens":231},{"id":"ho_j08","set":"heldout","source":"heldout_json","request":{"model":"clef-flash","state":{"part":"bracket QB-7","lab":"Quorrendale Labs","batch":"B-2611","unit":"mm","spec":{"length":{"nominal":120.0,"tol":0.5},"width":{"nominal":35.0,"tol":0.2},"hole_diameter":{"nominal":8.0,"tol":0.05},"thickness":{"nominal":4.0,"tol":0.1}},"samples":[{"n":1,"length":120.2,"width":35.1,"hole_diameter":8.03,"thickness":4.02},{"n":2,"length":119.7,"width":34.9,"hole_diameter":8.07,"thickness":3.97},{"n":3,"length":120.4,"width":35.0,"hole_diameter":8.01,"thickness":4.05}]},"questions":{"all_pass":{"type":"noul","instructions":"Every sample is within tolerance on every dimension."},"failing_dimension":{"type":"choice","instructions":"Which dimension is out of tolerance?","criteria":{"length":"length","width":"width","hole_diameter":"hole diameter","thickness":"thickness","none":"none"}},"failing_sample":{"type":"choice","instructions":"Which sample fails?","criteria":{"1":"sample 1","2":"sample 2","3":"sample 3","none":"no sample"}},"disposition":{"type":"choice","instructions":"What should happen to the batch?","criteria":{"release":{"action":"release the batch","when":"every sample passes"},"hold":{"action":"hold the batch and measure more parts","when":"one sample fails by a small margin"},"scrap":{"action":"scrap the batch","when":"most samples fail by a wide margin"}}},"capability":{"type":"score","instructions":"How capable does the process look?","criteria":["poor","marginal","adequate","good"]}}},"gold":{"all_pass":"false","failing_dimension":"hole_diameter","failing_sample":"2","disposition":"hold","capability":null},"note":"QC measurements with tolerances; option descriptions are JSON objects","state_tokens":207},{"id":"ho_l01","set":"heldout","source":"heldout_text_long","request":{"model":"clef-flash","state":"SKELMARDINE COLD STORE (operated by Ostrevale Logistics)\nIncident timeline: temperature excursion in freezer chamber 3, night of Friday 11 to Saturday 12 September 2026\nCompiled by Imogen Thackeray, quality manager, from the building management system export, the shift log, the data loggers and interviews with the staff on duty. All times are local.\n\nBackground\nChamber 3 stores frozen goods at a set point of -24 °C. Its high-temperature alarm is set at -18 °C, and a pre-alarm text message goes out at -21 °C. The chamber is cooled by two compressors: compressor 1 carries the load, and compressor 2 is the standby unit, which starts by itself if compressor 1 stops. At midnight on 11 September the chamber held 412 pallets: 168 of ice cream for two retail customers, 196 of frozen vegetables (including the 22 pallets of peas delivered at 22:15) and 48 of frozen bread dough for a bakery chain. Chamber 5, next door, runs at -25 °C and had 190 empty pallet spaces that night.\nAt 15:00 on 11 September compressor 2 was isolated for a planned bearing replacement. The maintenance contractor expected to finish the work by 10:00 the next morning. No temporary cooling was hired for the period, because the job had been booked as low risk.\n\nTimeline\n21:40 The night shift starts. Shift lead Fintan Gallagher signs the handover sheet, which records \"compressor 2 out of service until about 10:00 tomorrow\". Chamber 3 reads -24.3 °C.\n22:15 The last lorry of the day unloads 22 pallets of frozen peas into chamber 3. The loading door stands open for 11 minutes; the chamber warms to -22.9 °C and is back at its set point within 20 minutes, which is normal after a delivery.\n23:58 Night operator Dmitri Volkonsky completes the routine walk-round. All doors are shut, chamber 3 reads -24.1 °C and there is no ice build-up on the evaporator coils.\n00:36 Compressor 1 trips. The building management system logs \"C1 motor contactor fault\" and the compressor stops. Because compressor 2 is isolated, no standby unit takes over.\n00:37 A \"compressor tripped\" warning appears on the screen in the control room. The control room is not staffed at night and the warning is not sent anywhere else, so nobody sees it.\n01:20 Chamber 3 reaches -21.0 °C. The system sends the pre-alarm text by itself to the on-call refrigeration number stored in its contact list.\n01:21 The text is delivered to a mobile phone that was handed back in July by an engineer who has since left the company. The contact list in the system had never been updated, so nobody reads the message.\n02:47 Chamber 3 passes the alarm limit of -18 °C. The siren sounds in the dispatch area and a second text goes to the same old number.\n02:52 Dmitri Volkonsky hears the siren while picking an order in chamber 1. He checks the alarm panel and phones Fintan Gallagher, who is in the office upstairs.\n02:58 Fintan Gallagher confirms that compressor 1 has stopped and tries a reset from the panel. The compressor runs for about 40 seconds and trips again with the same fault.\n03:05 Fintan Gallagher calls the refrigeration contractor's 24-hour line. The call handler says the on-call engineer, Solveig Arnesen, can be on site in about an hour.\n03:10 All picking from chamber 3 is stopped and its doors are kept shut to hold in the cold. Chamber 3 reads -16.8 °C.\n03:25 After a phone call with Imogen Thackeray, Fintan Gallagher decides to move the ice cream first, because it is the product most sensitive to warming, into the free spaces in chamber 5. The vegetables and the dough stay in chamber 3 for now.\n03:31 Two reach-truck drivers start moving the ice cream pallets. Each door opening is kept as short as possible, and a strip curtain is hung across the doorway of chamber 3.\n04:09 Solveig Arnesen arrives on site and starts fault-finding on compressor 1.\n04:30 140 of the 168 ice cream pallets have been moved into chamber 5. Chamber 3 reads -14.9 °C.\n04:44 Solveig Arnesen finds that the main contactor of compressor 1 has burnt out. The coil had overheated, and the contacts show heavy wear that must have built up over several years.\n04:52 The remaining 28 ice cream pallets are now in chamber 5. Pallet moves stop, so the doors of chamber 3 stay shut during the repair.\n05:15 Chamber 3 reaches its highest temperature of the night, -12.4 °C, recorded by both the wall probe and the independent data logger.\n05:20 Solveig Arnesen fits a spare contactor from the stock in her van.\n05:23 Compressor 1 restarts and runs normally. Chamber 3 starts to cool.\n05:40 Fintan Gallagher asks the maintenance contractor to send a technician early to put compressor 2 back into service. A technician is promised for 08:00.\n06:30 Chamber 3 reads -16.0 °C and is falling by roughly 1.5 °C every half hour.\n07:12 Chamber 3 falls below the alarm limit of -18 °C again, and the alarm clears by itself.\n08:05 The maintenance technician arrives and finishes fitting the new bearing in compressor 2.\n09:40 Compressor 2 is back in service as the standby unit, earlier than the original plan.\n10:15 Chamber 3 is back at its set point of -24 °C.\n\nProduct assessment (12 September, 09:00 to 13:00)\nImogen Thackeray took core temperatures from sample cartons on pallets in every product group and compared them with each customer's specification.\nIce cream: the 140 pallets moved before 04:30 never rose above -16 °C at the core. Both retail customers accept short rises to -15 °C, so these pallets were released. The last 28 pallets moved had core temperatures of -13 °C. Ice cream that has warmed that far loses its texture when it refreezes, so these 28 pallets were rejected and sent for destruction.\nFrozen vegetables: the core temperatures stayed between -15 °C and -14 °C. The supplier's specification allows short rises to -12 °C for vegetables that will be cooked before eating, so all 196 pallets were released.\nBread dough: core temperatures reached -13.5 °C. The bakery chain asked for the 48 pallets to be held until its laboratory has tested whether the yeast is still active. The pallets were labelled as on hold and moved to the quarantine bay of chamber 5.\n\nCorrective actions agreed on 14 September\n1. The on-call contact list in the alarm system is to be checked every Monday by the shift lead, and again whenever an engineer joins or leaves.\n2. A \"compressor tripped\" warning will also be sent by text to the shift lead on duty, not only shown on the control-room screen.\n3. A standby compressor may be taken out of service for planned work only if a hired mobile cooling unit has been connected to the chamber first.\n4. The contactors on all compressors in the building will be replaced now, and after that every five years.\n5. Night operators will be trained to recognise a compressor fault on the panel and to call the contractor directly.\n\nCost of the incident\n28 pallets of ice cream destroyed (about 14,300 in product value), the contractor's night call-out fee, 6.5 hours of overtime for the moving crew, and the cost of the bakery's laboratory tests if the dough is rejected.\n\nSummary\nThe contactor failure itself was ordinary wear. The damage came from three gaps that lined up on one night: the standby compressor was out of service with no cover, the first warning was only shown on an empty screen, and the alarm texts went to a phone that no one carried. More than two hours passed between the trip at 00:36 and the moment someone on site knew about it at 02:52.","questions":{"root_cause":{"type":"choice","instructions":"What started the incident?","criteria":{"contactor":"An electrical contactor on a compressor failed","door":"A loading door was left open","power_cut":"The mains power failed","refrigerant_leak":"Refrigerant leaked out","sensor":"A temperature sensor gave false readings"}},"peak_temperature":{"type":"choice","instructions":"What was the highest air temperature recorded in chamber 3?","criteria":{"-18.0":"-18.0 °C","-14.9":"-14.9 °C","-12.4":"-12.4 °C","-9.6":"-9.6 °C"}},"first_text_reached":{"type":"noul","instructions":"The first automatic alarm text reached someone who could act on it."},"dough_outcome":{"type":"choice","instructions":"What happened to the bread dough pallets?","criteria":{"released":"Released to the customer","destroyed":"Sent for destruction","held":"Held until laboratory tests are done","left_in_chamber_3":"Left in chamber 3 without any check"}},"standby_ready":{"type":"noul","instructions":"A standby compressor was ready to take over when compressor 1 stopped."},"above_limit":{"type":"choice","instructions":"For how long was chamber 3 warmer than its alarm limit?","criteria":{"under_1h":"Less than an hour","1_to_3h":"One to three hours","3_to_6h":"Three to six hours","over_6h":"More than six hours"}},"seriousness":{"type":"score","instructions":"How serious was the incident for the business?","criteria":["minor","moderate","serious","critical"]}}},"gold":{"root_cause":"contactor","peak_temperature":"-12.4","first_text_reached":"false","dough_outcome":"held","standby_ready":"false","above_limit":"3_to_6h","seriousness":null},"note":"long state: cold-store incident timeline","state_tokens":1968},{"id":"ho_l02","set":"heldout","source":"heldout_text_long","request":{"model":"clef-flash","state":"LEASE OF COMMERCIAL PREMISES (extract: particulars and clauses 1 to 15)\n\nParticulars\nLandlord: Drossingham Estates Limited\nTenant: Bramblequist Ceramics Limited\nPremises: Unit 4, Lindenquay Trade Park, a single-storey workshop of about 640 square metres with a yard and six parking spaces, shown edged red on the attached plan\nTerm: ten (10) years starting on 1 November 2026 (the \"Term Start Date\")\nInitial rent: 48,000 a year, exclusive of service charge, insurance rent and any tax\nRent-free period: the first three (3) months of the Term\nRent deposit: a sum equal to six (6) months of the initial rent\nPermitted use: the design, manufacture, storage and sale of ceramic goods, with ancillary offices\n\n1. Interpretation\n1.1 In this lease, the \"Quarter Days\" are 1 January, 1 April, 1 July and 1 October in each year.\n1.2 A reference to the Premises includes the Landlord's fixtures and fittings in them, but not the Tenant's own trade fixtures.\n1.3 Headings are for convenience only and do not change how the lease is read.\n\n2. Grant\n2.1 The Landlord lets the Premises to the Tenant for the Term, together with the right to use the estate roads and the shared refuse area alongside the other occupiers of the trade park.\n2.2 The Landlord keeps the right to enter the Premises, after giving at least forty-eight (48) hours' notice except in an emergency, to inspect them, to carry out repairs for which it is responsible, and to show them to possible buyers or, during the last six months of the Term, to possible new tenants.\n\n3. Rent\n3.1 The Tenant shall pay the rent by four equal instalments in advance on the Quarter Days. The first payment, apportioned for the part of the quarter it covers, falls due on the day after the rent-free period ends.\n3.2 If any rent or other sum is more than fourteen (14) days late, the Tenant shall pay interest on it at four percent (4%) a year above the base rate of the Landlord's bank, from the due date until the day it is paid.\n\n4. Rent Review\n4.1 The rent shall be reviewed on the fifth anniversary of the Term Start Date (the \"Review Date\").\n4.2 The reviewed rent shall be the higher of (a) the rent payable immediately before the Review Date and (b) the open market rent of the Premises at the Review Date, as agreed between the parties or, if they have not agreed within three months, as decided by an independent surveyor.\n4.3 Until the reviewed rent has been decided, the Tenant shall go on paying the old rent, and shall pay any shortfall within twenty-one (21) days after the decision.\n\n5. Service Charge and Insurance\n5.1 The Tenant shall pay a fair share of the Landlord's costs of looking after the estate roads, landscaping, lighting and the shared refuse area. For the first three years of the Term this service charge shall not be more than 4,500 in any year.\n5.2 The Landlord shall insure the building that contains the Premises against fire, storm, flood, subsidence and the other usual risks, for the full cost of rebuilding it, and the Tenant shall repay to the Landlord the part of the premium that relates to the Premises (the \"insurance rent\") within twenty-one days of a written demand.\n5.3 If an insured risk makes the Premises unfit for use, no rent is payable until they are fit for use again, for at most three (3) years.\n\n6. Repairs\n6.1 The Landlord is responsible for keeping the roof, the foundations, the outside walls and the structural frame of the Premises in good repair.\n6.2 The Tenant shall keep the inside of the Premises, including floor surfaces, doors, windows, glazing, electrical installations and the drains that serve only the Premises, in good repair and decorative order, and shall redecorate the inside in the last year of the Term.\n6.3 The Tenant does not have to put the Premises into a better state of repair than the one shown in the photographic schedule of condition that both parties signed on the Term Start Date.\n\n7. Use\n7.1 The Tenant shall use the Premises only for the Permitted Use.\n7.2 The Tenant may install and run electric or gas-fired kilns, as long as it (a) obtains any approval that the fire authority or the local council requires, (b) keeps an up-to-date fire risk assessment that covers the kilns, and (c) has every kiln inspected by a qualified engineer at least once a year.\n7.3 The Tenant shall not keep on the Premises more flammable material than it reasonably needs for the Permitted Use, and shall not disturb the other occupiers with fumes, noise or vibration.\n\n8. Alterations and Signs\n8.1 The Tenant shall not make any structural alteration to the Premises and shall not cut into the roof or the outside walls.\n8.2 The Tenant may make internal alterations that are not structural, such as putting up or taking down partitions, if the Landlord agrees in writing; the Landlord shall not refuse or delay its agreement without good reason.\n8.3 The Tenant may show its name and logo on the fascia board of the Premises, in a size and style that the Landlord has approved beforehand.\n\n9. Assignment and Underletting\n9.1 The Tenant shall not assign, underlet, share or part with possession of only a part of the Premises.\n9.2 The Tenant shall not underlet the whole of the Premises either.\n9.3 The Tenant may assign the whole of the Premises if the Landlord gives its written consent beforehand, which the Landlord shall not refuse without good reason, and if the Tenant signs an authorised guarantee agreement for the obligations of the new tenant.\n\n10. Break Clause\n10.1 The Tenant may end this lease on the fifth anniversary of the Term Start Date by giving the Landlord at least six (6) months' notice in writing.\n10.2 A notice under clause 10.1 only takes effect if, on the break date, (a) the Tenant has paid all the rent that has fallen due under this lease, and (b) the Tenant hands back the Premises with nobody occupying them and none of its goods left inside.\n10.3 This clause gives the Landlord no right to end the lease early.\n\n11. Rent Deposit\n11.1 When this lease is signed the Tenant shall pay the rent deposit to the Landlord, who shall hold it in a separate account that earns interest.\n11.2 The Landlord may take from the deposit any rent or other sum that has been unpaid for more than fourteen (14) days, and the Tenant shall then top the deposit back up within ten (10) working days.\n11.3 The deposit, together with the interest earned on it, shall be returned within one month after the end of the Term, less any amounts the Landlord has properly taken from it.\n\n12. Forfeiture\n12.1 The Landlord may re-enter the Premises and bring this lease to an end if (a) any rent is still unpaid twenty-one (21) days after it falls due, whether or not it has been formally demanded, (b) the Tenant seriously breaks any other obligation and does not put it right within a reasonable time after being told about it, or (c) the Tenant goes into liquidation or administration.\n\n13. Quiet Enjoyment\n13.1 As long as the Tenant pays the rent and keeps to this lease, the Landlord shall not disturb the Tenant's use of the Premises.\n\n14. End of the Term\n14.1 At the end of the Term the Tenant shall hand back the Premises in the repair and condition that this lease requires, take away its trade fixtures, kilns and signs, and repair any damage caused by taking them away.\n14.2 If the Landlord asks in writing at least six months before the end of the Term, the Tenant shall also take out any internal alterations it has made and put back the original layout.\n\n15. Notices and Law\n15.1 A notice under this lease is valid only if it is written down and either handed over in person or sent by recorded post to the registered office of the party receiving it.\n15.2 This lease is governed by the law of the country in which the Premises are situated, and its courts alone may decide any dispute about it.\n\nSigned for the Landlord by Philippa Caradine, director\nSigned for the Tenant by Anselm Brightwater, director","questions":{"break_notice":{"type":"choice","instructions":"How much notice must the tenant give to use the break option?","criteria":{"3m":"3 months","6m":"6 months","9m":"9 months","12m":"12 months","none":"There is no break option"}},"review_direction":{"type":"choice","instructions":"How can the rent change at the review?","criteria":{"up_only":"It can stay the same or go up, but not down","up_or_down":"It can go up or down","index":"It follows a price index","fixed":"It cannot change"}},"sublet_part":{"type":"noul","instructions":"The tenant may sublet part of the premises."},"roof":{"type":"choice","instructions":"Who must keep the roof in repair?","criteria":{"landlord":"The landlord","tenant":"The tenant","shared":"Both, in equal shares","insurer":"The insurer only"}},"deposit":{"type":"choice","instructions":"How large is the rent deposit?","criteria":{"12000":"12,000","24000":"24,000","48000":"48,000","none":"There is no deposit"}},"kilns":{"type":"noul","instructions":"The tenant may run kilns in the premises if it meets certain conditions."},"balance":{"type":"score","instructions":"Which side does the lease favour?","criteria":["the landlord","neither side","the tenant"]}}},"gold":{"break_notice":"6m","review_direction":"up_only","sublet_part":"false","roof":"landlord","deposit":"24000","kilns":"true","balance":null},"note":"long state: commercial lease extract","state_tokens":1913},{"id":"ho_i01","set":"heldout","source":"heldout_image","request":{"model":"clef-flash","state":"A household's yearly energy use, split by purpose, is drawn as a pie chart.","questions":{"biggest_slice":{"type":"choice","instructions":"Which colour is the biggest slice?","criteria":{"orange":"Orange","blue":"Blue","grey":"Grey","yellow":"Yellow"}},"over_half":{"type":"noul","instructions":"A single slice takes up more than half of the circle."},"smallest_label":{"type":"choice","instructions":"Which legend label belongs to the smallest slice?","criteria":{"heat":"HEAT","water":"WATER","other":"OTHER","light":"LIGHT"}},"blue_share":{"type":"score","instructions":"Roughly what share of the pie is blue?","criteria":["under a quarter","between a quarter and a half","over a half"]}}},"image_files":["ho_pie_720x480.png"],"gold":{"biggest_slice":"orange","over_half":"false","smallest_label":"light","blue_share":"1"},"note":"720x480","state_tokens":18},{"id":"ho_i02","set":"heldout","source":"heldout_image","request":{"model":"clef-flash","state":{"task":"Look up trains on the departure table.","station":"Pottersmire"},"questions":{"harlowick_platform":{"type":"choice","instructions":"Which platform does the Harlowick train leave from?","criteria":{"1":"Platform 1","2":"Platform 2","3":"Platform 3","4":"Platform 4"}},"earliest":{"type":"choice","instructions":"Where does the earliest departure go?","criteria":{"glenvarrow":"Glenvarrow","harlowick":"Harlowick","tumbrelby":"Tumbrelby","emberdowne":"Emberdowne"}},"platform_3":{"type":"choice","instructions":"How many departures use platform 3?","criteria":{"0":"none","1":"one","2":"two","3":"three"}},"after_nine":{"type":"noul","instructions":"The table lists a departure later than 09:00."}}},"image_files":["ho_table_1000x600.png"],"gold":{"harlowick_platform":"1","earliest":"glenvarrow","platform_3":"2","after_nine":"true"},"note":"1000x600 (5:3), JSON state","state_tokens":20},{"id":"ho_i03","set":"heldout","source":"heldout_image","request":{"model":"clef-flash","state":"A driver passes this road sign.","questions":{"limit":{"type":"choice","instructions":"What limit does the sign show?","criteria":{"20":"20","30":"30","40":"40","50":"50","60":"60","70":"70","80":"80","90":"90","100":"100","120":"120"}},"outline":{"type":"choice","instructions":"What is the outline of the sign?","criteria":{"round":"Round","triangle":"Triangular","square":"Square","octagon":"Eight-sided"}},"seventy_ok":{"type":"noul","instructions":"Driving at 70 stays within the limit shown."}}},"image_files":["ho_sign_200x150.png"],"gold":{"limit":"60","outline":"round","seventy_ok":"false"},"note":"200x150; a 10-option choice","state_tokens":7},{"id":"ho_i04","set":"heldout","source":"heldout_image","request":{"model":"clef-flash","state":{"instrument":"outdoor thermometer","scale":"degrees Celsius"},"questions":{"reading":{"type":"choice","instructions":"What temperature does the thermometer show?","criteria":{"15":"about 15","25":"about 25","35":"about 35","45":"about 45"}},"below_zero":{"type":"noul","instructions":"The reading is below zero."},"warmth":{"type":"score","instructions":"How warm is it, going by the reading?","criteria":["freezing","cool","mild","hot"]}}},"image_files":["ho_thermometer_160x400.png"],"gold":{"reading":"35","below_zero":"false","warmth":"3"},"note":"160x400 (portrait), JSON state","state_tokens":12},{"id":"ho_i05","set":"heldout","source":"heldout_image","request":{"model":"clef-flash","state":"One page of a wall calendar is included.","questions":{"first_weekday":{"type":"choice","instructions":"On which weekday does the month begin?","criteria":{"mon":"Monday","tue":"Tuesday","wed":"Wednesday","thu":"Thursday","fri":"Friday","sat":"Saturday","sun":"Sunday"}},"circled":{"type":"choice","instructions":"Which date is circled?","criteria":{"7":"the 7th","14":"the 14th","21":"the 21st","28":"the 28th"}},"month_length":{"type":"choice","instructions":"How many days does the month shown have?","criteria":{"28":"28","29":"29","30":"30","31":"31"}},"shaded_weekend":{"type":"noul","instructions":"The shaded block of days includes a Saturday or a Sunday."},"shaded_count":{"type":"score","instructions":"How many days are shaded?","criteria":["none","one to three","four to six","seven or more"]}}},"image_files":["ho_calendar_760x700.png"],"gold":{"first_weekday":"mon","circled":"14","month_length":"28","shaded_weekend":"false","shaded_count":"2"},"note":"760x700","state_tokens":9},{"id":"ho_i06","set":"heldout","source":"heldout_image","request":{"model":"clef-flash","state":{"device":"water meter","task":"Report what the meter display shows."},"questions":{"third_digit":{"type":"choice","instructions":"What is the third digit from the left?","criteria":{"0":"0","1":"1","2":"2","3":"3","4":"4","5":"5","6":"6","7":"7","8":"8","9":"9"}},"digit_count":{"type":"choice","instructions":"How many digits does the display show?","criteria":{"4":"four","5":"five","6":"six","7":"seven","8":"eight"}},"has_seven":{"type":"noul","instructions":"The digit 7 appears on the display."},"red_last":{"type":"noul","instructions":"The rightmost digit sits on a red background."}}},"image_files":["ho_digits_960x240.png"],"gold":{"third_digit":"9","digit_count":"6","has_seven":"false","red_last":"true"},"note":"960x240 (4:1), JSON state; a 10-option choice","state_tokens":16},{"id":"ho_i07","set":"heldout","source":"heldout_image","request":{"model":"clef-flash","state":"A customer kept this till slip.","questions":{"total":{"type":"choice","instructions":"What total does the slip show?","criteria":{"10.20":"10.20","11.20":"11.20","11.40":"11.40","12.10":"12.10"}},"paid_by":{"type":"choice","instructions":"How was the bill paid?","criteria":{"cash":"In cash","card":"By card","voucher":"With a voucher"}},"priciest":{"type":"choice","instructions":"Which item cost the most?","criteria":{"soup":"Soup","roll":"Roll","tea":"Tea","cake":"Cake"}},"change_shown":{"type":"noul","instructions":"The slip shows change handed back to the customer."}}},"image_files":["ho_receipt_480x840.png"],"gold":{"total":"11.20","paid_by":"card","priciest":"soup","change_shown":"false"},"note":"480x840 (portrait), dot-matrix lettering, invented shop name","state_tokens":7},{"id":"ho_i08","set":"heldout","source":"heldout_image","request":{"model":"clef-flash","state":{"task":"Study the office layout in the drawing."},"questions":{"biggest_space":{"type":"choice","instructions":"Which named space has the largest floor area?","criteria":{"meeting":"MEETING","office":"OFFICE","lobby":"LOBBY","wc":"WC","store":"STORE"}},"entrance":{"type":"choice","instructions":"Into which space does the outside door open?","criteria":{"meeting":"MEETING","office":"OFFICE","lobby":"LOBBY","wc":"WC","store":"STORE"}},"wc_store_wall":{"type":"noul","instructions":"The WC and the store share a wall."},"space_count":{"type":"choice","instructions":"How many named spaces does the plan show?","criteria":{"3":"three","4":"four","5":"five","6":"six","7":"seven"}}}},"image_files":["ho_floorplan_1200x900.png"],"gold":{"biggest_space":"office","entrance":"lobby","wc_store_wall":"true","space_count":"5"},"note":"1200x900, JSON state","state_tokens":12},{"id":"ho_p01","set":"heldout","source":"heldout_photo","request":{"model":"clef-flash","state":"Questions follow about the building shown in the picture.","questions":{"lamp_on":{"type":"noul","instructions":"The lamp at the top of the tower is shining."},"time_of_day":{"type":"choice","instructions":"When was the photo most likely taken?","criteria":{"midday":"Around midday in bright sun","twilight":"At dusk or dawn, with the sky still partly lit","night":"In the middle of a dark night","grey_day":"On a grey, overcast day"}},"sea_behind":{"type":"noul","instructions":"The sea can be seen behind the tower."},"tower_colour":{"type":"choice","instructions":"What colour is the tower painted?","criteria":{"white":"White","red_white":"Red and white bands","black":"Black","yellow":"Yellow"}},"remoteness":{"type":"score","instructions":"How remote does the place look?","criteria":["in a busy town","at the edge of a village","isolated"]}}},"image_files":["ho_photo_lighthouse.png"],"gold":{"lamp_on":"true","time_of_day":"twilight","sea_behind":"true","tower_colour":"white","remoteness":null},"note":"CC0 photo, 960x960","state_tokens":10},{"id":"ho_p02","set":"heldout","source":"heldout_photo","request":{"model":"clef-flash","state":{"task":"Describe the flowering plants pictured."},"questions":{"petal_colour":{"type":"choice","instructions":"What colour are the petals of the flowers in front?","criteria":{"yellow":"Yellow","white":"White","purple":"Purple","red":"Red"}},"insect":{"type":"noul","instructions":"An insect is sitting on one of the flowers."},"centre_colour":{"type":"choice","instructions":"What colour is the middle of the largest flower?","criteria":{"brown":"Brown or dark orange","green":"Green","blue":"Blue","white":"White"}},"setting":{"type":"choice","instructions":"Where are the flowers?","criteria":{"vase":"In a vase indoors","outdoors":"Outdoors among leaves and other plants","sand":"In bare sand","snow":"In snow"}},"density":{"type":"score","instructions":"How dense is the vegetation?","criteria":["sparse","moderate","dense"]}}},"image_files":["ho_photo_sunflowers.png"],"gold":{"petal_colour":"yellow","insect":"true","centre_colour":"brown","setting":"outdoors","density":null},"note":"CC0 photo, 960x1280","state_tokens":10}],"runs":[{"id":"own_t01","arm":"text","set":"fixture","source":"own_text","tokens":436,"n_image_tokens":0,"state_tokens":64,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,32930,220,19,19,22,16,494,82551,78831,25,8439,411,6353,279,31882,12,19,15,17471,27056,353,10471,494,39905,33604,1227,5211,668,524,2404,383,13,353,6470,1330,2086,26216,321,264,2086,14231,13,1049,369,1990,2785,279,220,24,15,10902,8215,13,353,1000,1040,424,49992,466,12215,11,524,90233,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,9016,198,9222,25,5576,198,669,55530,25,220,22365,2015,1220,3579,411,11390,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,7037,3309,11,5353,321,3435,5154,2129,1979,816,3147,4459,8934,198,32929,220,17,25,220,4754,4532,3147,84624,11,61893,11,9892,321,70057,2129,1979,816,3147,37355,8934,198,32929,220,18,25,220,4754,4532,3147,36860,11,14419,11,5384,466,19510,81031,2129,1979,816,3147,25660,8934,198,32929,220,19,25,220,4754,4532,3147,32,1918,466,2393,1503,524,944,25,21851,11,54794,11,680,1098,2129,1979,816,3147,69685,8934,198,4537,38945,198,198,27806,220,17,198,887,25,6587,7547,1199,198,9222,25,307,10644,198,669,55530,25,220,760,5813,16561,364,264,20325,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,52011,198,9222,25,5284,198,669,55530,25,220,4199,32556,369,279,11390,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,6503,3655,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,1919,1936,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,14917,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,92382,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"department","type":"choice","question_span":[124,131],"option_spans":[[142,158],[164,180],[186,203],[209,232]],"option_ids":["account","billing","shipping","technical"],"logits":[-5.014065742492676,-3.4690165519714355,-3.4463307857513428,1.1417540311813354],"probs":[0.0020749589893966913,0.00972782727330923,0.009951034560799599,0.9782460927963257],"argmax_index":3,"argmax_id":"technical","top2_margin":0.9682950582355261,"near_tie":false,"gold":"technical","gold_correct":true},{"question_id":"wants_refund","type":"noul","question_span":[256,263],"option_spans":[[274,292],[298,316]],"option_ids":["true","false"],"logits":[-2.3682873249053955,1.9134502410888672],"probs":[0.013630282133817673,0.9863697290420532],"argmax_index":1,"argmax_id":"false","top2_margin":0.9727394469082355,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"urgency","type":"score","question_span":[337,343],"option_spans":[[354,365],[371,382],[388,398],[404,414]],"option_ids":["0","1","2","3"],"logits":[-0.16858680546283722,0.04761078953742981,-0.48045504093170166,-1.4255099296569824],"probs":[0.30694112181663513,0.3810206353664398,0.22470484673976898,0.08733338862657547],"argmax_index":1,"argmax_id":"1","top2_margin":0.07407951354980469,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"department":{"type":"choice","choice":"technical","confidence":0.9782,"probabilities":{"billing":0.0097,"technical":0.9782,"shipping":0.01,"account":0.0021}},"wants_refund":{"type":"noul","noul":0.0136},"urgency":{"type":"score","score":1.0924,"confidence":0.381,"legend":{"0":"Can wait","1":"This week","2":"Today","3":"Immediately"},"probabilities":{"0":0.3069,"1":0.381,"2":0.2247,"3":0.0873}}},"usage":{"input_tokens":436,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_t02","arm":"text","set":"fixture","source":"own_text","tokens":599,"n_image_tokens":0,"state_tokens":119,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,37947,1653,220,17,15,17,21,12,16,16,22,13,2332,220,15,17,25,16,19,279,26384,2393,314,279,14175,1134,633,3370,3436,5820,13064,9830,220,20,15,18,364,883,220,19,15,4,314,7154,13,561,383,56905,22839,11,423,5997,207520,11,21748,1142,4722,220,23,13,19,13,16,506,220,15,17,25,18,24,11,321,279,1412,4238,557,1142,310,4472,539,220,15,17,25,19,21,13,2233,795,4557,557,1669,13,18284,5073,25,279,4475,3511,2195,1687,3798,995,978,37536,494,220,17,15,15,310,220,17,15,303,279,491,4722,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,28943,198,9222,25,5284,198,669,55530,25,220,4199,14938,557,279,10142,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,896,53,20,43935,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,896,53,19,8652,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,896,53,18,22469,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,896,53,17,3478,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,896,53,16,8751,2129,1979,816,3147,19,8934,198,4537,38945,198,198,27806,220,17,198,887,25,19072,198,9222,25,307,10644,198,669,55530,25,220,760,10142,369,19072,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,3578,648,3344,198,9222,25,5576,198,669,55530,25,220,3710,8613,279,10142,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,28633,9340,9163,2129,1979,816,3147,19941,8934,198,32929,220,17,25,220,4754,4532,3147,22503,291,9163,32689,8390,2129,1979,816,3147,36151,8934,198,32929,220,18,25,220,4754,4532,3147,32,6343,869,5324,303,264,4722,2129,1979,816,3147,1617,15476,8934,198,32929,220,19,25,220,4754,4532,3147,32,11436,7652,2129,1979,816,3147,65909,8934,198,32929,220,20,25,220,4754,4532,3147,2014,85093,506,449,8976,8836,2129,1979,816,3147,30686,42564,8934,198,4537,38945,198,198,27806,220,19,198,887,25,795,10868,198,9222,25,307,10644,198,669,55530,25,220,25202,866,795,5384,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,8114,5813,466,1906,795,557,5384,466,47739,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,2665,795,557,5384,466,47739,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"severity","type":"score","question_span":[179,185],"option_spans":[[196,209],[215,228],[234,247],[253,266],[272,285]],"option_ids":["0","1","2","3","4"],"logits":[-2.0960822105407715,-0.410036563873291,0.6448873281478882,0.1041063666343689,-1.594435691833496],"probs":[0.030695321038365364,0.1656961888074875,0.47583913803100586,0.27707794308662415,0.05069142207503319],"argmax_index":2,"argmax_id":"2","top2_margin":0.1987611949443817,"near_tie":false,"gold":null,"gold_correct":null},{"question_id":"resolved","type":"noul","question_span":[307,312],"option_spans":[[323,341],[347,365]],"option_ids":["true","false"],"logits":[0.3342161178588867,-3.0688323974609375],"probs":[0.9677996635437012,0.03220032528042793],"argmax_index":0,"argmax_id":"true","top2_margin":0.9355993382632732,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"root_cause","type":"choice","question_span":[388,393],"option_spans":[[404,416],[422,436],[442,459],[465,477],[483,499]],"option_ids":["attack","capacity","config_change","hardware","third_party"],"logits":[-4.971649646759033,-2.1872875690460205,1.6277532577514648,-4.624090671539307,-4.367697238922119],"probs":[0.0013243439607322216,0.021440450102090836,0.9729377627372742,0.0018747516442090273,0.002422668505460024],"argmax_index":2,"argmax_id":"config_change","top2_margin":0.9514973126351833,"near_tie":false,"gold":"config_change","gold_correct":true},{"question_id":"data_loss","type":"noul","question_span":[522,527],"option_spans":[[538,556],[562,577]],"option_ids":["true","false"],"logits":[-2.6482880115509033,2.4712586402893066],"probs":[0.005943198688328266,0.9940568208694458],"argmax_index":1,"argmax_id":"false","top2_margin":0.9881136221811175,"near_tie":false,"gold":"false","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"severity":{"type":"score","score":2.1514,"confidence":0.4758,"legend":{"0":"SEV5 cosmetic","1":"SEV4 minor","2":"SEV3 moderate","3":"SEV2 major","4":"SEV1 critical"},"probabilities":{"0":0.0307,"1":0.1657,"2":0.4758,"3":0.2771,"4":0.0507}},"resolved":{"type":"noul","noul":0.9678},"root_cause":{"type":"choice","choice":"config_change","confidence":0.9729,"probabilities":{"config_change":0.9729,"hardware":0.0019,"third_party":0.0024,"attack":0.0013,"capacity":0.0214}},"data_loss":{"type":"noul","noul":0.0059}},"usage":{"input_tokens":599,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_t03","arm":"text","set":"fixture","source":"own_text","tokens":463,"n_image_tokens":0,"state_tokens":117,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,3702,25,733,287,3968,60644,198,1206,25,81235,15948,83,198,12646,25,997,25,7548,2660,3814,271,183426,11,279,43241,494,426,2580,375,588,25914,628,874,4860,2445,383,7548,13,2838,8632,6796,506,220,24,25,15,15,466,7348,506,220,16,19,25,15,15,4370,13,5456,8019,22002,369,7629,660,1834,383,6796,364,577,28403,4379,11,748,1292,1362,524,615,279,6875,303,13,2885,488,7440,864,9167,4138,364,488,30,271,29,5369,25,81235,15948,83,198,29,733,287,11,4338,7440,279,7548,220,16,15,25,15,15,24865,369,1990,383,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,491,26669,198,9222,25,5576,198,669,55530,25,220,22365,9167,1220,381,31877,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,34072,506,220,24,25,15,15,2129,1979,816,3147,1588,62,15,24,15,15,8934,198,32929,220,17,25,220,4754,4532,3147,36539,506,220,16,15,25,15,15,11,430,12830,12539,2129,1979,816,3147,199231,62,16,15,15,15,8934,198,32929,220,18,25,220,4754,4532,3147,38867,506,220,16,19,25,15,15,2129,1979,816,3147,83,356,62,16,19,15,15,8934,198,4537,38945,198,198,27806,220,17,198,887,25,270,6955,1217,473,4322,198,9222,25,307,10644,198,669,55530,25,220,760,7548,24865,1990,4829,1925,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,9559,55503,198,9222,25,307,10644,198,669,55530,25,220,20390,55503,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"new_slot","type":"choice","question_span":[178,184],"option_spans":[[195,216],[222,248],[254,277]],"option_ids":["mon_0900","thu_1000","tue_1400"],"logits":[-3.775102376937866,-2.1430068016052246,0.557356059551239],"probs":[0.012158675119280815,0.062186531722545624,0.9256548285484314],"argmax_index":2,"argmax_id":"tue_1400","top2_margin":0.8634682968258858,"near_tie":false,"gold":"tue_1400","gold_correct":true},{"question_id":"thursday_still_on","type":"noul","question_span":[303,310],"option_spans":[[321,339],[345,363]],"option_ids":["true","false"],"logits":[-1.7946747541427612,0.5223401784896851],"probs":[0.08972356468439102,0.910276472568512],"argmax_index":1,"argmax_id":"false","top2_margin":0.8205529078841209,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"reply_needed","type":"noul","question_span":[386,388],"option_spans":[[399,417],[423,441]],"option_ids":["true","false"],"logits":[-1.3103346824645996,-2.4391109943389893],"probs":[0.7556129693984985,0.24438700079917908],"argmax_index":0,"argmax_id":"true","top2_margin":0.5112259685993195,"near_tie":false,"gold":"true","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"new_slot":{"type":"choice","choice":"tue_1400","confidence":0.9257,"probabilities":{"mon_0900":0.0122,"tue_1400":0.9257,"thu_1000":0.0622}},"thursday_still_on":{"type":"noul","noul":0.0897},"reply_needed":{"type":"noul","noul":0.7556}},"usage":{"input_tokens":463,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_t04","arm":"text","set":"fixture","source":"own_text","tokens":493,"n_image_tokens":0,"state_tokens":76,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,77739,12443,11,2790,2015,13,76080,5343,25,2233,269,11,54048,11,53286,1323,13,3588,6591,25,8026,279,6121,14548,14565,303,343,17,13,18,26,91389,279,383,36325,53751,310,279,1118,8079,13,5411,3470,25,54048,61444,279,4722,5020,539,6394,26,53286,1323,25689,279,12503,9584,383,279,4842,2081,13,5097,3296,25,3315,310,2426,279,13053,2306,383,279,14565,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,6121,7075,198,9222,25,5576,198,669,55530,25,220,3710,557,6426,883,279,6121,14548,14565,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,8994,424,2129,1979,816,3147,17961,8934,198,32929,220,17,25,220,4754,4532,3147,3997,78126,424,2129,1979,816,3147,2134,78126,8934,198,32929,220,18,25,220,4754,4532,3147,28657,424,303,343,17,13,18,2129,1979,816,3147,5102,2200,17,18,8934,198,4537,38945,198,198,27806,220,17,198,887,25,4722,26331,28080,198,9222,25,5576,198,669,55530,25,220,14749,61444,279,4722,5020,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,32,1323,1323,2129,1979,816,3147,7972,1323,8934,198,32929,220,17,25,220,4754,4532,3147,37,65435,2129,1979,816,3147,56307,911,8934,198,32929,220,18,25,220,4754,4532,3147,2665,269,2129,1979,816,3147,2083,269,8934,198,4537,38945,198,198,27806,220,18,198,887,25,1724,11789,198,9222,25,5284,198,669,55530,25,220,4199,1599,4602,513,1990,1724,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,3928,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,3833,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,33613,2129,1979,816,3147,17,8934,198,4537,38945,198,198,27806,220,19,198,887,25,13053,5916,13387,4342,198,9222,25,307,10644,198,669,55530,25,220,760,2015,6426,3315,310,2426,279,13053,2306,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"dark_mode","type":"choice","question_span":[137,146],"option_spans":[[157,168],[174,187],[193,212]],"option_ids":["cancel","postpone","ship_v23"],"logits":[-4.841253280639648,-4.835235595703125,0.08089065551757812],"probs":[0.00717861158773303,0.007221940904855728,0.9855993986129761],"argmax_index":2,"argmax_id":"ship_v23","top2_margin":0.9783774577081203,"near_tie":false,"gold":"ship_v23","gold_correct":true},{"question_id":"release_note_owner","type":"choice","question_span":[235,241],"option_spans":[[252,265],[271,283],[289,301]],"option_ids":["abebe","felix","noor"],"logits":[-4.723631381988525,-0.2690659761428833,-4.90962028503418],"probs":[0.011383161880075932,0.9791655540466309,0.009451243095099926],"argmax_index":1,"argmax_id":"felix","top2_margin":0.9677823921665549,"near_tie":false,"gold":"felix","gold_correct":true},{"question_id":"open_items","type":"score","question_span":[323,330],"option_spans":[[341,351],[357,367],[373,383]],"option_ids":["0","1","2"],"logits":[-3.878234386444092,0.5191258788108826,-4.402272701263428],"probs":[0.012073171325027943,0.9807779788970947,0.007148820906877518],"argmax_index":1,"argmax_id":"1","top2_margin":0.9687048075720668,"near_tie":false,"gold":"1","gold_correct":true},{"question_id":"beta_label_decided","type":"noul","question_span":[408,418],"option_spans":[[429,447],[453,471]],"option_ids":["true","false"],"logits":[-2.855055093765259,0.7833472490310669],"probs":[0.025620637461543083,0.974379301071167],"argmax_index":1,"argmax_id":"false","top2_margin":0.9487586636096239,"near_tie":false,"gold":"false","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"dark_mode":{"type":"choice","choice":"ship_v23","confidence":0.9856,"probabilities":{"ship_v23":0.9856,"postpone":0.0072,"cancel":0.0072}},"release_note_owner":{"type":"choice","choice":"felix","confidence":0.9792,"probabilities":{"noor":0.0095,"felix":0.9792,"abebe":0.0114}},"open_items":{"type":"score","score":0.9951,"confidence":0.9808,"legend":{"0":"None","1":"One","2":"Several"},"probabilities":{"0":0.0121,"1":0.9808,"2":0.0071}},"beta_label_decided":{"type":"noul","noul":0.0256}},"usage":{"input_tokens":493,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_t05","arm":"text","set":"fixture","source":"own_text","tokens":553,"n_image_tokens":0,"state_tokens":56,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,33497,220,17,680,314,220,20,13,561,74812,12,18,69862,5686,2172,321,85895,4778,11,694,1238,2250,5381,279,25493,73013,41423,11,321,2957,279,16465,9035,12105,1467,524,7926,11,864,369,264,1865,6925,3377,13,11926,1761,19386,2785,264,1834,321,8632,264,13329,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,25144,198,9222,25,5284,198,669,55530,25,220,3710,369,279,7830,25144,314,279,3286,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1164,7968,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,40823,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,54789,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,29492,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,1164,6572,2129,1979,816,3147,19,8934,198,4537,38945,198,198,27806,220,17,198,887,25,6925,51550,198,9222,25,307,10644,198,669,55530,25,220,760,3286,6473,264,6925,3377,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,1822,2870,30267,198,9222,25,5576,198,669,55530,25,220,22365,12523,369,279,1822,11833,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,21624,53985,3147,91020,321,1304,2129,8422,3147,64,61990,6052,421,4774,74054,56658,1979,816,3147,24065,8934,198,32929,220,17,25,220,4754,4532,21624,53985,3147,17710,14293,466,11884,680,2129,8422,3147,64,73013,45441,1238,264,2182,56658,1979,816,3147,68298,2803,8934,198,32929,220,18,25,220,4754,4532,21624,53985,3147,4993,4778,279,1918,4138,2129,8422,3147,726,8405,303,1330,4276,56658,1979,816,3147,19755,8934,198,32929,220,19,25,220,4754,4532,21624,53985,3147,1719,5813,2393,3039,2129,8422,3147,276,8057,852,2496,19386,56658,1979,816,3147,22637,8934,198,4537,38945,198,198,27806,220,19,198,887,25,6718,198,9222,25,307,10644,198,669,55530,25,220,27083,279,53650,6718,279,69862,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,53650,1000,6718,279,1918,310,3672,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"sentiment","type":"score","question_span":[116,125],"option_spans":[[136,147],[153,163],[169,179],[185,195],[201,212]],"option_ids":["0","1","2","3","4"],"logits":[-1.984655737876892,1.0854285955429077,0.46497538685798645,-3.244117259979248,-4.230101585388184],"probs":[0.028970826417207718,0.6241393685340881,0.3356001377105713,0.008222117088735104,0.0030674401205033064],"argmax_index":1,"argmax_id":"1","top2_margin":0.28853923082351685,"near_tie":false,"gold":"1","gold_correct":true},{"question_id":"safety_issue","type":"noul","question_span":[235,242],"option_spans":[[253,271],[277,295]],"option_ids":["true","false"],"logits":[0.5273074507713318,-2.941615581512451],"probs":[0.9697904586791992,0.030209515243768692],"argmax_index":0,"argmax_id":"true","top2_margin":0.9395809434354305,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"main_complaint","type":"choice","question_span":[318,325],"option_spans":[[336,359],[365,391],[397,421],[427,450]],"option_ids":["design","durability","speed","support"],"logits":[-4.888810634613037,0.1615869104862213,-4.528262615203857,-4.337464809417725],"probs":[0.00624008709564805,0.9739807844161987,0.00894900318235159,0.010830216109752655],"argmax_index":1,"argmax_id":"durability","top2_margin":0.9631505683064461,"near_tie":false,"gold":"durability","gold_correct":true},{"question_id":"recommend","type":"noul","question_span":[472,479],"option_spans":[[490,507],[513,531]],"option_ids":["true","false"],"logits":[-2.5793654918670654,1.2993135452270508],"probs":[0.020259197801351547,0.9797407388687134],"argmax_index":1,"argmax_id":"false","top2_margin":0.9594815410673618,"near_tie":false,"gold":"false","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"sentiment":{"type":"score","score":1.3323,"confidence":0.6241,"legend":{"0":"very negative","1":"negative","2":"mixed","3":"positive","4":"very positive"},"probabilities":{"0":0.029,"1":0.6241,"2":0.3356,"3":0.0082,"4":0.0031}},"safety_issue":{"type":"noul","noul":0.9698},"main_complaint":{"type":"choice","choice":"durability","confidence":0.974,"probabilities":{"durability":0.974,"speed":0.0089,"design":0.0062,"support":0.0108}},"recommend":{"type":"noul","noul":0.0203}},"usage":{"input_tokens":553,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_t06","arm":"text","set":"fixture","source":"own_text","tokens":411,"n_image_tokens":0,"state_tokens":88,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,58,15,24,25,15,17,60,173462,25,369,279,46170,4475,1441,364,5316,745,30,198,58,15,24,25,15,18,60,51761,25,9542,11,88295,2394,883,220,15,23,25,20,15,198,58,15,24,25,15,20,60,173462,25,5492,5686,6699,198,58,15,24,25,15,22,60,51761,25,77870,279,46170,35729,11,424,579,1142,685,198,58,15,24,25,15,23,60,173462,25,9061,0,4162,279,11567,1228,1495,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,4424,198,9222,25,5576,198,669,55530,25,220,22365,4424,557,11164,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,43467,321,46170,2129,1979,816,3147,20385,8934,198,32929,220,17,25,220,4754,4532,3147,2665,4424,2129,1979,816,3147,785,2405,8934,198,32929,220,18,25,220,4754,4532,3147,43467,1132,2129,1979,816,3147,21924,8934,198,32929,220,19,25,220,4754,4532,3147,608,3982,1132,2129,1979,816,3147,267,3982,8934,198,4537,38945,198,198,27806,220,17,198,887,25,8097,198,9222,25,307,10644,198,669,55530,25,220,760,3377,557,8097,2261,279,10125,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,52011,198,9222,25,5284,198,669,55530,25,220,5382,2179,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,9997,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,25252,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,11553,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"environment","type":"choice","question_span":[148,153],"option_spans":[[164,176],[182,194],[200,211],[217,230]],"option_ids":["both","neither","production","staging"],"logits":[-4.784824371337891,-5.029145240783691,-4.8935866355896,-0.12063926458358765],"probs":[0.009194618090987206,0.007201558444648981,0.008247054181993008,0.9753568172454834],"argmax_index":3,"argmax_id":"staging","top2_margin":0.9661621991544962,"near_tie":false,"gold":"staging","gold_correct":true},{"question_id":"fixed","type":"noul","question_span":[252,260],"option_spans":[[271,289],[295,313]],"option_ids":["true","false"],"logits":[-2.3134121894836426,-4.5995588302612305],"probs":[0.9077232480049133,0.09227681905031204],"argmax_index":0,"argmax_id":"true","top2_margin":0.8154464289546013,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"urgency","type":"score","question_span":[334,336],"option_spans":[[347,357],[363,373],[379,389]],"option_ids":["0","1","2"],"logits":[-3.705456018447876,-1.7998912334442139,-3.1520354747772217],"probs":[0.10568148642778397,0.7105181217193604,0.1838003695011139],"argmax_index":1,"argmax_id":"1","top2_margin":0.5267177522182465,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"environment":{"type":"choice","choice":"staging","confidence":0.9754,"probabilities":{"production":0.0082,"staging":0.9754,"both":0.0092,"neither":0.0072}},"fixed":{"type":"noul","noul":0.9077},"urgency":{"type":"score","score":1.0781,"confidence":0.7105,"legend":{"0":"low","1":"medium","2":"high"},"probabilities":{"0":0.1057,"1":0.7105,"2":0.1838}}},"usage":{"input_tokens":411,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_t07","arm":"text","set":"fixture","source":"own_text","tokens":438,"n_image_tokens":0,"state_tokens":41,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,6503,567,1423,220,18,15,4276,1727,1936,364,279,7786,3286,30,353,2688,680,383,6796,321,383,7670,12967,13,65788,628,1132,635,55287,13,26882,7509,369,29726,383,7548,321,6394,13,471,78268,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,1786,26669,198,9222,25,5576,198,669,55530,25,220,22365,9167,4138,364,4858,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,33475,6353,2129,1979,816,3147,188335,22167,8934,198,32929,220,17,25,220,4754,4532,3147,34072,6353,2129,1979,816,3147,1588,22167,8934,198,32929,220,18,25,220,4754,4532,3147,36539,6353,2129,1979,816,3147,199231,22167,8934,198,32929,220,19,25,220,4754,4532,3147,38867,6353,2129,1979,816,3147,83,356,22167,8934,198,32929,220,20,25,220,4754,4532,3147,38867,12967,2129,1979,816,3147,83,356,44953,8934,198,32929,220,21,25,220,4754,4532,3147,39181,12967,2129,1979,816,3147,65092,44953,8934,198,4537,38945,198,198,27806,220,17,198,887,25,2984,198,9222,25,5576,198,669,55530,25,220,4199,1248,1220,279,6236,381,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16,20,4276,2129,1979,816,3147,16,20,8934,198,32929,220,17,25,220,4754,4532,3147,18,15,4276,2129,1979,816,3147,18,15,8934,198,32929,220,18,25,220,4754,4532,3147,588,6258,2129,1979,816,3147,21,15,8934,198,32929,220,19,25,220,4754,4532,3147,24,15,4276,2129,1979,816,3147,24,15,8934,198,4537,38945,198,198,27806,220,18,198,887,25,3749,88174,198,9222,25,307,10644,198,669,55530,25,220,52681,682,310,5640,310,9138,279,6236,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"best_slot","type":"choice","question_span":[102,108],"option_spans":[[119,131],[137,149],[155,167],[173,186],[192,205],[211,223]],"option_ids":["fri_am","mon_am","thu_am","tue_am","tue_pm","wed_pm"],"logits":[-3.2395803928375244,-1.6637797355651855,-2.2338924407958984,0.07446765899658203,-3.04740309715271,-4.568448066711426],"probs":[0.026637353003025055,0.1287812441587448,0.07282086461782455,0.7324261665344238,0.03228142112493515,0.007052955683320761],"argmax_index":3,"argmax_id":"tue_am","top2_margin":0.603644922375679,"near_tie":false,"gold":"tue_am","gold_correct":true},{"question_id":"length","type":"choice","question_span":[244,251],"option_spans":[[262,275],[281,294],[300,312],[318,331]],"option_ids":["15","30","60","90"],"logits":[-4.158176898956299,1.272179365158081,-4.392836570739746,-4.849423408508301],"probs":[0.0043379757553339005,0.9900583624839783,0.003430643118917942,0.0021731131710112095],"argmax_index":1,"argmax_id":"30","top2_margin":0.9857203867286444,"near_tie":false,"gold":"30","gold_correct":true},{"question_id":"needs_travel","type":"noul","question_span":[354,363],"option_spans":[[374,392],[398,416]],"option_ids":["true","false"],"logits":[-1.7808036804199219,0.20156973600387573],"probs":[0.12106605619192123,0.8789339661598206],"argmax_index":1,"argmax_id":"false","top2_margin":0.7578679099678993,"near_tie":false,"gold":"false","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"best_slot":{"type":"choice","choice":"tue_am","confidence":0.7324,"probabilities":{"mon_am":0.1288,"tue_am":0.7324,"tue_pm":0.0323,"wed_pm":0.0071,"thu_am":0.0728,"fri_am":0.0266}},"length":{"type":"choice","choice":"30","confidence":0.9901,"probabilities":{"15":0.0043,"30":0.9901,"60":0.0034,"90":0.0022}},"needs_travel":{"type":"noul","noul":0.1211}},"usage":{"input_tokens":438,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_t08","arm":"text","set":"fixture","source":"own_text","tokens":604,"n_image_tokens":0,"state_tokens":70,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,1853,9011,81267,364,1330,25,220,16,10212,21748,81267,11,220,16,10212,13671,11,4139,264,10212,314,52647,11,220,17,53626,170454,18483,11,220,16,59437,24924,13,62025,11,3311,321,28417,344,364,506,3140,220,21,3979,13,6694,440,60292,1518,13092,13,29804,32274,13,2885,381,1801,28196,440,69219,13671,11,27798,52647,321,52556,42752,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,9673,198,9222,25,5576,198,669,55530,25,220,2053,5158,11,864,9673,1503,279,10793,4786,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,23060,12668,466,7397,2129,1979,816,3147,12985,16124,266,8934,198,32929,220,17,25,220,4754,4532,3147,53,15293,25,874,9572,3741,2129,1979,816,3147,571,28468,8934,198,32929,220,18,25,220,4754,4532,3147,41197,447,8555,25,874,12668,466,7397,11,1189,6435,32274,466,24924,2129,1979,816,3147,79628,8555,8934,198,4537,38945,198,198,27806,220,17,198,887,25,3749,648,1874,286,198,9222,25,307,10644,198,669,55530,25,220,760,11659,3749,8364,310,10236,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,4879,198,9222,25,5284,198,669,55530,25,220,4199,1691,17439,4879,1503,424,1831,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,89413,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,14265,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,64,2609,2129,1979,816,3147,17,8934,198,4537,38945,198,198,27806,220,19,198,887,25,75281,198,9222,25,5576,198,669,55530,25,220,4199,1599,75281,1503,279,10793,1236,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,1893,2129,1979,816,3147,16,15,8934,198,32929,220,18,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,21,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,22,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,32929,220,23,25,220,4754,4532,3147,77420,2129,1979,816,3147,22,8934,198,32929,220,24,25,220,4754,4532,3147,65254,2129,1979,816,3147,23,8934,198,32929,220,16,15,25,220,4754,4532,3147,90012,2129,1979,816,3147,24,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"diet","type":"choice","question_span":[130,140],"option_spans":[[151,166],[172,188],[194,218]],"option_ids":["contains_meat","vegan","vegetarian"],"logits":[-4.845274925231934,-2.8615903854370117,0.5351378917694092],"probs":[0.004436923190951347,0.03225412219762802,0.9633089303970337],"argmax_index":2,"argmax_id":"vegetarian","top2_margin":0.9310548081994057,"near_tie":false,"gold":"vegetarian","gold_correct":true},{"question_id":"needs_cooking","type":"noul","question_span":[243,250],"option_spans":[[261,279],[285,303]],"option_ids":["true","false"],"logits":[-3.29707670211792,2.0163044929504395],"probs":[0.004901106469333172,0.995098888874054],"argmax_index":1,"argmax_id":"false","top2_margin":0.9901977824047208,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"effort","type":"score","question_span":[324,332],"option_spans":[[343,353],[359,369],[375,386]],"option_ids":["0","1","2"],"logits":[0.2960386276245117,-1.5994032621383667,-3.9329004287719727],"probs":[0.8585019707679749,0.12899157404899597,0.012506517581641674],"argmax_index":0,"argmax_id":"0","top2_margin":0.7295103967189789,"near_tie":false,"gold":"0","gold_correct":true},{"question_id":"servings","type":"choice","question_span":[407,415],"option_spans":[[426,436],[442,453],[459,469],[475,485],[491,501],[507,517],[523,533],[539,549],[555,565],[572,582]],"option_ids":["1","10","2","3","4","5","6","7","8","9"],"logits":[-3.1558568477630615,-4.267952919006348,0.595272958278656,-3.5706729888916016,-3.8001761436462402,-4.7421441078186035,-4.995565414428711,-5.2224507331848145,-5.217245101928711,-5.315762996673584],"probs":[0.021826304495334625,0.007177992723882198,0.9291272759437561,0.014415438286960125,0.01145924162119627,0.0044674985110759735,0.003467407776042819,0.002763566095381975,0.0027779899537563324,0.0025173574686050415],"argmax_index":2,"argmax_id":"2","top2_margin":0.9073009714484215,"near_tie":false,"gold":"2","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"diet":{"type":"choice","choice":"vegetarian","confidence":0.9633,"probabilities":{"vegan":0.0323,"vegetarian":0.9633,"contains_meat":0.0044}},"needs_cooking":{"type":"noul","noul":0.0049},"effort":{"type":"score","score":0.154,"confidence":0.8585,"legend":{"0":"minimal","1":"some","2":"a lot"},"probabilities":{"0":0.8585,"1":0.129,"2":0.0125}},"servings":{"type":"choice","choice":"2","confidence":0.9291,"probabilities":{"1":0.0218,"2":0.9291,"3":0.0144,"4":0.0115,"5":0.0045,"6":0.0035,"7":0.0028,"8":0.0028,"9":0.0025,"10":0.0072}}},"usage":{"input_tokens":604,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_t09","arm":"text","set":"fixture","source":"own_text","tokens":582,"n_image_tokens":0,"state_tokens":52,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,12675,11,353,557,11102,10598,364,821,5297,14701,25,1330,9892,314,220,16,19,13,24,24,383,5297,220,18,13,5044,20325,279,21814,6545,13,1061,369,279,2018,854,411,682,6716,321,353,2688,5732,310,8779,28342,13,10791,11,40152,48080,17039,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,9016,198,9222,25,5576,198,669,55530,25,220,22365,2015,1220,3579,411,1876,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,7037,3309,11,5353,321,3435,5154,2129,1979,816,3147,4459,8934,198,32929,220,17,25,220,4754,4532,3147,84624,11,61893,11,9892,321,70057,2129,1979,816,3147,37355,8934,198,32929,220,18,25,220,4754,4532,3147,36860,11,14419,11,5384,466,19510,81031,2129,1979,816,3147,25660,8934,198,32929,220,19,25,220,4754,4532,3147,32,1918,466,2393,1503,524,944,25,21851,11,54794,11,680,1098,2129,1979,816,3147,69685,8934,198,4537,38945,198,198,27806,220,17,198,887,25,21814,44471,198,9222,25,307,10644,198,669,55530,25,220,760,5813,557,11102,777,1056,2957,364,279,1788,3065,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,20325,13083,198,9222,25,5576,198,669,55530,25,220,4199,1691,1220,381,90233,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,39850,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,588,6545,314,220,16,19,13,24,24,2129,1979,816,3147,16,19,13,24,24,8934,198,32929,220,18,25,220,4754,4532,3147,20385,9892,2129,1979,816,3147,17,24,13,24,23,8934,198,4537,38945,198,198,27806,220,19,198,887,25,30933,198,9222,25,5284,198,669,55530,25,220,4199,31457,369,279,5813,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,5248,76,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,3122,69504,54147,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,985,2181,290,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,1569,3304,639,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,1164,31457,2129,1979,816,3147,19,8934,198,32929,220,21,25,220,4754,4532,3147,512,856,2129,1979,816,3147,20,8934,198,32929,220,22,25,220,4754,4532,3147,69,26634,2129,1979,816,3147,21,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"department","type":"choice","question_span":[112,119],"option_spans":[[130,146],[152,168],[174,191],[197,220]],"option_ids":["account","billing","shipping","technical"],"logits":[-4.732706069946289,0.0852729082107544,-4.959470272064209,-4.195807933807373],"probs":[0.00786023959517479,0.9724278450012207,0.006265465170145035,0.013446450233459473],"argmax_index":1,"argmax_id":"billing","top2_margin":0.9589813947677612,"near_tie":false,"gold":"billing","gold_correct":true},{"question_id":"duplicate_charge","type":"noul","question_span":[243,255],"option_spans":[[266,284],[290,308]],"option_ids":["true","false"],"logits":[-1.0171366930007935,-4.151845932006836],"probs":[0.9583020210266113,0.041698019951581955],"argmax_index":0,"argmax_id":"true","top2_margin":0.9166040010750294,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"refund_amount","type":"choice","question_span":[330,336],"option_spans":[[347,357],[363,385],[391,406]],"option_ids":["0","14.99","29.98"],"logits":[-4.830482482910156,-0.15584880113601685,-4.388627529144287],"probs":[0.009111710824072361,0.9767141342163086,0.014174086973071098],"argmax_index":1,"argmax_id":"14.99","top2_margin":0.9625400472432375,"near_tie":false,"gold":"14.99","gold_correct":true},{"question_id":"frustration","type":"score","question_span":[427,433],"option_spans":[[444,455],[461,473],[479,491],[497,509],[515,526],[532,543],[549,560]],"option_ids":["0","1","2","3","4","5","6"],"logits":[-3.5055623054504395,-3.1033267974853516,-1.4373812675476074,-0.2983569800853729,-0.6165322065353394,-2.103412628173828,-2.7381935119628906],"probs":[0.016860734671354294,0.02520955167710781,0.13337551057338715,0.41662758588790894,0.3030862808227539,0.06852078437805176,0.036319550126791],"argmax_index":3,"argmax_id":"3","top2_margin":0.11354130506515503,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"department":{"type":"choice","choice":"billing","confidence":0.9724,"probabilities":{"billing":0.9724,"technical":0.0134,"shipping":0.0063,"account":0.0079}},"duplicate_charge":{"type":"noul","noul":0.9583},"refund_amount":{"type":"choice","choice":"14.99","confidence":0.9767,"probabilities":{"0":0.0091,"14.99":0.9767,"29.98":0.0142}},"frustration":{"type":"score","score":3.3147,"confidence":0.4166,"legend":{"0":"calm","1":"slightly annoyed","2":"annoyed","3":"frustrated","4":"very frustrated","5":"angry","6":"furious"},"probabilities":{"0":0.0169,"1":0.0252,"2":0.1334,"3":0.4166,"4":0.3031,"5":0.0685,"6":0.0363}}},"usage":{"input_tokens":582,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_t10","arm":"text","set":"fixture","source":"own_text","tokens":396,"n_image_tokens":0,"state_tokens":68,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,1835,494,69767,496,70295,318,44029,11,3603,6206,1590,353,1144,310,13881,821,7483,220,17,17,25,15,15,12,15,21,25,15,15,6206,383,279,220,16,19,337,440,245894,26342,67220,375,11,852,682,7003,13,1165,513,2107,22196,364,10248,2017,19024,13,353,1044,9859,1330,5381,303,11573,11,430,279,4687,7225,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,19417,52284,198,9222,25,307,10644,198,669,55530,25,220,760,6206,13881,19417,279,10661,4530,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,4687,51550,198,9222,25,5576,198,669,55530,25,220,3742,1017,264,4687,3377,440,279,1622,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,32,7167,26711,2129,1979,816,3147,11896,2325,8934,198,32929,220,17,25,220,4754,4532,3147,760,975,11525,682,524,7003,2129,1979,816,3147,6058,305,8934,198,32929,220,18,25,220,4754,4532,3147,2665,4687,3377,2129,1979,816,3147,6487,8934,198,32929,220,19,25,220,4754,4532,3147,2536,3213,5125,2129,1979,816,3147,22821,8934,198,4537,38945,198,198,27806,220,18,198,887,25,3286,198,9222,25,5284,198,669,55530,25,220,20810,279,1622,1144,264,6436,579,3286,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,4660,381,11454,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,53390,3286,2129,1979,816,3147,16,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"meets_conditions","type":"noul","question_span":[130,138],"option_spans":[[149,167],[173,191]],"option_ids":["true","false"],"logits":[-0.4278266131877899,-2.7490053176879883],"probs":[0.910615861415863,0.08938407152891159],"argmax_index":0,"argmax_id":"true","top2_margin":0.8212317898869514,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"policy_issue","type":"choice","question_span":[213,222],"option_spans":[[233,246],[252,268],[274,286],[292,304]],"option_ids":["certification","consent","none","notice"],"logits":[-4.006008148193359,-4.273046493530273,0.013337314128875732,-3.9151058197021484],"probs":[0.017086582258343697,0.013082234188914299,0.9511185884475708,0.018712572753429413],"argmax_index":2,"argmax_id":"none","top2_margin":0.9324060156941414,"near_tie":false,"gold":"none","gold_correct":true},{"question_id":"review","type":"score","question_span":[325,334],"option_spans":[[345,357],[363,374]],"option_ids":["0","1"],"logits":[0.8107467293739319,-2.0345983505249023],"probs":[0.9450775980949402,0.0549224391579628],"argmax_index":0,"argmax_id":"0","top2_margin":0.8901551589369774,"near_tie":false,"gold":"0","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"meets_conditions":{"type":"noul","noul":0.9106},"policy_issue":{"type":"choice","choice":"none","confidence":0.9511,"probabilities":{"none":0.9511,"notice":0.0187,"certification":0.0171,"consent":0.0131}},"review":{"type":"score","score":0.0549,"confidence":0.9451,"legend":{"0":"can be approved","1":"needs review"},"probabilities":{"0":0.9451,"1":0.0549}}},"usage":{"input_tokens":396,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_t11","arm":"text","set":"fixture","source":"own_text","tokens":497,"n_image_tokens":0,"state_tokens":62,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,20100,579,4610,46268,2640,5813,3425,513,23680,506,2645,440,37555,12,17,20,21,321,303,23312,440,40278,220,16,13,18,11,57650,513,8341,364,220,18,20,2756,11,321,5395,2528,7225,11436,6679,13,12909,71344,6813,513,1542,42800,26,279,1379,3110,1830,369,220,16,19,3818,2235,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,23680,3625,31168,198,9222,25,307,10644,198,669,55530,25,220,12425,3425,513,23680,506,2645,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,19576,467,10757,198,9222,25,307,10644,198,669,55530,25,220,3742,279,42058,1228,1428,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,32,42058,1228,557,2725,2785,279,1483,220,16,17,3818,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,1483,42058,1228,369,8742,1056,220,16,17,3818,11,466,1141,2321,369,9496,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,5048,198,9222,25,5284,198,669,55530,25,220,4199,38841,369,279,20100,7830,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,9997,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,201175,639,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,11553,2129,1979,816,3147,18,8934,198,4537,38945,198,198,27806,220,19,198,887,25,1727,11607,198,9222,25,5576,198,669,55530,25,220,3710,1220,3420,1727,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,27654,571,279,20100,430,369,2129,1979,816,3147,47566,8934,198,32929,220,17,25,220,4754,4532,3147,75725,279,20100,2129,1979,816,3147,45427,8934,198,32929,220,18,25,220,4754,4532,3147,25337,364,264,491,42058,1228,1518,73551,2129,1979,816,3147,1967,605,305,467,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"encrypted_at_rest","type":"noul","question_span":[125,132],"option_spans":[[143,161],[167,185]],"option_ids":["true","false"],"logits":[-0.7038840055465698,-3.833425283432007],"probs":[0.9580949544906616,0.04190501570701599],"argmax_index":0,"argmax_id":"true","top2_margin":0.9161899387836456,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"pentest_current","type":"noul","question_span":[209,215],"option_spans":[[226,247],[253,279]],"option_ids":["true","false"],"logits":[-2.7440996170043945,-0.018853962421417236],"probs":[0.061500001698732376,0.9385000467300415],"argmax_index":1,"argmax_id":"false","top2_margin":0.8770000450313091,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"risk","type":"score","question_span":[300,307],"option_spans":[[318,328],[334,345],[351,362],[368,378]],"option_ids":["0","1","2","3"],"logits":[-1.9642385244369507,0.04531747102737427,-2.0737226009368896,-3.268866539001465],"probs":[0.10386830568313599,0.7748581171035767,0.09309681504964828,0.028176715597510338],"argmax_index":1,"argmax_id":"1","top2_margin":0.6709898114204407,"near_tie":false,"gold":null,"gold_correct":null},{"question_id":"next_step","type":"choice","question_span":[400,405],"option_spans":[[416,431],[437,449],[455,475]],"option_ids":["approve","reject","request_pentest"],"logits":[-4.097079277038574,-4.464418888092041,0.2929137945175171],"probs":[0.012145879678428173,0.008411926217377186,0.9794421792030334],"argmax_index":2,"argmax_id":"request_pentest","top2_margin":0.9672962995246053,"near_tie":false,"gold":"request_pentest","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"encrypted_at_rest":{"type":"noul","noul":0.9581},"pentest_current":{"type":"noul","noul":0.0615},"risk":{"type":"score","score":1.0456,"confidence":0.7749,"legend":{"0":"low","1":"moderate","2":"elevated","3":"high"},"probabilities":{"0":0.1039,"1":0.7749,"2":0.0931,"3":0.0282}},"next_step":{"type":"choice","choice":"request_pentest","confidence":0.9794,"probabilities":{"approve":0.0121,"request_pentest":0.9794,"reject":0.0084}}},"usage":{"input_tokens":497,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_t12","arm":"text","set":"fixture","source":"own_text","tokens":967,"n_image_tokens":0,"state_tokens":88,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4286,5020,494,1898,220,16,17,25,1330,12236,321,799,1623,318,416,220,21,553,55321,25,799,43868,4976,23123,11,321,799,58588,76981,1973,13696,318,67,21263,57638,553,9112,25,13983,35243,11,874,18607,13,93654,25,799,61366,2919,11,799,28503,986,11,799,18016,22105,13,2302,4439,364,279,3896,310,381,6507,303,1330,321,1318,279,41334,4861,1238,279,65835,13,5044,2113,264,14756,36549,383,279,1623,579,41334,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,32274,5527,40558,198,9222,25,307,10644,198,669,55530,25,220,52681,506,279,1898,682,264,32274,57638,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,14284,198,9222,25,5576,198,669,55530,25,220,4199,1599,1208,513,506,279,1898,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,20,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,21,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,18,198,887,25,6507,61022,198,9222,25,307,10644,198,669,55530,25,220,760,3896,1220,381,6507,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,6507,1611,912,198,9222,25,5576,198,669,55530,25,220,25741,1204,1599,5306,1220,279,3896,381,6507,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,17,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,18,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,4537,38945,198,198,27806,220,20,198,887,25,12723,198,9222,25,5576,198,669,55530,25,220,3710,369,279,12723,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,2014,20921,2129,1979,816,3147,985,17053,8934,198,32929,220,17,25,220,4754,4532,3147,32,14756,2129,1979,816,3147,49541,8934,198,32929,220,18,25,220,4754,4532,3147,32,2478,14704,2129,1979,816,3147,25316,8934,198,32929,220,19,25,220,4754,4532,3147,32,37177,2129,1979,816,3147,6547,3868,8934,198,32929,220,20,25,220,4754,4532,3147,2665,3175,12723,2129,1979,816,3147,6487,8934,198,4537,38945,198,198,27806,220,21,198,887,25,9488,50199,366,198,9222,25,5284,198,669,55530,25,220,4199,24339,369,279,1906,364,279,9488,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,371,25806,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,43518,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,13654,8466,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,22523,2129,1979,816,3147,19,8934,198,4537,38945,198,198,27806,220,22,198,887,25,41334,6119,7068,198,9222,25,307,10644,198,669,55530,25,220,760,14284,3019,310,1906,41334,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,23,198,887,25,1623,16124,278,198,9222,25,5576,198,669,55530,25,220,3710,1503,279,1623,7923,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,32,43144,2129,1979,816,3147,60347,8934,198,32929,220,17,25,220,4754,4532,3147,41197,447,8555,4976,23123,2129,1979,816,3147,14078,23123,8934,198,32929,220,18,25,220,4754,4532,3147,26117,35243,2129,1979,816,3147,20139,605,13890,8934,198,32929,220,19,25,220,4754,4532,3147,6262,4233,76981,2129,1979,816,3147,371,403,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"dairy_allergy","type":"noul","question_span":[151,160],"option_spans":[[171,189],[195,213]],"option_ids":["true","false"],"logits":[-1.3067123889923096,-4.048829555511475],"probs":[0.9394665956497192,0.06053338572382927],"argmax_index":0,"argmax_id":"true","top2_margin":0.87893320992589,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"guests","type":"choice","question_span":[234,242],"option_spans":[[253,263],[269,279],[285,295],[301,311],[317,327],[333,343]],"option_ids":["1","2","3","4","5","6"],"logits":[-4.762718200683594,-4.451813220977783,-0.7537885904312134,-4.693892478942871,-4.789631366729736,-4.968771934509277],"probs":[0.016580715775489807,0.022627033293247223,0.9133965373039246,0.017762085422873497,0.016140427440404892,0.013493211008608341],"argmax_index":2,"argmax_id":"3","top2_margin":0.8907695040106773,"near_tie":false,"gold":"3","gold_correct":true},{"question_id":"split_bill","type":"noul","question_span":[366,372],"option_spans":[[383,401],[407,425]],"option_ids":["true","false"],"logits":[-1.2601420879364014,-3.9936697483062744],"probs":[0.9389763474464417,0.06102372333407402],"argmax_index":0,"argmax_id":"true","top2_margin":0.8779526241123676,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"split_ways","type":"choice","question_span":[448,458],"option_spans":[[469,479],[485,495],[501,511]],"option_ids":["2","3","4"],"logits":[-0.5901689529418945,-4.390591144561768,-4.814335346221924],"probs":[0.9643211960792542,0.02156350575387478,0.014115282334387302],"argmax_index":0,"argmax_id":"2","top2_margin":0.9427576903253794,"near_tie":false,"gold":"2","gold_correct":true},{"question_id":"occasion","type":"choice","question_span":[532,537],"option_spans":[[548,560],[566,577],[583,595],[601,613],[619,631]],"option_ids":["anniversary","birthday","business","graduation","none"],"logits":[-4.306947708129883,-0.4257246255874634,-4.6238508224487305,-4.749966621398926,-3.625526189804077],"probs":[0.018928399309515953,0.9177147746086121,0.013787471689283848,0.012153832241892815,0.037415508180856705],"argmax_index":1,"argmax_id":"birthday","top2_margin":0.8802992664277554,"near_tie":false,"gold":"birthday","gold_correct":true},{"question_id":"kitchen_effort","type":"score","question_span":[654,663],"option_spans":[[674,685],[691,701],[707,718],[724,735],[741,751]],"option_ids":["0","1","2","3","4"],"logits":[-3.509247064590454,-2.9446372985839844,-1.7754170894622803,-2.7404539585113525,-2.7882497310638428],"probs":[0.0791470855474472,0.139200821518898,0.4481544494628906,0.1707330346107483,0.16276465356349945],"argmax_index":2,"argmax_id":"2","top2_margin":0.27742141485214233,"near_tie":false,"gold":null,"gold_correct":null},{"question_id":"dessert_planned","type":"noul","question_span":[775,782],"option_spans":[[793,811],[817,835]],"option_ids":["true","false"],"logits":[-2.388313055038452,-4.016894340515137],"probs":[0.835975170135498,0.16402478516101837],"argmax_index":0,"argmax_id":"true","top2_margin":0.6719503849744797,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"child_meal","type":"choice","question_span":[858,864],"option_spans":[[875,886],[892,907],[913,926],[932,945]],"option_ids":["burger","lasagne","plain_pasta","trout"],"logits":[-4.761372089385986,-4.9939165115356445,-0.08394628763198853,-4.882278919219971],"probs":[0.009076745249330997,0.007193453144282103,0.9756867289543152,0.008043053559958935],"argmax_index":2,"argmax_id":"plain_pasta","top2_margin":0.9666099837049842,"near_tie":false,"gold":"plain_pasta","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"dairy_allergy":{"type":"noul","noul":0.9395},"guests":{"type":"choice","choice":"3","confidence":0.9134,"probabilities":{"1":0.0166,"2":0.0226,"3":0.9134,"4":0.0178,"5":0.0161,"6":0.0135}},"split_bill":{"type":"noul","noul":0.939},"split_ways":{"type":"choice","choice":"2","confidence":0.9643,"probabilities":{"2":0.9643,"3":0.0216,"4":0.0141}},"occasion":{"type":"choice","choice":"birthday","confidence":0.9177,"probabilities":{"none":0.0374,"birthday":0.9177,"anniversary":0.0189,"business":0.0138,"graduation":0.0122}},"kitchen_effort":{"type":"score","score":2.1988,"confidence":0.4482,"legend":{"0":"trivial","1":"easy","2":"moderate","3":"involved","4":"complex"},"probabilities":{"0":0.0791,"1":0.1392,"2":0.4482,"3":0.1707,"4":0.1628}},"dessert_planned":{"type":"noul","noul":0.836},"child_meal":{"type":"choice","choice":"plain_pasta","confidence":0.9757,"probabilities":{"plain_pasta":0.9757,"lasagne":0.0072,"trout":0.008,"burger":0.0091}}},"usage":{"input_tokens":967,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_t13","arm":"text","set":"fixture","source":"own_text","tokens":370,"n_image_tokens":0,"state_tokens":53,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,33061,25,279,5395,1740,5926,506,3135,474,33604,27652,369,7629,364,29201,494,5361,220,17,310,5361,220,21,13,16454,1189,5926,303,279,10663,2609,26,14811,1220,958,279,8329,13,561,23213,12361,506,279,3008,19204,668,381,25331,383,5361,220,18,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,1740,605,812,10984,5206,2792,62,19,198,9222,25,307,10644,198,669,55530,25,220,6702,605,812,10984,5206,2792,62,19,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,8329,605,32342,198,9222,25,5576,198,669,55530,25,220,14749,1220,5926,383,279,8329,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,31929,2129,1979,816,3147,88787,8934,198,32929,220,17,25,220,4754,4532,3147,32108,2129,1979,816,3147,26250,8934,198,32929,220,18,25,220,4754,4532,3147,3020,11566,2129,1979,816,3147,2595,11566,8934,198,4537,38945,198,198,27806,220,18,198,887,25,41696,198,9222,25,5284,198,669,55530,25,220,4199,59465,369,279,21278,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,44149,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,35299,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"car_park_open_june_4","type":"noul","question_span":[121,129],"option_spans":[[140,158],[164,182]],"option_ids":["true","false"],"logits":[-4.499675750732422,-0.36167627573013306],"probs":[0.015704181045293808,0.9842958450317383],"argmax_index":1,"argmax_id":"false","top2_margin":0.9685916639864445,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"street_parking","type":"choice","question_span":[205,212],"option_spans":[[223,233],[239,249],[255,267]],"option_ids":["everyone","staff","visitors"],"logits":[-4.520214080810547,-4.956259727478027,-0.05747881531715393],"probs":[0.01131593156605959,0.007316748145967722,0.981367290019989],"argmax_index":2,"argmax_id":"visitors","top2_margin":0.9700513584539294,"near_tie":false,"gold":"visitors","gold_correct":true},{"question_id":"disruption","type":"score","question_span":[288,294],"option_spans":[[305,315],[321,332],[338,348]],"option_ids":["0","1","2"],"logits":[-1.7137874364852905,-0.17640650272369385,-2.580592393875122],"probs":[0.1646718978881836,0.7661177515983582,0.06921032816171646],"argmax_index":1,"argmax_id":"1","top2_margin":0.6014458537101746,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"car_park_open_june_4":{"type":"noul","noul":0.0157},"street_parking":{"type":"choice","choice":"visitors","confidence":0.9814,"probabilities":{"staff":0.0073,"visitors":0.9814,"everyone":0.0113}},"disruption":{"type":"score","score":0.9045,"confidence":0.7661,"legend":{"0":"minor","1":"moderate","2":"major"},"probabilities":{"0":0.1647,"1":0.7661,"2":0.0692}}},"usage":{"input_tokens":370,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_t14","arm":"text","set":"fixture","source":"own_text_long","tokens":2389,"n_image_tokens":0,"state_tokens":1893,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,15,15,25,15,19,13,15,24,19,57,29765,220,9861,23157,406,1622,816,30101,12,16,15,17,18,23,12500,593,7667,2552,28,17,15,16,37972,20759,28,22,22,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,15,16,25,15,21,13,16,23,20,57,29765,220,13767,2294,11700,1622,816,30101,12,16,15,17,19,20,12500,593,50274,2552,28,17,15,15,37972,20759,28,16,22,22,7087,1201,15480,28,16,18,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,15,17,25,15,23,13,17,22,21,57,29765,220,9861,23157,406,1622,816,30101,12,16,15,17,20,17,12500,593,7667,2552,28,17,15,16,37972,20759,28,21,24,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,15,18,25,16,15,13,18,21,22,57,29765,220,14991,1284,7130,256,1622,816,30101,12,16,15,17,20,24,7641,593,13091,15797,34149,5084,21,2552,28,17,15,15,37972,20759,28,19,16,6297,28,21793,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,15,19,25,16,17,13,19,20,23,57,29765,220,9861,23157,406,1622,816,30101,12,16,15,17,21,21,12500,593,7667,2552,28,17,15,16,37972,20759,28,22,23,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,15,20,25,16,19,13,20,19,24,57,29765,220,13767,2294,11700,1622,816,30101,12,16,15,17,22,18,12500,593,50274,2552,28,17,15,15,37972,20759,28,17,16,19,7087,1201,15480,28,16,22,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,15,21,25,16,21,13,21,19,15,57,29765,220,9861,23157,406,1622,816,30101,12,16,15,17,23,15,12500,593,7667,2552,28,17,15,16,37972,20759,28,23,15,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,15,22,25,16,23,13,22,18,16,57,29765,220,14991,1284,7130,256,1622,816,30101,12,16,15,17,23,22,7641,593,13091,15797,34149,5084,21,2552,28,17,15,15,37972,20759,28,20,15,6297,28,21793,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,15,24,25,15,17,13,19,16,15,57,66191,220,13767,2294,11700,280,24,20,72285,20759,28,16,17,16,15,11823,20759,28,23,15,15,7087,1201,15480,28,16,22,21,7087,2291,28,17,15,15,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,15,24,25,18,15,13,23,17,17,57,29765,220,9861,23157,406,1622,816,30101,12,16,15,17,24,19,12500,593,7667,2552,28,17,15,16,37972,20759,28,16,15,15,17,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,15,24,25,19,19,13,24,16,18,57,29765,220,9861,23157,406,1622,816,30101,12,16,15,18,15,16,12500,593,7667,2552,28,17,15,16,37972,20759,28,24,19,17,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,16,25,20,17,13,15,16,23,57,66191,220,13767,2294,11700,3511,7087,47821,7087,1201,15480,28,17,15,15,7087,2291,28,17,15,15,8318,28,18,22,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,17,25,15,22,13,17,18,18,57,12505,13767,2294,11700,40352,4415,1412,25,3511,15578,318,4791,12,28513,7087,36667,8,4621,28,16,14,18,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,17,25,15,24,13,15,15,19,57,12505,9861,23157,406,1622,816,30101,12,16,15,18,15,23,12500,593,7667,2552,28,20,15,17,1412,28,13360,22225,1954,33900,40352,28,60917,2294,11700,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,17,25,18,24,13,15,24,20,57,12505,13767,2294,11700,1622,816,30101,12,16,15,18,16,20,12500,593,50274,2552,28,20,15,18,1412,28,446,3892,4748,10028,4621,28,17,14,18,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,18,25,15,24,13,16,23,21,57,66191,220,9861,23157,406,1622,816,30101,12,16,15,18,17,17,21979,13145,1142,1783,20759,28,22,20,15,2781,28,13360,22225,1954,33900,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,18,25,18,24,13,17,22,22,57,12505,9861,23157,406,1622,816,30101,12,16,15,18,17,24,12500,593,7667,2552,28,20,15,17,1412,28,13360,22225,1954,33900,40352,28,60917,2294,11700,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,19,25,15,24,13,18,21,23,57,12505,13767,2294,11700,1622,816,30101,12,16,15,18,18,21,12500,593,50274,2552,28,20,15,18,1412,28,446,3892,4748,10028,4621,28,17,14,18,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,19,25,16,15,13,15,15,20,57,29765,220,15002,63307,256,55249,220,19,16,5813,378,58189,3683,28,13360,21510,290,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,19,25,18,24,13,19,20,24,57,66191,220,9861,23157,406,1622,816,30101,12,16,15,18,19,18,21979,13145,1142,1783,20759,28,20,15,15,2781,28,13360,22225,1954,33900,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,20,25,15,24,13,20,20,15,57,12505,9861,23157,406,1622,816,30101,12,16,15,18,20,15,12500,593,7667,2552,28,20,15,17,1412,28,13360,22225,1954,33900,40352,28,60917,2294,11700,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,20,25,16,15,13,15,22,22,57,29765,220,4953,286,285,2081,3106,310,383,56905,12331,28,60917,28943,28,11553,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,20,25,18,24,13,21,19,16,57,12505,13767,2294,11700,1622,816,30101,12,16,15,18,20,22,12500,593,50274,2552,28,20,15,18,1412,28,446,3892,4748,10028,4621,28,17,14,18,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,21,25,15,24,13,22,18,17,57,66191,220,9861,23157,406,1622,816,30101,12,16,15,18,21,19,21979,13145,1142,1783,20759,28,17,20,15,2781,28,13360,22225,1954,33900,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,21,25,18,24,13,23,17,18,57,12505,9861,23157,406,1622,816,30101,12,16,15,18,22,16,12500,593,7667,2552,28,20,15,17,1412,28,13360,22225,1954,33900,40352,28,60917,2294,11700,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,22,25,18,15,13,19,19,15,57,29765,220,13767,2294,11700,16526,10897,539,383,56905,2124,15476,28,9979,2291,25,17,15,15,397,19,15,15,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,23,25,19,16,13,24,15,17,57,29765,220,13767,2294,11700,1817,3727,2243,28,20,13,16,17,13,18,7087,2291,28,19,15,15,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,24,25,15,17,13,16,16,21,57,29765,220,13767,2294,11700,2730,1716,5642,40352,28,4791,12,28513,37972,20759,28,23,23,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,24,25,16,15,13,24,16,19,57,29765,220,9861,23157,406,1622,816,30101,12,16,15,18,22,23,12500,593,7667,2552,28,17,15,16,37972,20759,28,23,20,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,16,24,25,20,15,13,15,15,20,57,29765,220,9861,23157,406,1622,816,30101,12,16,15,18,23,20,12500,593,7667,2552,28,17,15,16,37972,20759,28,24,17,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,17,15,25,18,15,13,15,24,21,57,29765,220,13767,2294,11700,1622,816,30101,12,16,15,18,24,17,12500,593,50274,2552,28,17,15,15,37972,20759,28,16,15,15,7087,1201,15480,28,18,17,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,17,16,25,16,15,13,16,23,22,57,29765,220,9861,23157,406,1622,816,30101,12,16,15,18,24,24,12500,593,7667,2552,28,17,15,16,37972,20759,28,19,15,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,17,16,25,20,15,13,17,22,23,57,29765,220,9861,23157,406,1622,816,30101,12,16,15,19,15,21,21979,24526,3889,4620,28,20,15,17,13161,28,17,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,17,17,25,18,15,13,18,21,24,57,29765,220,9861,23157,406,1622,816,30101,12,16,15,19,16,18,12500,593,7667,2552,28,17,15,16,37972,20759,28,20,15,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,17,19,25,19,23,13,18,15,15,57,29765,220,9861,23157,406,68328,58371,25,220,18,16,17,9861,311,12,18510,11,220,15,4490,29754,198,17,15,17,21,12,15,24,12,16,23,51,15,18,25,17,20,25,15,18,13,15,16,17,57,29765,220,15002,63307,256,55249,220,19,16,5813,378,58189,3683,28,13360,47424,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,1118,736,13868,22678,198,9222,25,5576,198,669,55530,25,220,22365,3566,4490,1118,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16163,2393,2129,1979,816,3147,3137,8934,198,32929,220,17,25,220,4754,4532,3147,760,5839,4475,2129,1979,816,3147,11868,8934,198,32929,220,18,25,220,4754,4532,3147,760,14991,2393,2129,1979,816,3147,30075,8934,198,32929,220,19,25,220,4754,4532,3147,760,11208,11525,2129,1979,816,3147,36923,8934,198,32929,220,20,25,220,4754,4532,3147,760,13767,27853,2129,1979,816,3147,60917,8934,198,4537,38945,198,198,27806,220,17,198,887,25,24530,198,9222,25,307,10644,198,669,55530,25,220,1311,279,809,314,279,1433,279,20738,3566,682,24530,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,28943,198,9222,25,5284,198,669,55530,25,220,4199,14938,557,279,1484,364,6112,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,24275,278,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,44149,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,35299,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,40167,2129,1979,816,3147,18,8934,198,4537,38945,198,198,27806,220,19,198,887,25,1412,12211,198,9222,25,5576,198,669,55530,25,220,4199,1248,1467,279,5787,1483,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,3833,310,4097,4276,2129,1979,816,3147,16,2270,62,20,76,8934,198,32929,220,17,25,220,4754,4532,3147,35792,310,35442,4276,2129,1979,816,3147,20,2270,62,16,20,76,8934,198,32929,220,18,25,220,4754,4532,3147,7418,1056,35442,4276,2129,1979,816,3147,1908,62,16,20,76,8934,198,32929,220,19,25,220,4754,4532,3147,15771,799,9104,2129,1979,816,3147,7744,62,16,76,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"first_failing_component","type":"choice","question_span":[1956,1961],"option_spans":[[1972,1984],[1990,2002],[2008,2020],[2026,2038],[2044,2056]],"option_ids":["auth","database","inventory","notifications","payments"],"logits":[-4.381930828094482,-5.388772487640381,-5.561859130859375,-4.972869873046875,-0.35453853011131287],"probs":[0.017140639945864677,0.006262694951146841,0.005267331842333078,0.009492605924606323,0.9618366956710815],"argmax_index":4,"argmax_id":"payments","top2_margin":0.9446960557252169,"near_tie":false,"gold":"payments","gold_correct":true},{"question_id":"recovered","type":"noul","question_span":[2078,2090],"option_spans":[[2101,2119],[2125,2143]],"option_ids":["true","false"],"logits":[-0.5772510766983032,-3.5877504348754883],"probs":[0.9530462026596069,0.04695378988981247],"argmax_index":0,"argmax_id":"true","top2_margin":0.9060924127697945,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"severity","type":"score","question_span":[2164,2172],"option_spans":[[2183,2194],[2200,2210],[2216,2226],[2232,2242]],"option_ids":["0","1","2","3"],"logits":[-3.502514123916626,-2.8022656440734863,-0.3279270827770233,-2.008856773376465],"probs":[0.03186281770467758,0.06417977809906006,0.7620610594749451,0.14189638197422028],"argmax_index":2,"argmax_id":"2","top2_margin":0.6201646775007248,"near_tie":false,"gold":null,"gold_correct":null},{"question_id":"error_window","type":"choice","question_span":[2264,2271],"option_spans":[[2282,2299],[2305,2323],[2329,2346],[2352,2367]],"option_ids":["1_to_5m","5_to_15m","over_15m","under_1m"],"logits":[-1.4632933139801025,0.10061192512512207,-2.87170147895813,-4.480104923248291],"probs":[0.1647193729877472,0.7869372367858887,0.040279194712638855,0.00806417502462864],"argmax_index":1,"argmax_id":"5_to_15m","top2_margin":0.6222178637981415,"near_tie":false,"gold":"5_to_15m","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"first_failing_component":{"type":"choice","choice":"payments","confidence":0.9618,"probabilities":{"auth":0.0171,"database":0.0063,"payments":0.9618,"inventory":0.0053,"notifications":0.0095}},"recovered":{"type":"noul","noul":0.953},"severity":{"type":"score","score":2.014,"confidence":0.7621,"legend":{"0":"informational","1":"minor","2":"major","3":"critical"},"probabilities":{"0":0.0319,"1":0.0642,"2":0.7621,"3":0.1419}},"error_window":{"type":"choice","choice":"5_to_15m","confidence":0.7869,"probabilities":{"under_1m":0.0081,"1_to_5m":0.1647,"5_to_15m":0.7869,"over_15m":0.0403}}},"usage":{"input_tokens":2389,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_t15","arm":"text","set":"fixture","source":"own_text_long","tokens":2603,"n_image_tokens":0,"state_tokens":1836,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,87075,24817,72914,15950,318,69139,25,57042,220,16,310,220,16,21,8,271,1919,10507,8049,21800,318,1719,328,8770,16646,871,369,10321,1083,383,220,19,5297,220,17,15,17,21,318,1719,328,65261,2544,871,1881,5061,633,29364,8338,8049,11971,13,3376,5014,871,321,63227,273,19003,45004,382,82745,3376,2865,1764,22203,321,8163,513,1754,264,328,36126,1,321,3658,279,328,5619,536,3158,271,16,13,44959,198,16,13,16,328,22020,5870,1,3254,264,1834,975,1056,264,7483,11,264,7043,466,264,569,12872,506,279,9468,8163,2660,13,198,16,13,17,328,22020,28141,1,3254,220,15,23,25,15,15,310,220,16,23,25,15,15,383,264,7773,5870,13,198,16,13,18,328,15121,10908,7831,1,3254,866,1928,34218,539,799,8292,310,279,975,421,369,12495,430,25357,466,421,264,12891,1637,1000,3418,310,381,25357,13,271,17,13,8049,198,17,13,16,22203,668,7892,279,60361,16070,3401,7246,303,22777,357,318,1719,328,10703,3842,2583,279,13319,314,16665,1395,78,375,538,24296,506,279,8163,579,6386,11,279,5488,314,279,20367,16070,26103,11,321,2018,8187,10597,1761,2261,7773,28141,13,198,17,13,17,22203,1189,2222,279,1560,424,26437,279,8049,11,3766,421,279,2222,1503,524,86514,7698,836,14512,466,4610,13,22203,668,2873,8163,506,3140,25292,318,18,15,8,2756,6,5158,5125,314,866,1680,2222,13,271,18,13,16999,321,47344,278,198,18,13,16,561,21800,8211,383,279,45356,2544,321,9257,364,449,2763,4496,314,16570,39663,318,17,19,8,3818,318,1719,328,6140,16999,1764,198,18,13,17,4486,279,3991,16999,11,279,21800,311,9689,9184,364,47286,17800,314,28279,318,16,17,8,3818,318,9264,264,328,33479,360,278,16999,871,7014,2892,8292,6486,5158,5125,314,2397,5331,902,278,506,3140,75211,318,24,15,8,2756,1518,279,809,314,279,1179,42707,4496,13,198,18,13,18,22203,1189,5096,279,12079,364,264,47344,278,16999,539,874,777,1056,4097,3161,318,20,11007,888,279,12079,364,279,36497,28279,3818,11,3766,22203,93251,8163,314,279,5096,506,3140,799,7493,321,16570,318,16,17,15,8,2756,1518,279,47344,278,16999,11690,13,271,19,13,56599,321,19829,198,19,13,16,8163,668,2216,279,14647,12079,716,680,303,22777,417,13,22203,668,23839,14647,303,229488,1506,13,198,19,13,17,733,39339,513,43113,2785,25292,318,18,15,8,2756,314,279,23839,2321,13,24962,82,524,6945,948,4016,11385,2647,506,799,321,264,4139,3161,318,16,13,20,11007,791,2182,2980,6945,13,198,19,13,18,8163,1189,24240,449,23839,303,1603,9588,539,5158,5125,2785,35442,318,16,20,8,2756,314,21867,11,26895,1141,7715,13,561,57883,668,1380,310,8563,279,24240,2785,25292,318,18,15,8,2756,13,16393,284,18709,18740,6922,43113,383,279,3889,4016,2321,13,198,19,13,19,1368,866,78146,18709,3202,8198,52269,364,777,1056,34227,34129,318,19,20,8,2756,1238,1141,4016,2321,11,22203,1189,25410,279,8049,1238,6866,5600,318,16,15,8,2756,6,5158,5125,13,271,20,13,5194,50920,198,20,13,16,22203,668,1236,279,16070,26103,2420,220,24,24,13,20,4,314,279,854,303,1754,12786,2182,11,42328,13145,13017,421,369,7002,506,3140,4097,318,20,8,2756,303,11573,321,6973,310,7810,318,23,8,3979,791,2182,13,198,20,13,17,1368,17509,16563,3559,220,24,24,13,20,4,303,264,2182,11,8163,369,19262,310,264,2393,6459,314,4097,3161,318,20,11007,314,421,2182,579,12079,364,1754,2400,220,15,13,20,4,3559,279,2100,11,685,310,264,6966,314,16570,34129,3161,318,17,20,11007,314,421,2182,579,12079,13,198,20,13,18,5194,19528,513,8163,579,12924,37843,364,264,7652,310,3258,279,17509,2100,11,3525,1332,17509,16563,3559,220,24,20,4,303,2250,23173,3818,11,303,864,1105,8163,1189,28685,1172,10790,220,16,15,13,17,13,271,21,13,8163,70530,198,21,13,16,8163,668,2873,22203,579,50600,5902,2528,310,1754,2660,2261,7773,28141,11,440,506,3140,1330,318,17,8,7773,20542,6,5125,13,198,21,13,17,8163,369,8220,364,279,13016,314,279,2660,1928,424,5529,321,364,21496,279,24296,494,6745,5383,8613,539,1141,1765,5395,466,31655,13,271,22,13,70868,477,198,22,13,16,8618,8292,668,2426,279,975,8292,579,70868,7831,6038,321,958,424,1132,310,2649,466,5091,279,8049,13,198,22,13,17,4081,28201,17171,364,2250,318,18,8,1578,1238,279,21800,10029,11,3525,364,6355,22857,11,864,6922,2530,364,430,1248,430,781,6922,6355,22857,13,271,23,13,2791,198,23,13,16,8163,23495,660,11661,37780,321,6473,7658,364,1141,6386,3376,2865,2791,1764,22203,1189,958,8163,2791,1132,310,3300,279,8049,321,11,303,449,68422,1304,421,1503,524,10229,8163,466,866,2660,11,310,7042,1141,3741,13,198,23,13,17,1848,33265,11,22203,668,1236,8163,2791,2420,364,7250,364,50614,318,21,15,8,2756,321,668,1179,3572,424,11,7014,279,2253,7225,22203,310,2426,424,13,271,24,13,86946,198,24,13,16,23472,8292,369,33231,364,23838,466,86986,4557,11,2583,4557,314,29566,11,11044,466,12585,11,1442,413,421,4557,557,75238,13,198,24,13,17,8618,8292,579,2702,22686,38564,680,314,466,303,3511,440,411,21800,303,866,28279,21598,4031,369,6973,310,279,2702,12079,6945,466,43113,539,8163,303,279,28279,318,16,17,8,3818,1518,279,1484,6866,9704,310,279,3591,13,198,24,13,18,561,13001,303,411,10790,635,524,3668,310,22686,364,4397,466,4204,10577,8613,539,56298,11,364,15649,11,364,29382,314,10790,220,22,318,15121,10908,477,681,466,364,19344,56298,466,29245,1218,38665,13,271,16,15,13,9827,31039,198,16,15,13,16,20348,8292,1189,28685,279,21800,539,5158,5125,413,279,975,8292,40161,264,3558,29382,321,14107,310,37843,424,2785,25292,318,18,15,8,2756,1238,11955,5158,5125,21988,279,29382,13,198,16,15,13,17,8163,1189,28685,279,21800,440,13522,2376,539,5158,5125,303,279,12779,7246,303,10790,220,20,13,18,13,198,16,15,13,18,20348,8292,1189,28685,279,21800,6849,413,279,975,8292,8772,81918,666,466,3633,2191,310,6564,383,2478,13,198,16,15,13,19,9827,31039,1503,524,7556,12079,2582,4016,11,321,57042,220,22,11,220,23,13,17,11,220,24,321,220,16,17,17171,33265,13,271,16,16,13,33284,198,16,16,13,16,23472,8292,1189,9501,466,8059,411,21800,1973,279,4711,5158,14020,314,279,975,8292,11,864,1189,524,381,632,19290,2752,73402,13,198,16,16,13,17,22203,1189,11,4611,11,9501,279,21800,310,449,20935,1973,14020,539,6866,5158,5125,310,8163,11,3766,421,279,20935,369,2858,310,2649,279,8049,310,279,1788,5129,13,271,16,17,13,91966,321,90646,1187,7271,198,16,17,13,16,91966,1902,381,303,4237,321,12238,539,1375,11,539,71982,11,466,539,378,11139,310,279,13822,303,22777,351,13,357,5125,3106,539,378,11139,369,3816,383,279,1727,7773,5870,1238,424,369,3106,13,198,16,17,13,17,1061,21800,369,25849,539,279,6657,314,279,27095,303,864,22203,369,9393,11,321,279,18259,314,421,27095,599,13366,27095,888,866,24240,13,271,16,18,13,21008,198,16,18,13,16,22203,668,10000,569,22686,7857,314,506,3140,1330,3410,318,17,11,15,15,15,11,15,15,15,8,303,279,11085,314,22777,417,321,6376,85945,477,7857,314,506,3140,799,3410,318,16,11,15,15,15,11,15,15,15,8,364,279,7836,314,279,21800,321,364,799,318,16,8,1007,25950,13,198,16,18,13,17,1848,1622,11,22203,668,2873,8163,264,15286,314,7857,8755,421,279,3311,2483,539,411,10790,369,303,5176,13,271,16,19,13,11395,380,10735,538,198,16,19,13,16,23472,8292,369,33231,364,264,7383,466,7652,310,2649,8613,539,4216,7551,1141,12891,2444,11,2583,17198,11,35528,11,3807,11,13071,11,40146,11,3980,11,466,264,7652,314,569,28459,11,3766,421,424,93251,279,975,8292,38117,321,5533,12891,8601,310,15227,4906,13,198,16,19,13,17,1368,1680,449,1484,27028,4906,364,777,1056,50614,318,21,15,8,23173,2756,11,2892,8292,1189,28685,279,21800,539,5158,5125,1973,22686,13,271,16,20,13,3593,19641,286,321,46058,198,16,20,13,16,22203,1189,958,91119,1052,310,4433,466,10000,24296,11,3766,421,22203,8198,8220,364,836,944,430,413,424,998,1141,1765,321,421,1754,91119,269,369,6608,539,62028,28201,874,2599,27206,1056,10790,220,22,13,198,16,20,13,17,9358,303,1754,4954,1007,11,383,506,3140,16570,318,17,15,8,7773,20542,6,5158,5125,11,8163,1189,23513,22203,579,7189,21827,310,279,8049,321,310,279,11256,314,8163,2791,13,8618,8292,28754,1141,1765,6829,314,279,23513,11,7014,279,23513,20352,264,3558,29382,539,22203,11,303,864,1105,22203,668,91876,8163,579,12891,23513,6829,13,271,16,21,13,76521,21800,198,16,21,13,16,1061,21800,11,3658,440,1141,326,47942,11,369,279,4307,8855,1881,279,57883,883,1141,3704,4766,321,39356,660,6575,23497,321,3418,793,13,198,16,21,13,17,2233,26266,369,7139,7014,424,369,303,4237,321,8239,539,67034,22409,314,2107,57883,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,2397,1243,902,278,50063,198,9222,25,5576,198,669,55530,25,220,4199,1691,5125,369,4221,310,2842,279,16465,41386,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16,17,15,2756,2129,1979,816,3147,16,17,15,8934,198,32929,220,17,25,220,4754,4532,3147,18,15,2756,2129,1979,816,3147,18,15,8934,198,32929,220,18,25,220,4754,4532,3147,21,15,2756,2129,1979,816,3147,21,15,8934,198,32929,220,19,25,220,4754,4532,3147,24,15,2756,2129,1979,816,3147,24,15,8934,198,32929,220,20,25,220,4754,4532,3147,2665,5125,369,4221,2129,1979,816,3147,6487,8934,198,4537,38945,198,198,27806,220,17,198,887,25,3129,1243,9689,198,9222,25,307,10644,198,669,55530,25,220,760,8855,311,9689,9184,1238,279,2763,4496,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,22686,15907,198,9222,25,5576,198,669,55530,25,220,4199,369,1754,4464,579,22686,58118,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,12079,6945,466,43113,303,279,220,16,17,3818,1518,279,3591,2129,1979,816,3147,1795,287,62,16,17,76,8934,198,32929,220,17,25,220,4754,4532,3147,32,8097,3202,10661,303,279,8855,2129,1979,816,3147,21343,13083,8934,198,32929,220,18,25,220,4754,4532,3147,760,3202,314,279,8836,579,7857,3311,2129,1979,816,3147,67583,8934,198,32929,220,19,25,220,4754,4532,3147,3733,369,874,1994,2129,1979,816,3147,354,17693,8934,198,4537,38945,198,198,27806,220,19,198,887,25,9501,38581,3273,23737,198,9222,25,307,10644,198,669,55530,25,220,47587,4464,1189,9501,279,8855,24765,1973,279,975,579,14020,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,20,198,887,25,2849,1650,3086,198,9222,25,5284,198,669,55530,25,220,4199,78386,513,279,3665,310,279,2849,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1164,9194,25167,470,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,354,69,25167,470,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,56435,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,69,25167,470,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,1164,78386,2129,1979,816,3147,19,8934,198,4537,38945,198,198,27806,220,21,198,887,25,83697,10800,198,9222,25,5576,198,669,55530,25,220,3710,17509,1503,279,8836,5099,310,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,24,24,13,15,4,2129,1979,816,3147,24,24,13,15,8934,198,32929,220,17,25,220,4754,4532,3147,24,24,13,20,4,2129,1979,816,3147,24,24,13,20,8934,198,32929,220,18,25,220,4754,4532,3147,24,24,13,24,4,2129,1979,816,3147,24,24,13,24,8934,198,32929,220,19,25,220,4754,4532,3147,24,24,13,24,24,4,2129,1979,816,3147,24,24,13,24,24,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"non_renewal_notice","type":"choice","question_span":[1900,1911],"option_spans":[[1922,1937],[1943,1956],[1962,1975],[1981,1994],[2000,2013]],"option_ids":["120","30","60","90","none"],"logits":[-3.9560117721557617,-4.408750534057617,-4.585799217224121,1.5387500524520874,-4.386011600494385],"probs":[0.0040611992590129375,0.0025824527256190777,0.0021634213626384735,0.9885510802268982,0.002641847589984536],"argmax_index":3,"argmax_id":"90","top2_margin":0.9844898809678853,"near_tie":false,"gold":"90","gold_correct":true},{"question_id":"auto_renews","type":"noul","question_span":[2037,2047],"option_spans":[[2058,2076],[2082,2100]],"option_ids":["true","false"],"logits":[-0.8675358295440674,-3.808976650238037],"probs":[0.9498574137687683,0.05014260485768318],"argmax_index":0,"argmax_id":"true","top2_margin":0.8997148089110851,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"liability_cap","type":"choice","question_span":[2122,2130],"option_spans":[[2141,2169],[2175,2192],[2198,2215],[2221,2235]],"option_ids":["fees_12m","fixed_amount","insurance","unlimited"],"logits":[0.706987202167511,-4.302256107330322,-4.53178596496582,-4.146567344665527],"probs":[0.9806005358695984,0.006546442396938801,0.0052038137800991535,0.007649271748960018],"argmax_index":0,"argmax_id":"fees_12m","top2_margin":0.9729512641206384,"near_tie":false,"gold":"fees_12m","gold_correct":true},{"question_id":"assign_without_consent","type":"noul","question_span":[2260,2273],"option_spans":[[2284,2302],[2308,2326]],"option_ids":["true","false"],"logits":[-2.9932966232299805,1.0375432968139648],"probs":[0.0174495130777359,0.9825504422187805],"argmax_index":1,"argmax_id":"false","top2_margin":0.9651009291410446,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"client_risk","type":"score","question_span":[2349,2358],"option_spans":[[2369,2382],[2388,2401],[2407,2417],[2423,2435],[2441,2452]],"option_ids":["0","1","2","3","4"],"logits":[-2.6278955936431885,-1.7573552131652832,-0.11282582581043243,-0.40905189514160156,-2.1929948329925537],"probs":[0.03773987665772438,0.09013041108846664,0.46674680709838867,0.3470819592475891,0.0583009235560894],"argmax_index":2,"argmax_id":"2","top2_margin":0.11966484785079956,"near_tie":false,"gold":null,"gold_correct":null},{"question_id":"uptime_target","type":"choice","question_span":[2474,2482],"option_spans":[[2493,2510],[2516,2533],[2539,2556],[2562,2581]],"option_ids":["99.0","99.5","99.9","99.99"],"logits":[-3.340916156768799,1.581406831741333,-3.498359203338623,-3.5589675903320312],"probs":[0.007143894210457802,0.9810086488723755,0.006103210616856813,0.0057442947290837765],"argmax_index":1,"argmax_id":"99.5","top2_margin":0.9738647546619177,"near_tie":false,"gold":"99.5","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"non_renewal_notice":{"type":"choice","choice":"90","confidence":0.9886,"probabilities":{"30":0.0026,"60":0.0022,"90":0.9886,"120":0.0041,"none":0.0026}},"auto_renews":{"type":"noul","noul":0.9499},"liability_cap":{"type":"choice","choice":"fees_12m","confidence":0.9806,"probabilities":{"unlimited":0.0076,"fees_12m":0.9806,"fixed_amount":0.0065,"insurance":0.0052}},"assign_without_consent":{"type":"noul","noul":0.0174},"client_risk":{"type":"score","score":2.2981,"confidence":0.4667,"legend":{"0":"very unfavourable","1":"unfavourable","2":"balanced","3":"favourable","4":"very favourable"},"probabilities":{"0":0.0377,"1":0.0901,"2":0.4667,"3":0.3471,"4":0.0583}},"uptime_target":{"type":"choice","choice":"99.5","confidence":0.981,"probabilities":{"99.0":0.0071,"99.5":0.981,"99.9":0.0061,"99.99":0.0057}}},"usage":{"input_tokens":2603,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_t16","arm":"text","set":"fixture","source":"own_text_long","tokens":2494,"n_image_tokens":0,"state_tokens":1811,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,3496,3718,841,8431,11261,8488,15284,5160,34384,61824,3381,198,26822,314,279,24157,3146,28107,11,5467,383,7483,220,17,15,5930,220,17,15,17,21,506,220,16,15,25,15,15,303,279,3809,13592,13,271,20545,25,220,18,19,3492,318,1719,893,31913,369,220,17,20,553,5065,8817,25,220,21,3492,13,733,279,10183,25,36306,496,636,35420,13,39645,4284,539,93188,357,1474,1323,7869,1199,318,19849,640,553,271,16,13,19553,321,69721,198,760,10183,8660,279,6236,506,220,16,15,25,15,19,11,29952,2943,491,6868,27770,321,1301,279,69721,13,2838,29171,3492,421,1132,3492,6483,12079,364,220,17,15,17,21,513,6945,1189,6695,26,279,18494,10457,421,660,220,18,19,3492,2945,513,19262,310,6695,13,271,17,13,39645,314,279,3555,14642,44,198,760,4276,314,279,14642,44,314,220,16,19,5930,220,17,15,17,20,998,11454,1973,26266,318,751,3736,539,40414,12909,11527,71370,11,2018,290,539,8105,50581,457,1223,24524,839,72,553,271,18,13,84847,579,1830,198,760,92094,11,220348,246586,11,10137,279,9331,364,279,1007,310,220,18,16,5963,220,17,15,17,21,13,31447,557,6722,23,11,19,17,15,318,4321,12079,6722,22,11,16,20,15,11,6416,506,279,2660,7808,6722,16,11,15,19,15,11,23209,6722,17,18,15,553,1326,3613,9004,557,6722,21,11,24,23,15,318,12615,6722,17,11,18,16,15,11,7857,6722,16,11,16,17,15,11,29201,6722,16,11,23,24,15,11,10390,17447,6722,22,23,15,11,48879,3884,6722,23,23,15,553,1737,12895,506,279,1007,809,14217,506,6722,16,16,11,21,15,15,13,198,33613,3492,4439,3069,279,2919,3896,995,39045,539,220,16,23,13948,561,92094,10921,421,1379,314,279,5096,3571,494,35479,383,279,9897,1752,11,1332,2919,10298,364,3679,2756,1518,1754,22627,557,1669,13,16457,1048,4439,883,279,7808,25,6416,998,685,1521,314,279,491,10014,37452,1906,11,321,279,7808,12433,668,12771,424,1727,10151,13,198,760,9331,998,11330,318,751,3736,539,192328,42762,800,11,2018,290,539,24328,11698,677,681,440,799,60729,2845,13,271,19,13,9664,1785,198,760,1822,13254,3055,279,9897,1752,32728,2250,2942,411,1007,11,321,279,2418,13372,506,6868,220,19,16,4490,303,5589,13,561,12433,10137,1330,2519,25,198,2784,8,12368,279,19510,13753,314,279,9897,1542,11,12572,506,883,6722,16,11,19,15,15,11,440,264,4222,1144,364,4473,29201,2785,1330,466,2250,1578,26,198,1818,8,8032,279,4220,9897,1542,440,491,9569,84523,13254,321,884,1330,2418,83728,3043,29693,220,16,17,321,220,18,15,11,12572,506,883,6722,20,11,24,15,15,11,6945,494,28538,13,198,623,279,9915,11,1010,3492,6273,421,2904,318,65,8,1000,5106,279,28538,2166,3238,413,279,13592,14693,4221,944,13,24235,13874,680,421,279,11173,35479,998,2582,51922,3117,321,421,279,491,2418,83728,1000,1438,8742,3492,852,1381,6564,2919,264,1248,1560,13,561,92094,10457,421,28538,1000,6922,3294,6722,20,11,15,15,15,1238,2904,318,65,553,198,38,16114,11698,677,4439,3315,279,14575,1362,3668,364,264,12816,494,279,9014,14403,579,5983,12258,3674,13,561,18494,1018,421,279,3674,311,23417,303,5869,321,5802,14103,685,310,4139,314,279,2695,314,6511,4138,364,3809,5038,11,694,421,264,5307,1000,524,2445,1518,5632,11,2166,3200,364,279,7165,944,13,16457,7003,421,279,12433,1220,1990,3668,11,321,421,866,12816,3816,1000,708,1142,1083,279,28538,13,198,49,910,12909,11527,71370,4439,852,1000,6564,680,279,944,13,561,12433,41804,310,17447,264,28994,364,279,51463,286,321,12850,11,1345,22293,668,2708,279,1752,321,14800,2374,279,26075,25950,26,6868,27770,3055,279,9897,1752,668,381,3125,506,3140,2943,5381,303,11573,13,198,760,10183,2113,2904,318,65,8,310,279,6695,25,364,220,17,16,11,2272,220,16,16,11,60729,61552,220,17,13,561,11051,310,8032,279,9897,1542,557,11354,13,561,944,668,381,13145,364,7165,11,948,279,2919,7892,369,28898,974,364,279,12050,11,321,279,12433,668,6637,506,3140,1330,16687,13,271,20,13,25201,12079,198,760,12433,10660,17987,279,9483,10737,364,264,2400,6868,494,6722,19,20,310,6722,20,17,11,321,364,264,4139,6868,494,6722,17,20,310,6722,17,24,11,494,5869,220,17,15,17,22,11,310,30456,279,28538,1238,279,2919,4138,13,198,23602,17736,421,279,5096,557,2166,3349,303,264,3074,1007,321,421,279,7808,39110,321,23209,1220,381,1429,1118,13,357,23032,310,4673,12079,539,264,8882,3202,557,524,2113,430,264,15440,11051,13,198,40041,383,279,12433,579,13340,25,364,220,16,17,11,2272,220,16,24,11,60729,61552,220,18,13,561,11051,557,22559,11,321,12079,6922,33702,364,220,17,15,17,22,13,561,12433,668,4370,3286,279,7857,41386,11,864,1189,381,22330,440,264,2086,21623,11,321,1830,1142,506,279,10151,6236,13,271,21,13,50433,1103,198,3733,513,220,17,18,1208,383,279,8318,1103,11,321,279,5289,3655,369,1381,220,16,19,3818,13,561,12433,10660,421,2400,29693,864,3512,47066,381,6507,1083,1330,4139,29693,1332,279,6576,5954,11,321,421,29693,2962,310,381,8632,310,279,8318,1103,24660,303,1906,314,3638,13,357,4314,11488,6866,10304,310,10509,314,279,13721,26,1238,9915,279,6236,7003,310,2426,279,1906,314,3638,13,24876,3492,1018,421,4139,29693,7535,65180,2577,11,1521,264,2400,6868,369,264,2609,314,944,303,279,1118,1007,321,3679,491,27770,6348,685,2785,28279,3818,13,8105,50581,457,1223,24524,839,72,8632,310,34256,491,6868,27770,2261,836,1118,3098,11,321,2250,975,3492,71166,310,4973,1384,13,561,12433,668,1048,2466,1208,383,279,1103,2957,264,1007,3315,781,1990,1318,264,6868,11,748,421,279,1103,25710,12995,13,198,760,13340,557,7003,539,264,1420,314,5888,11,440,874,12320,2272,13,271,22,13,1158,2134,321,7567,53221,198,760,12433,10660,421,279,19074,314,5983,11939,383,2660,381,20199,494,220,16,6492,220,17,15,17,21,11,321,421,54826,89446,2785,1330,35058,314,279,18169,40289,381,12215,539,7629,54826,27588,11,2614,20522,494,34667,883,15728,321,30378,13,561,14575,668,3569,5600,27588,303,18985,321,4410,1070,310,3492,506,2695,13,16457,852,4021,54826,22996,1041,534,354,793,1189,5106,1070,506,279,509,5477,2982,539,279,1822,17587,11,864,668,381,90636,10598,264,1007,13,198,760,13340,557,7003,440,220,17,12320,2272,13,271,23,13,24934,314,9324,198,42767,10975,496,636,35420,7002,421,1292,369,34294,1441,430,10183,1238,4590,1578,13,8771,57584,995,978,3816,25,26882,88,25819,17041,261,2482,318,751,3736,539,40414,12909,11527,71370,11,2018,290,539,192328,42762,800,8,321,6920,367,73552,318,751,3736,539,24328,11698,677,11,2018,290,539,8105,50581,457,1223,24524,839,72,553,11396,10853,11938,25899,13,357,25029,557,5467,25,26882,88,25819,17041,261,2482,220,17,17,12320,11,6920,367,73552,220,16,17,12320,13,26882,88,25819,17041,261,2482,557,15811,10183,13,198,56,350,1644,246586,557,311,92394,92094,632,445,3736,13,93188,357,1474,1323,7869,1199,557,311,92394,18494,632,445,3736,13,6920,367,73552,7003,310,4973,279,12433,430,449,18541,4314,13,198,760,6236,54601,36306,496,636,35420,364,1024,1578,303,279,10183,26,1292,668,2962,310,1542,279,5819,6080,13,271,24,13,5586,975,2478,198,2784,8,91189,4610,25,1330,73342,998,10548,1083,303,5963,321,7141,998,4284,13,16457,998,4439,310,1804,836,7141,321,524,310,5106,10106,383,2660,13,561,12433,668,3240,264,491,17587,5128,13,198,1818,8,5097,1834,25,449,1724,1834,364,2136,8259,369,12539,364,7483,220,16,18,5361,220,17,15,17,22,13,82232,1220,2873,836,4874,310,279,18494,13,198,1290,8,25201,55068,25,279,12433,668,6564,680,279,13087,55068,303,3114,321,5963,13,25201,27770,6483,29693,513,2599,1056,4139,63836,668,5091,264,25680,6321,11,321,264,2018,6321,1238,2943,5381,413,4161,682,5324,13,16457,4439,421,279,24865,12521,381,16218,383,279,5125,4331,26,279,18494,7003,310,635,748,13,198,1446,8,10791,998,12076,310,279,22293,852,22235,279,888,59207,29693,303,279,10151,321,310,279,3492,852,5395,279,7808,13,271,2432,3470,198,12,10035,25,6637,506,3140,1330,16687,364,24101,279,9897,2919,1542,11,539,220,18,15,6492,220,17,15,17,21,13,198,12,220348,246586,25,9150,7857,41386,16687,321,1830,506,279,10151,6236,13,198,12,93188,357,1474,1323,7869,1199,25,3165,310,3492,383,279,8318,1103,883,279,4139,12,4321,12490,11,539,220,18,16,6324,220,17,15,17,21,13,198,12,26882,88,25819,17041,261,2482,25,1906,5600,7629,54826,27588,11,539,220,16,20,6324,220,17,15,17,21,13,198,12,10035,25,3240,264,491,17587,5128,11,539,279,1727,12433,6236,13,271,760,6236,7629,506,220,16,17,25,16,21,13,561,1727,12433,6236,668,381,5467,383,7348,220,22,6324,220,17,15,17,21,506,220,16,24,25,18,15,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,2919,87519,198,9222,25,5576,198,669,55530,25,220,3710,557,6426,883,279,2919,1785,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,2665,5307,557,1801,2129,1979,816,3147,2083,87519,8934,198,32929,220,17,25,220,4754,4532,3147,3997,78126,279,5307,310,1727,1007,2129,1979,816,3147,2134,78126,8934,198,32929,220,18,25,220,4754,4532,3147,94987,279,19510,13753,2129,1979,816,3147,73825,8934,198,32929,220,19,25,220,4754,4532,3147,22386,279,9897,1542,321,884,2418,83728,2129,1979,816,3147,7899,8934,198,4537,38945,198,198,27806,220,17,198,887,25,12079,71299,198,9222,25,307,10644,198,669,55530,25,220,77720,6868,12079,668,5096,303,220,17,15,17,22,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,491,4002,1264,198,9222,25,5576,198,669,55530,25,220,14749,369,279,491,10183,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,56,350,1644,246586,2129,1979,816,3147,171573,278,8934,198,32929,220,17,25,220,4754,4532,3147,42767,10975,496,636,35420,2129,1979,816,3147,935,35420,8934,198,32929,220,18,25,220,4754,4532,3147,6994,367,73552,2129,1979,816,3147,7820,287,8934,198,32929,220,19,25,220,4754,4532,3147,89680,88,25819,17041,261,2482,2129,1979,816,3147,82,24554,2482,8934,198,4537,38945,198,198,27806,220,19,198,887,25,2919,52536,30919,13233,198,9222,25,5284,198,669,55530,25,220,4199,1691,8855,557,1017,303,279,2919,6695,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,31788,391,17249,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,6744,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,7187,8421,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,354,15496,757,2129,1979,816,3147,18,8934,198,4537,38945,198,198,27806,220,20,198,887,25,8318,1951,7621,198,9222,25,5576,198,669,55530,25,220,4199,668,47066,29693,381,8632,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1206,279,8318,1103,303,1906,314,3638,2129,1979,816,3147,4968,7621,8934,198,32929,220,17,25,220,4754,4532,3147,2278,10304,364,10509,314,279,13721,2129,1979,816,3147,2359,36899,8934,198,32929,220,18,25,220,4754,4532,3147,1311,12945,9780,2129,1979,816,3147,8911,17090,8934,198,4537,38945,198,198,27806,220,21,198,887,25,7567,10479,853,276,198,9222,25,307,10644,198,669,55530,25,220,33,51554,5983,11939,383,2660,668,381,20199,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"water_decision","type":"choice","question_span":[1872,1880],"option_spans":[[1891,1905],[1911,1928],[1934,1947],[1953,1970]],"option_ids":["no_decision","postpone","repair","replace"],"logits":[-4.7417097091674805,-4.700109958648682,-4.521092414855957,1.1604564189910889],"probs":[0.002709157532081008,0.002824236173182726,0.003377902554348111,0.9910887479782104],"argmax_index":3,"argmax_id":"replace","top2_margin":0.9877108454238623,"near_tie":false,"gold":"replace","gold_correct":true},{"question_id":"fees_increase","type":"noul","question_span":[1993,2005],"option_spans":[[2016,2034],[2040,2058]],"option_ids":["true","false"],"logits":[-3.3435232639312744,1.2454297542572021],"probs":[0.010061236098408699,0.9899387359619141],"argmax_index":1,"argmax_id":"false","top2_margin":0.9798774998635054,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"new_chair","type":"choice","question_span":[2081,2087],"option_spans":[[2098,2112],[2118,2133],[2139,2152],[2158,2175]],"option_ids":["kemal","ollerton","reyes","somerfield"],"logits":[-5.380885124206543,-4.996088027954102,-4.5313801765441895,0.5311657786369324],"probs":[0.0026718617882579565,0.003925809636712074,0.006248118821531534,0.9871541857719421],"argmax_index":3,"argmax_id":"somerfield","top2_margin":0.9809060669504106,"near_tie":false,"gold":"somerfield","gold_correct":true},{"question_id":"water_vote_consensus","type":"score","question_span":[2199,2209],"option_spans":[[2220,2232],[2238,2248],[2254,2265],[2271,2283]],"option_ids":["0","1","2","3"],"logits":[-2.21404767036438,-2.1970598697662354,0.3940242528915405,-3.7331907749176025],"probs":[0.06325555592775345,0.06433930993080139,0.8585585355758667,0.013846603222191334],"argmax_index":2,"argmax_id":"2","top2_margin":0.7942192256450653,"near_tie":false,"gold":"2","gold_correct":true},{"question_id":"waiting_list_order","type":"choice","question_span":[2306,2313],"option_spans":[[2324,2342],[2348,2365],[2371,2384]],"option_ids":["application_order","local_priority","lottery"],"logits":[0.6724512577056885,-4.505770206451416,-5.589963912963867],"probs":[0.9925118684768677,0.005595805589109659,0.0018923586467280984],"argmax_index":0,"argmax_id":"application_order","top2_margin":0.986916062887758,"near_tie":false,"gold":"application_order","gold_correct":true},{"question_id":"bonfire_ban","type":"noul","question_span":[2409,2419],"option_spans":[[2430,2448],[2454,2472]],"option_ids":["true","false"],"logits":[0.39692556858062744,-3.1971936225891113],"probs":[0.9732503294944763,0.02674967236816883],"argmax_index":0,"argmax_id":"true","top2_margin":0.9465006571263075,"near_tie":false,"gold":"true","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"water_decision":{"type":"choice","choice":"replace","confidence":0.9911,"probabilities":{"repair":0.0034,"replace":0.9911,"postpone":0.0028,"no_decision":0.0027}},"fees_increase":{"type":"noul","noul":0.0101},"new_chair":{"type":"choice","choice":"somerfield","confidence":0.9872,"probabilities":{"ollerton":0.0039,"somerfield":0.9872,"reyes":0.0062,"kemal":0.0027}},"water_vote_consensus":{"type":"score","score":1.823,"confidence":0.8586,"legend":{"0":"deeply divided","1":"split","2":"clear majority","3":"unanimous"},"probabilities":{"0":0.0633,"1":0.0643,"2":0.8586,"3":0.0138}},"waiting_list_order":{"type":"choice","choice":"application_order","confidence":0.9925,"probabilities":{"application_order":0.9925,"lottery":0.0019,"local_priority":0.0056}},"bonfire_ban":{"type":"noul","noul":0.9733}},"usage":{"input_tokens":2494,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_j01","arm":"text","set":"fixture","source":"own_json","tokens":476,"n_image_tokens":0,"state_tokens":117,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,21529,21624,15501,3147,52897,2129,22906,3147,17,15,17,21,12,15,24,12,15,16,2129,56268,3147,17,15,17,21,12,15,23,12,15,17,2129,7718,64331,5856,763,16,16,17,13,20,1288,1163,3147,32,19,5392,11,220,20,14273,35303,5856,763,23,24,13,15,1288,1163,3147,752,261,49151,8934,27573,3946,3147,60810,12,17,15,24,18,16,2129,33530,763,15,13,15,1288,2738,3147,2428,2129,4873,763,17,15,16,13,20,1288,18633,3147,34,12896,273,20538,7988,28835,56658,29129,3147,17,15,17,21,12,15,24,12,16,19,8934,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,2552,198,9222,25,5576,198,669,55530,25,220,3710,369,279,23839,2552,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,2536,3106,3482,2129,1979,816,3147,43494,8934,198,32929,220,17,25,220,4754,4532,3147,1743,33530,694,524,3482,4016,2129,1979,816,3147,2428,7663,73140,8934,198,32929,220,18,25,220,4754,4532,3147,48427,279,4016,2321,321,52269,2129,1979,816,3147,1908,22906,8934,198,32929,220,19,25,220,4754,4532,3147,61462,303,2400,2129,1979,816,3147,33530,8934,198,4537,38945,198,198,27806,220,17,198,887,25,3349,198,9222,25,307,10644,198,669,55530,25,220,760,23839,2702,369,3294,220,20,15,15,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,10304,198,9222,25,5284,198,669,55530,25,220,4199,4970,1220,424,381,6945,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,2083,12601,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,559,2182,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,559,1936,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,29129,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"status","type":"choice","question_span":[177,183],"option_spans":[[194,206],[212,229],[235,251],[257,269]],"option_ids":["draft","open_not_due","overdue","paid"],"logits":[-4.615352630615234,-2.821337938308716,-0.3034135699272156,-4.601002216339111],"probs":[0.012104634195566177,0.07279179245233536,0.9028239846229553,0.012279592454433441],"argmax_index":2,"argmax_id":"overdue","top2_margin":0.83003219217062,"near_tie":false,"gold":"overdue","gold_correct":true},{"question_id":"large","type":"noul","question_span":[291,301],"option_spans":[[312,330],[336,354]],"option_ids":["true","false"],"logits":[-4.486606121063232,0.9491997957229614],"probs":[0.004338814876973629,0.995661199092865],"argmax_index":1,"argmax_id":"false","top2_margin":0.9913223842158914,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"priority","type":"score","question_span":[375,382],"option_spans":[[393,404],[410,421],[427,438],[444,454]],"option_ids":["0","1","2","3"],"logits":[-1.704256534576416,-1.1174001693725586,-1.7597167491912842,-3.007636547088623],"probs":[0.24900473654270172,0.44779172539711,0.23557083308696747,0.06763271242380142],"argmax_index":1,"argmax_id":"1","top2_margin":0.19878698885440826,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"status":{"type":"choice","choice":"overdue","confidence":0.9028,"probabilities":{"paid":0.0123,"overdue":0.9028,"open_not_due":0.0728,"draft":0.0121}},"large":{"type":"noul","noul":0.0043},"priority":{"type":"score","score":1.1218,"confidence":0.4478,"legend":{"0":"no rush","1":"this month","2":"this week","3":"today"},"probabilities":{"0":0.249,"1":0.4478,"2":0.2356,"3":0.0676}}},"usage":{"input_tokens":476,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_j02","arm":"text","set":"fixture","source":"own_json","tokens":475,"n_image_tokens":0,"state_tokens":68,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,10726,26331,3147,65308,471,874,7336,303,279,3618,4338,2129,3494,64331,28344,763,17,1288,38605,3147,44,2847,12,16,16,35303,28344,763,16,1288,38605,3147,2366,32,12,15,19,8934,27573,1310,816,3147,33,12,20,20,16,15,17,2129,13360,21624,2738,3147,18510,56658,25660,21624,4834,69429,763,3721,1288,4249,3147,13392,29958,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,628,30024,198,9222,25,307,10644,198,669,55530,25,220,760,1906,628,8026,1381,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,49198,198,9222,25,5576,198,669,55530,25,220,3710,369,21500,279,1906,11,413,3977,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,11268,2534,2129,1979,816,3147,4834,8934,198,32929,220,17,25,220,4754,4532,3147,23036,2129,1979,816,3147,6487,8934,198,32929,220,18,25,220,4754,4532,3147,19575,2129,1979,816,3147,13360,8934,198,32929,220,19,25,220,4754,4532,3147,18782,2129,1979,816,3147,13091,8934,198,4537,38945,198,198,27806,220,18,198,887,25,7932,198,9222,25,307,10644,198,669,55530,25,220,760,1906,369,264,7932,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,10304,198,9222,25,5284,198,669,55530,25,220,4199,1220,279,1906,381,43195,3920,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,7994,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,11553,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,82118,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"can_ship","type":"noul","question_span":[130,136],"option_spans":[[147,165],[171,189]],"option_ids":["true","false"],"logits":[-2.223015785217285,1.5737695693969727],"probs":[0.02195017971098423,0.9780498147010803],"argmax_index":1,"argmax_id":"false","top2_margin":0.9560996349900961,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"blocker","type":"choice","question_span":[210,219],"option_spans":[[230,242],[248,258],[264,274],[280,290]],"option_ids":["address","none","payment","stock"],"logits":[0.005842924118041992,-3.9764723777770996,-3.876716136932373,-4.556771278381348],"probs":[0.9526756405830383,0.01776018552482128,0.019623257219791412,0.009940925054252148],"argmax_index":0,"argmax_id":"address","top2_margin":0.9330523833632469,"near_tie":false,"gold":"address","gold_correct":true},{"question_id":"gift","type":"noul","question_span":[312,318],"option_spans":[[329,347],[353,371]],"option_ids":["true","false"],"logits":[-0.19658726453781128,-3.2964093685150146],"probs":[0.9568854570388794,0.04311459884047508],"argmax_index":0,"argmax_id":"true","top2_margin":0.9137708581984043,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"priority","type":"score","question_span":[392,400],"option_spans":[[411,421],[427,437],[443,453]],"option_ids":["0","1","2"],"logits":[-1.0228947401046753,-0.5102713704109192,-1.178887128829956],"probs":[0.2836693525314331,0.47363296151161194,0.24269776046276093],"argmax_index":1,"argmax_id":"1","top2_margin":0.18996360898017883,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"can_ship":{"type":"noul","noul":0.022},"blocker":{"type":"choice","choice":"address","confidence":0.9527,"probabilities":{"none":0.0178,"payment":0.0196,"address":0.9527,"stock":0.0099}},"gift":{"type":"noul","noul":0.9569},"priority":{"type":"score","score":0.959,"confidence":0.4736,"legend":{"0":"normal","1":"high","2":"urgent"},"probabilities":{"0":0.2837,"1":0.4736,"2":0.2427}}},"usage":{"input_tokens":475,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_j03","arm":"text","set":"fixture","source":"own_json","tokens":585,"n_image_tokens":0,"state_tokens":111,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,5920,3147,12865,7432,12,18,2129,39721,21624,89843,7033,763,20,15,1288,3757,6144,763,18,15,49471,851,793,64331,89843,763,23,16,1288,83,3147,15,21,25,15,15,2129,3757,763,16,19,13,17,35607,89843,763,22,15,1288,83,3147,15,24,25,15,15,2129,3757,763,16,24,13,23,35607,89843,763,20,17,1288,83,3147,16,17,25,15,15,2129,3757,763,18,16,13,20,35607,89843,763,19,22,1288,83,3147,16,20,25,15,15,2129,3757,763,18,18,13,16,92,27573,3715,3147,66,38915,8934,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,888,14335,198,9222,25,307,10644,198,669,55530,25,220,32,5122,32689,264,3798,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,1118,2200,24113,362,198,9222,25,5576,198,669,55530,25,220,4350,557,264,3798,1118,32689,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,15,21,25,15,15,2129,1979,816,3147,15,21,25,15,15,8934,198,32929,220,17,25,220,4754,4532,3147,15,24,25,15,15,2129,1979,816,3147,15,24,25,15,15,8934,198,32929,220,18,25,220,4754,4532,3147,16,17,25,15,15,2129,1979,816,3147,16,17,25,15,15,8934,198,32929,220,19,25,220,4754,4532,3147,16,20,25,15,15,2129,1979,816,3147,16,20,25,15,15,8934,198,32929,220,20,25,220,4754,4532,3147,25320,2129,1979,816,3147,35288,8934,198,4537,38945,198,198,27806,220,18,198,887,25,8876,198,9222,25,5576,198,669,55530,25,220,4199,369,279,9039,9722,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,37,16249,2129,1979,816,3147,13077,286,8934,198,32929,220,17,25,220,4754,4532,3147,30001,2129,1979,816,3147,25393,8934,198,32929,220,18,25,220,4754,4532,3147,49,3663,2129,1979,816,3147,56930,8934,198,4537,38945,198,198,27806,220,19,198,887,25,1852,198,9222,25,5284,198,669,55530,25,220,3710,1965,369,8053,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,2083,1852,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,12718,16070,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,2428,279,77442,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,95390,26054,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"over_limit","type":"noul","question_span":[173,179],"option_spans":[[190,208],[214,232]],"option_ids":["true","false"],"logits":[-0.583406925201416,-4.192796230316162],"probs":[0.973645031452179,0.026354987174272537],"argmax_index":0,"argmax_id":"true","top2_margin":0.9472900442779064,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"first_violation","type":"choice","question_span":[256,263],"option_spans":[[274,292],[298,316],[322,340],[346,364],[370,380]],"option_ids":["06:00","09:00","12:00","15:00","never"],"logits":[-4.4565839767456055,-4.560187339782715,0.1262110471725464,-3.8764750957489014,-4.604808330535889],"probs":[0.009771598502993584,0.008809906430542469,0.9555386900901794,0.0174543559551239,0.008425440639257431],"argmax_index":2,"argmax_id":"12:00","top2_margin":0.9380843341350555,"near_tie":false,"gold":"12:00","gold_correct":true},{"question_id":"trend","type":"choice","question_span":[401,407],"option_spans":[[418,430],[436,446],[452,463]],"option_ids":["falling","flat","rising"],"logits":[-4.382380485534668,-4.230378150939941,-0.014925628900527954],"probs":[0.012344629503786564,0.014371160417795181,0.9732841849327087],"argmax_index":2,"argmax_id":"rising","top2_margin":0.9589130245149136,"near_tie":false,"gold":"rising","gold_correct":true},{"question_id":"action","type":"score","question_span":[484,489],"option_spans":[[500,511],[517,528],[534,546],[552,563]],"option_ids":["0","1","2","3"],"logits":[-2.6713309288024902,-2.0546746253967285,0.7659738659858704,0.6540852785110474],"probs":[0.016190055757761,0.029995685443282127,0.5035595297813416,0.4502546787261963],"argmax_index":2,"argmax_id":"2","top2_margin":0.053304851055145264,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"over_limit":{"type":"noul","noul":0.9736},"first_violation":{"type":"choice","choice":"12:00","confidence":0.9555,"probabilities":{"06:00":0.0098,"09:00":0.0088,"12:00":0.9555,"15:00":0.0175,"never":0.0084}},"trend":{"type":"choice","choice":"rising","confidence":0.9733,"probabilities":{"rising":0.9733,"falling":0.0123,"flat":0.0144}},"action":{"type":"score","score":2.3879,"confidence":0.5036,"legend":{"0":"no action","1":"keep monitoring","2":"open the vents","3":"emergency cooling"},"probabilities":{"0":0.0162,"1":0.03,"2":0.5036,"3":0.4503}}},"usage":{"input_tokens":585,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_j04","arm":"text","set":"fixture","source":"own_json","tokens":569,"n_image_tokens":0,"state_tokens":121,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,2987,21624,54564,5343,64331,591,3147,39,3253,86791,2129,2246,3147,52742,35303,591,3147,36,24349,37770,2129,2246,3147,9709,1533,35303,591,3147,50,1002,710,3974,2129,2246,3147,69911,1340,35303,591,3147,92055,443,86,200637,2129,2246,3147,2083,9367,8934,27573,400,3147,17,15,17,21,12,16,15,12,16,17,51,16,21,25,18,15,2129,2447,3147,13601,220,19,33,2129,8202,3035,3147,39,3253,86791,2129,2670,19934,763,2827,1288,2388,3147,17,15,17,21,12,16,15,12,16,17,51,16,20,25,15,15,2129,41843,3147,20533,10,17,2129,2034,3147,83618,391,8842,29958,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,7836,198,9222,25,5576,198,669,55530,25,220,4199,1248,369,279,1484,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16,17,15,4276,2129,1979,816,3147,16,17,15,8934,198,32929,220,17,25,220,4754,4532,3147,18,15,4276,2129,1979,816,3147,18,15,8934,198,32929,220,18,25,220,4754,4532,3147,21,15,4276,2129,1979,816,3147,21,15,8934,198,32929,220,19,25,220,4754,4532,3147,24,15,4276,2129,1979,816,3147,24,15,8934,198,4537,38945,198,198,27806,220,17,198,887,25,43741,198,9222,25,307,10644,198,669,55530,25,220,760,1484,42151,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,11330,198,9222,25,5576,198,669,55530,25,220,4199,1599,37888,599,11330,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,13749,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,4537,38945,198,198,27806,220,19,198,887,25,26151,1650,3086,198,9222,25,5284,198,669,55530,25,220,4199,4222,369,424,421,2166,2342,1208,9138,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,53164,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,9735,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,13873,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"duration","type":"choice","question_span":[181,187],"option_spans":[[198,213],[219,232],[238,251],[257,270]],"option_ids":["120","30","60","90"],"logits":[-1.5175272226333618,-2.1720452308654785,0.5997335910797119,0.8237107992172241],"probs":[0.04945075511932373,0.025699183344841003,0.4108541011810303,0.5139959454536438],"argmax_index":3,"argmax_id":"90","top2_margin":0.10314184427261353,"near_tie":false,"gold":"90","gold_correct":true},{"question_id":"recurring","type":"noul","question_span":[292,296],"option_spans":[[307,325],[331,349]],"option_ids":["true","false"],"logits":[-3.0114715099334717,2.11220121383667],"probs":[0.005918874870985746,0.9940810799598694],"argmax_index":1,"argmax_id":"false","top2_margin":0.9881622050888836,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"accepted","type":"choice","question_span":[370,376],"option_spans":[[387,397],[403,413],[419,429],[435,445],[451,461]],"option_ids":["0","1","2","3","4"],"logits":[-3.186922073364258,-0.09093964099884033,-3.096848249435425,-4.2704386711120605,-4.513523101806641],"probs":[0.04031122475862503,0.8912389278411865,0.044110771268606186,0.01364145614206791,0.010697702877223492],"argmax_index":1,"argmax_id":"1","top2_margin":0.8471281565725803,"near_tie":false,"gold":"1","gold_correct":true},{"question_id":"attendance_risk","type":"score","question_span":[484,494],"option_spans":[[505,515],[521,531],[537,547]],"option_ids":["0","1","2"],"logits":[-2.4343957901000977,-0.061693161725997925,-0.7511076331138611],"probs":[0.05844682827591896,0.6269206404685974,0.3146325349807739],"argmax_index":1,"argmax_id":"1","top2_margin":0.3122881054878235,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"duration":{"type":"choice","choice":"90","confidence":0.514,"probabilities":{"30":0.0257,"60":0.4109,"90":0.514,"120":0.0495}},"recurring":{"type":"noul","noul":0.0059},"accepted":{"type":"choice","choice":"1","confidence":0.8912,"probabilities":{"0":0.0403,"1":0.8912,"2":0.0441,"3":0.0136,"4":0.0107}},"attendance_risk":{"type":"score","score":1.2562,"confidence":0.6269,"legend":{"0":"unlikely","1":"possible","2":"likely"},"probabilities":{"0":0.0584,"1":0.6269,"2":0.3146}}},"usage":{"input_tokens":569,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_j05","arm":"text","set":"fixture","source":"own_json","tokens":539,"n_image_tokens":0,"state_tokens":68,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,846,21624,10818,3147,2737,2129,6898,3147,17,15,16,24,12,15,19,12,16,16,2129,5320,1224,3147,42204,76041,2129,10837,35580,3705,1243,2506,66550,306,763,23,23,17,16,18,1288,4119,13288,3147,17,15,17,21,12,15,24,12,18,15,2129,10083,3147,10279,2129,19186,18029,763,3721,1288,20392,8899,763,1802,3307,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,3019,198,9222,25,5576,198,669,55530,25,220,22365,3019,369,279,1156,383,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,82688,2129,1979,816,3147,75267,8934,198,32929,220,17,25,220,4754,4532,3147,10621,2129,1979,816,3147,10279,8934,198,32929,220,18,25,220,4754,4532,3147,1289,2129,1979,816,3147,751,8934,198,32929,220,19,25,220,4754,4532,3147,14177,2129,1979,816,3147,8923,8934,198,4537,38945,198,198,27806,220,17,198,887,25,1330,18029,198,9222,25,307,10644,198,669,55530,25,220,11280,71577,16163,369,8699,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,2605,1650,3086,198,9222,25,5284,198,669,55530,25,220,4199,506,5048,369,279,2605,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1164,3238,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,9997,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,25252,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,11553,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,1164,1496,2129,1979,816,3147,19,8934,198,4537,38945,198,198,27806,220,19,198,887,25,1852,198,9222,25,5576,198,669,55530,25,220,3710,1220,279,4610,2015,635,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,18014,264,3435,7345,2129,1979,816,3147,8566,18410,8934,198,32929,220,17,25,220,4754,4532,3147,23036,2129,1979,816,3147,6487,8934,198,32929,220,18,25,220,4754,4532,3147,52782,279,1156,310,7056,1330,71577,16163,2129,1979,816,3147,39242,62,17,3510,8934,198,32929,220,19,25,220,4754,4532,3147,50,12393,279,2605,2129,1979,816,3147,95205,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"plan","type":"choice","question_span":[128,135],"option_spans":[[146,156],[162,172],[178,188],[194,204]],"option_ids":["enterprise","free","pro","team"],"logits":[-4.741209030151367,1.284430980682373,-3.85894775390625,-4.483442306518555],"probs":[0.0023888193536549807,0.9887477159500122,0.0057722413912415504,0.0030912202782928944],"argmax_index":1,"argmax_id":"free","top2_margin":0.9829754745587707,"near_tie":false,"gold":"free","gold_correct":true},{"question_id":"two_factor","type":"noul","question_span":[227,233],"option_spans":[[244,262],[268,286]],"option_ids":["true","false"],"logits":[-3.1545026302337646,2.205063581466675],"probs":[0.004680931568145752,0.9953190684318542],"argmax_index":1,"argmax_id":"false","top2_margin":0.9906381368637085,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"account_risk","type":"score","question_span":[309,316],"option_spans":[[327,338],[344,354],[360,370],[376,386],[392,403]],"option_ids":["0","1","2","3","4"],"logits":[-2.1122591495513916,-1.5637683868408203,0.46052271127700806,0.8221768736839294,-1.2256078720092773],"probs":[0.026975451037287712,0.04668476805090904,0.3534383475780487,0.5074322819709778,0.06546911597251892],"argmax_index":3,"argmax_id":"3","top2_margin":0.15399393439292908,"near_tie":false,"gold":null,"gold_correct":null},{"question_id":"action","type":"choice","question_span":[424,431],"option_spans":[[442,456],[462,472],[478,498],[504,517]],"option_ids":["force_reset","none","prompt_2fa","suspend"],"logits":[-1.0069653987884521,-2.8443024158477783,1.0957783460617065,-3.0893890857696533],"probs":[0.10556904971599579,0.016810912638902664,0.8644632697105408,0.013156836852431297],"argmax_index":2,"argmax_id":"prompt_2fa","top2_margin":0.758894219994545,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"plan":{"type":"choice","choice":"free","confidence":0.9887,"probabilities":{"free":0.9887,"pro":0.0058,"team":0.0031,"enterprise":0.0024}},"two_factor":{"type":"noul","noul":0.0047},"account_risk":{"type":"score","score":2.5377,"confidence":0.5074,"legend":{"0":"very low","1":"low","2":"medium","3":"high","4":"very high"},"probabilities":{"0":0.027,"1":0.0467,"2":0.3534,"3":0.5074,"4":0.0655}},"action":{"type":"choice","choice":"prompt_2fa","confidence":0.8645,"probabilities":{"none":0.0168,"prompt_2fa":0.8645,"force_reset":0.1056,"suspend":0.0132}}},"usage":{"input_tokens":539,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_j06","arm":"text","set":"fixture","source":"own_json","tokens":616,"n_image_tokens":0,"state_tokens":57,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,2877,21624,10083,14335,5500,74314,763,18,15,15,1288,35044,11847,74314,763,18,16,17,49471,815,21624,1787,3147,7454,879,30474,2129,1927,3147,30220,1599,7154,2129,1967,816,3147,2864,62,22,69,18,64,2129,43355,19239,33671,763,18,15,1288,2738,763,19,17,24,3307,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,1412,4637,198,9222,25,5576,198,669,55530,25,220,3710,2993,314,1412,369,411,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,18714,466,7634,2129,1979,816,3147,3137,8934,198,32929,220,17,25,220,4754,4532,3147,4630,524,1669,2129,1979,816,3147,1856,20822,8934,198,32929,220,18,25,220,4754,4532,3147,11231,31802,2129,1979,816,3147,7454,14335,8934,198,32929,220,19,25,220,4754,4532,3147,5302,14109,2129,1979,816,3147,3895,8934,198,32929,220,20,25,220,4754,4532,3147,7393,2129,1979,816,3147,13745,8934,198,32929,220,21,25,220,4754,4532,3147,7678,1622,2129,1979,816,3147,11932,8934,198,4537,38945,198,198,27806,220,17,198,887,25,21979,470,198,9222,25,307,10644,198,669,55530,25,220,760,1622,628,381,36482,1086,2843,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,3655,33671,198,9222,25,5576,198,669,55530,25,220,4199,1248,1220,279,2849,3655,1518,21979,286,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,2665,3655,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,18,15,6283,2129,1979,816,3147,18,15,8934,198,32929,220,18,25,220,4754,4532,3147,20,4276,2129,1979,816,3147,18,15,15,8934,198,32929,220,19,25,220,4754,4532,3147,16,6258,2129,1979,816,3147,18,21,15,15,8934,198,4537,38945,198,198,27806,220,19,198,887,25,2849,3625,67724,198,9222,25,307,10644,198,669,55530,25,220,760,2849,8613,279,1412,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,20,198,887,25,28943,198,9222,25,5284,198,669,55530,25,220,4199,14938,369,279,1412,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,9997,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,25252,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,11553,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"error_class","type":"choice","question_span":[118,125],"option_spans":[[136,148],[154,167],[173,185],[191,202],[208,218],[224,235]],"option_ids":["auth","not_found","rate_limit","server","timeout","validation"],"logits":[-4.908199310302734,-5.0785627365112305,0.6832016706466675,-4.576873302459717,-4.747196674346924,-4.798703670501709],"probs":[0.0036544690374284983,0.003082026494666934,0.9798033833503723,0.005089994985610247,0.004292861558496952,0.004077346529811621],"argmax_index":2,"argmax_id":"rate_limit","top2_margin":0.9747133883647621,"near_tie":false,"gold":"rate_limit","gold_correct":true},{"question_id":"retryable","type":"noul","question_span":[258,266],"option_spans":[[277,295],[301,319]],"option_ids":["true","false"],"logits":[-1.6817891597747803,-4.37984561920166],"probs":[0.9369118809700012,0.06308813393115997],"argmax_index":0,"argmax_id":"true","top2_margin":0.8738237470388412,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"wait_seconds","type":"choice","question_span":[341,351],"option_spans":[[362,373],[379,392],[398,411],[417,431]],"option_ids":["0","30","300","3600"],"logits":[-4.31928014755249,0.27284568548202515,-4.3908610343933105,-4.50761604309082],"probs":[0.009855780750513077,0.972805380821228,0.0091749532148242,0.00816390197724104],"argmax_index":1,"argmax_id":"30","top2_margin":0.962949600070715,"near_tie":false,"gold":"30","gold_correct":true},{"question_id":"client_at_fault","type":"noul","question_span":[455,461],"option_spans":[[472,490],[496,514]],"option_ids":["true","false"],"logits":[-2.257399559020996,-3.862826347351074],"probs":[0.8327754735946655,0.16722451150417328],"argmax_index":0,"argmax_id":"true","top2_margin":0.6655509620904922,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"severity","type":"score","question_span":[535,541],"option_spans":[[552,562],[568,578],[584,594]],"option_ids":["0","1","2"],"logits":[-3.5363292694091797,-1.4051491022109985,-2.1698408126831055],"probs":[0.07492678612470627,0.6312435269355774,0.2938295900821686],"argmax_index":1,"argmax_id":"1","top2_margin":0.3374139368534088,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"error_class":{"type":"choice","choice":"rate_limit","confidence":0.9798,"probabilities":{"auth":0.0037,"rate_limit":0.9798,"not_found":0.0031,"server":0.0051,"validation":0.0041,"timeout":0.0043}},"retryable":{"type":"noul","noul":0.9369},"wait_seconds":{"type":"choice","choice":"30","confidence":0.9728,"probabilities":{"0":0.0099,"30":0.9728,"300":0.0092,"3600":0.0082}},"client_at_fault":{"type":"noul","noul":0.8328},"severity":{"type":"score","score":1.2189,"confidence":0.6312,"legend":{"0":"low","1":"medium","2":"high"},"probabilities":{"0":0.0749,"1":0.6312,"2":0.2938}}},"usage":{"input_tokens":616,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_j07","arm":"text","set":"fixture","source":"own_json","tokens":595,"n_image_tokens":0,"state_tokens":93,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,3494,64331,263,22470,763,16,17,15,15,1288,265,1310,5895,763,20,15,15,1288,38605,3147,33,34149,5084,21,35303,263,22470,763,18,19,15,1288,265,1310,5895,763,20,15,15,1288,38605,3147,45,1333,5084,21,35303,263,22470,763,15,1288,265,1310,5895,763,18,15,15,1288,38605,3147,54,1846,2916,12,21,35303,263,22470,763,22,20,1288,265,1310,5895,763,20,15,1288,38605,3147,12717,56205,7869,8934,27573,44029,3147,59810,8934,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,680,3456,29543,198,9222,25,5576,198,669,55530,25,220,22365,1455,369,680,314,5416,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,33,34149,5084,21,2129,1979,816,3147,33,34149,5084,21,8934,198,32929,220,17,25,220,4754,4532,3147,12717,56205,7869,2129,1979,816,3147,12717,56205,7869,8934,198,32929,220,18,25,220,4754,4532,3147,45,1333,5084,21,2129,1979,816,3147,45,1333,5084,21,8934,198,32929,220,19,25,220,4754,4532,3147,54,1846,2916,12,21,2129,1979,816,3147,54,1846,2916,12,21,8934,198,32929,220,20,25,220,4754,4532,3147,2665,1455,2129,1979,816,3147,6487,8934,198,4537,38945,198,198,27806,220,17,198,887,25,80313,3079,198,9222,25,5576,198,669,55530,25,220,4199,1599,3470,513,506,466,3559,836,80313,1406,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,15,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,16,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,17,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,18,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,19,2129,1979,816,3147,19,8934,198,4537,38945,198,198,27806,220,18,198,887,25,52011,198,9222,25,5284,198,669,55530,25,220,4199,32556,369,2645,67718,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1856,32556,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,63826,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,82118,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,40167,2129,1979,816,3147,18,8934,198,4537,38945,198,198,27806,220,19,198,887,25,866,79894,198,9222,25,307,10644,198,669,55530,25,220,1597,3140,799,1455,369,3559,1141,80313,1406,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"out_of_stock","type":"choice","question_span":[155,162],"option_spans":[[173,189],[195,209],[215,231],[237,255],[261,272]],"option_ids":["BOLT-M6","BRACKET-L","NUT-M6","WASHER-6","none"],"logits":[-4.897684097290039,-4.779381275177002,-4.539413928985596,0.3838192820549011,-4.834470748901367],"probs":[0.00496803130954504,0.005591941066086292,0.00710851838812232,0.9770392775535583,0.005292215384542942],"argmax_index":3,"argmax_id":"WASHER-6","top2_margin":0.969930759165436,"near_tie":false,"gold":"WASHER-6","gold_correct":true},{"question_id":"reorder_count","type":"choice","question_span":[294,305],"option_spans":[[316,326],[332,342],[348,358],[364,374],[380,390]],"option_ids":["0","1","2","3","4"],"logits":[-4.41380500793457,-3.7029359340667725,-1.4405324459075928,-2.281491756439209,-4.1372857093811035],"probs":[0.030917195603251457,0.06293998658657074,0.6046109199523926,0.2607666254043579,0.04076532647013664],"argmax_index":2,"argmax_id":"2","top2_margin":0.34384429454803467,"near_tie":false,"gold":"2","gold_correct":true},{"question_id":"urgency","type":"score","question_span":[411,417],"option_spans":[[428,439],[445,455],[461,471],[477,487]],"option_ids":["0","1","2","3"],"logits":[-2.793246030807495,-2.122694253921509,-0.5249106884002686,-1.9165003299713135],"probs":[0.06657043099403381,0.13016623258590698,0.6432901620864868,0.15997318923473358],"argmax_index":2,"argmax_id":"2","top2_margin":0.48331697285175323,"near_tie":false,"gold":null,"gold_correct":null},{"question_id":"any_below","type":"noul","question_span":[510,520],"option_spans":[[531,549],[555,573]],"option_ids":["true","false"],"logits":[-1.130994200706482,-4.188776969909668],"probs":[0.9551172852516174,0.04488265886902809],"argmax_index":0,"argmax_id":"true","top2_margin":0.9102346263825893,"near_tie":false,"gold":"true","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"out_of_stock":{"type":"choice","choice":"WASHER-6","confidence":0.977,"probabilities":{"BOLT-M6":0.005,"NUT-M6":0.0071,"WASHER-6":0.977,"BRACKET-L":0.0056,"none":0.0053}},"reorder_count":{"type":"choice","choice":"2","confidence":0.6046,"probabilities":{"0":0.0309,"1":0.0629,"2":0.6046,"3":0.2608,"4":0.0408}},"urgency":{"type":"score","score":1.8967,"confidence":0.6433,"legend":{"0":"not urgent","1":"soon","2":"urgent","3":"critical"},"probabilities":{"0":0.0666,"1":0.1302,"2":0.6433,"3":0.16}},"any_below":{"type":"noul","noul":0.9551}},"usage":{"input_tokens":595,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_j08","arm":"text","set":"fixture","source":"own_json","tokens":473,"n_image_tokens":0,"state_tokens":76,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,3264,12211,763,3721,1288,13102,64331,767,3147,9699,50457,2129,902,763,17,1288,787,763,21,35607,767,3147,813,8016,2129,902,3147,8090,2129,787,3147,2646,35303,767,3147,1932,15248,2291,2129,902,763,20,15,1288,787,763,20,15,35607,767,3147,12388,13725,4468,37927,38623,2129,902,763,1802,1288,787,763,3721,92,27573,22570,3147,21924,2129,7686,3147,37355,23157,8934,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,38841,198,9222,25,307,10644,198,669,55530,25,220,760,2222,369,38841,310,10204,1381,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,5048,13051,15476,198,9222,25,5576,198,669,55530,25,220,22365,2222,22651,279,1379,5048,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,21624,25228,3147,11868,12850,2129,767,3147,1932,15248,2291,56658,1979,816,3147,1932,15248,2291,8934,198,32929,220,17,25,220,4754,4532,21624,25228,3147,64,491,1970,1752,364,61893,2129,767,3147,12388,13725,4468,37927,38623,56658,1979,816,3147,12388,10614,8934,198,32929,220,18,25,220,4754,4532,21624,25228,3147,813,7867,321,2695,2129,767,3147,813,8016,56658,1979,816,3147,813,8016,8934,198,32929,220,19,25,220,4754,4532,21624,25228,3147,82,18913,8390,2129,767,3147,9699,50457,56658,1979,816,3147,9699,50457,8934,198,4537,38945,198,198,27806,220,18,198,887,25,3286,198,9222,25,5284,198,669,55530,25,220,3710,3286,1503,279,2222,1144,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,3771,12,47566,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,16049,3286,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,11422,2382,3286,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,4425,2980,264,2222,3136,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"risky","type":"noul","question_span":[137,145],"option_spans":[[156,174],[180,198]],"option_ids":["true","false"],"logits":[-0.2945409119129181,-1.3260526657104492],"probs":[0.7372088432312012,0.26279112696647644],"argmax_index":0,"argmax_id":"true","top2_margin":0.47441771626472473,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"riskiest_change","type":"choice","question_span":[221,228],"option_spans":[[239,260],[266,292],[298,319],[325,345]],"option_ids":["db_pool_size","feature_flag","log_level","replicas"],"logits":[-4.0184173583984375,-0.15917468070983887,-0.9919117093086243,0.16540458798408508],"probs":[0.007425523828715086,0.3521883487701416,0.1531517207622528,0.4872344434261322],"argmax_index":3,"argmax_id":"replicas","top2_margin":0.1350460946559906,"near_tie":false,"gold":"replicas","gold_correct":true},{"question_id":"review","type":"score","question_span":[366,373],"option_spans":[[384,396],[402,413],[419,431],[437,451]],"option_ids":["0","1","2","3"],"logits":[-1.4695255756378174,0.22877347469329834,-0.3386768698692322,-0.5914581418037415],"probs":[0.08354794979095459,0.4565598666667938,0.2588553726673126,0.20103685557842255],"argmax_index":1,"argmax_id":"1","top2_margin":0.1977044939994812,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"risky":{"type":"noul","noul":0.7372},"riskiest_change":{"type":"choice","choice":"replicas","confidence":0.4872,"probabilities":{"replicas":0.4872,"log_level":0.1532,"db_pool_size":0.0074,"feature_flag":0.3522}},"review":{"type":"score","score":1.5774,"confidence":0.4566,"legend":{"0":"auto-approve","1":"peer review","2":"senior review","3":"block until a change window"},"probabilities":{"0":0.0835,"1":0.4566,"2":0.2589,"3":0.201}}},"usage":{"input_tokens":473,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_j09","arm":"text","set":"fixture","source":"own_json","tokens":573,"n_image_tokens":0,"state_tokens":129,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,12034,64331,2447,3147,7591,349,357,2129,2738,3147,90278,7980,2129,1619,3147,17,15,17,21,12,15,24,12,17,23,220,15,23,25,16,15,35303,2447,3147,18732,220,18,2129,2738,3147,258,7714,275,2129,1619,3147,17,15,17,21,12,15,24,12,17,24,220,16,24,25,19,15,35303,14165,3147,4834,2306,58256,470,2129,2447,3147,18732,220,18,2129,2738,3147,7557,2129,1619,3147,17,15,17,21,12,15,24,12,18,15,220,15,22,25,16,20,8934,27573,23693,3920,48859,3147,17,15,17,21,12,15,24,12,18,15,2129,43075,3147,2301,42,12,19,19,16,15,12,17,17,8934,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,2552,198,9222,25,5576,198,669,55530,25,220,3710,369,279,1428,2552,314,279,28350,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16044,41636,2129,1979,816,3147,9302,41636,8934,198,32929,220,17,25,220,4754,4532,3147,39,758,1521,314,264,3377,2129,1979,816,3147,7557,8934,198,32929,220,18,25,220,4754,4532,3147,623,23312,2129,1979,816,3147,258,7714,275,8934,198,32929,220,19,25,220,4754,4532,3147,2575,364,9403,2129,1979,816,3147,403,5305,48859,8934,198,32929,220,20,25,220,4754,4532,3147,47,17952,685,2129,1979,816,3147,90278,7980,8934,198,4537,38945,198,198,27806,220,17,198,887,25,3200,198,9222,25,307,10644,198,669,55530,25,220,760,28350,668,3016,279,18378,9403,2321,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,5813,7681,198,9222,25,307,10644,198,669,55530,25,220,760,5813,3749,310,635,2414,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,67909,198,9222,25,5284,198,669,55530,25,220,4199,2944,1220,411,381,77819,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,2083,67909,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,8923,2895,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,63758,79265,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"status","type":"choice","question_span":[189,198],"option_spans":[[209,221],[227,242],[248,261],[267,281],[287,300]],"option_ids":["delivered","exception","in_transit","out_for_delivery","picked_up"],"logits":[-4.918759822845459,-0.7272152900695801,-4.419200897216797,-4.69509220123291,-4.75492000579834],"probs":[0.014044644311070442,0.9286999106407166,0.023145491257309914,0.017565006390213966,0.016544951125979424],"argmax_index":1,"argmax_id":"exception","top2_margin":0.9055544193834066,"near_tie":false,"gold":"exception","gold_correct":true},{"question_id":"late","type":"noul","question_span":[322,331],"option_spans":[[342,360],[366,384]],"option_ids":["true","false"],"logits":[-2.1768767833709717,-2.3260817527770996],"probs":[0.5372322201728821,0.4627678394317627],"argmax_index":0,"argmax_id":"true","top2_margin":0.07446438074111938,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"customer_action","type":"noul","question_span":[407,414],"option_spans":[[425,443],[449,467]],"option_ids":["true","false"],"logits":[-1.7936185598373413,-3.865729331970215],"probs":[0.8881627917289734,0.11183721572160721],"argmax_index":0,"argmax_id":"true","top2_margin":0.7763255760073662,"near_tie":false,"gold":null,"gold_correct":null},{"question_id":"escalation","type":"score","question_span":[488,495],"option_spans":[[506,517],[523,534],[540,551]],"option_ids":["0","1","2"],"logits":[-2.9449658393859863,-1.4727774858474731,-2.228926181793213],"probs":[0.1350424885749817,0.5886182188987732,0.27633920311927795],"argmax_index":1,"argmax_id":"1","top2_margin":0.31227901577949524,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"status":{"type":"choice","choice":"exception","confidence":0.9287,"probabilities":{"picked_up":0.0165,"in_transit":0.0231,"exception":0.9287,"out_for_delivery":0.0176,"delivered":0.014}},"late":{"type":"noul","noul":0.5372},"customer_action":{"type":"noul","noul":0.8882},"escalation":{"type":"score","score":1.1413,"confidence":0.5886,"legend":{"0":"no escalation","1":"team lead","2":"carrier liaison"},"probabilities":{"0":0.135,"1":0.5886,"2":0.2763}}},"usage":{"input_tokens":573,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_j10","arm":"text","set":"fixture","source":"own_json","tokens":413,"n_image_tokens":0,"state_tokens":56,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,5989,3147,20740,9989,11,694,567,31178,1472,279,1483,1330,13753,46292,68589,64331,80,3147,1733,2129,11985,763,20,35607,80,3147,1260,2129,11985,763,17,35607,80,3147,74917,2129,11985,763,19,35607,80,3147,6917,2129,11985,763,18,92,27573,45872,3147,2134,27686,8411,8934,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,77976,198,9222,25,5576,198,669,55530,25,220,22365,12523,16060,11447,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,2674,2129,1979,816,3147,1733,8934,198,32929,220,17,25,220,4754,4532,3147,58474,2129,1979,816,3147,74917,8934,198,32929,220,18,25,220,4754,4532,3147,47,563,2129,1979,816,3147,1260,8934,198,32929,220,19,25,220,4754,4532,3147,87037,2129,1979,816,3147,6917,8934,198,4537,38945,198,198,27806,220,17,198,887,25,7830,198,9222,25,5284,198,669,55530,25,220,4199,18933,369,279,64628,7830,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1164,13376,44723,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,67,1003,44723,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,57555,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,82,44723,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,1164,18933,2129,1979,816,3147,19,8934,198,4537,38945,198,198,27806,220,18,198,887,25,18933,198,9222,25,307,10644,198,669,55530,25,220,760,64628,369,41510,18933,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"weakest","type":"choice","question_span":[116,121],"option_spans":[[132,142],[148,158],[164,175],[181,191]],"option_ids":["content","materials","pace","venue"],"logits":[-2.810525894165039,-3.247713804244995,0.34381166100502014,-0.9233772158622742],"probs":[0.03156178817152977,0.020384183153510094,0.7397297024726868,0.20832428336143494],"argmax_index":2,"argmax_id":"pace","top2_margin":0.5314054191112518,"near_tie":false,"gold":"pace","gold_correct":true},{"question_id":"overall","type":"score","question_span":[212,219],"option_spans":[[230,242],[248,260],[266,276],[282,293],[299,310]],"option_ids":["0","1","2","3","4"],"logits":[-4.147209644317627,-2.73783016204834,-1.3682769536972046,0.14668205380439758,-1.945178508758545],"probs":[0.009662805125117302,0.03955387696623802,0.15558885037899017,0.7078101634979248,0.08738427609205246],"argmax_index":3,"argmax_id":"3","top2_margin":0.5522213131189346,"near_tie":false,"gold":null,"gold_correct":null},{"question_id":"satisfied","type":"noul","question_span":[332,338],"option_spans":[[349,367],[373,391]],"option_ids":["true","false"],"logits":[-0.07124483585357666,-1.817452311515808],"probs":[0.851473867893219,0.14852620661258698],"argmax_index":0,"argmax_id":"true","top2_margin":0.702947661280632,"near_tie":false,"gold":"true","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"weakest":{"type":"choice","choice":"pace","confidence":0.7397,"probabilities":{"content":0.0316,"pace":0.7397,"materials":0.0204,"venue":0.2083}},"overall":{"type":"score","score":2.8237,"confidence":0.7078,"legend":{"0":"very dissatisfied","1":"dissatisfied","2":"neutral","3":"satisfied","4":"very satisfied"},"probabilities":{"0":0.0097,"1":0.0396,"2":0.1556,"3":0.7078,"4":0.0874}},"satisfied":{"type":"noul","noul":0.8515}},"usage":{"input_tokens":413,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_j11","arm":"text","set":"fixture","source":"own_json","tokens":463,"n_image_tokens":0,"state_tokens":85,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,58,4754,36151,763,19,1288,10279,35580,16,15,25,15,15,2129,16,19,25,15,15,66550,4826,269,763,1802,1288,2871,3147,32,35303,36151,763,16,17,1288,10279,35580,16,19,25,15,15,66550,4826,269,763,1802,1288,2871,3147,33,35303,36151,763,23,1288,10279,35580,16,15,25,15,15,2129,16,16,25,15,15,2129,16,19,25,15,15,66550,4826,269,763,3721,1288,2871,3147,34,8934,60,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,2956,198,9222,25,5576,198,669,55530,25,220,22365,2956,17759,7810,1208,440,264,65690,506,220,16,19,25,15,15,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,13601,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,13601,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,13601,351,2129,1979,816,3147,34,8934,198,32929,220,19,25,220,4754,4532,3147,2665,2956,17759,2129,1979,816,3147,6487,8934,198,4537,38945,198,198,27806,220,17,198,887,25,9167,62,16,15,198,9222,25,5576,198,669,55530,25,220,22365,11683,513,1845,506,220,16,15,25,15,15,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,56300,357,321,351,2129,1979,816,3147,32,892,8934,198,32929,220,17,25,220,4754,4532,3147,13601,417,1132,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,2324,11683,2129,1979,816,3147,528,8934,198,32929,220,19,25,220,4754,4532,3147,2665,2956,2129,1979,816,3147,6487,8934,198,4537,38945,198,198,27806,220,18,198,887,25,40803,198,9222,25,307,10644,198,669,55530,25,220,32,2956,440,264,65690,364,7810,1208,369,1845,506,220,16,15,25,15,15,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"room","type":"choice","question_span":[145,161],"option_spans":[[172,183],[189,200],[206,217],[223,235]],"option_ids":["A","B","C","none"],"logits":[-2.909062385559082,1.0614107847213745,-3.4641852378845215,-1.789039969444275],"probs":[0.017346492037177086,0.9195308685302734,0.00995691679418087,0.05316566675901413],"argmax_index":1,"argmax_id":"B","top2_margin":0.8663652017712593,"near_tie":false,"gold":"B","gold_correct":true},{"question_id":"slot_10","type":"choice","question_span":[259,271],"option_spans":[[282,296],[302,314],[320,331],[337,348]],"option_ids":["A_C","B","all","none"],"logits":[1.5430512428283691,-4.636912822723389,-2.5394644737243652,-3.380810022354126],"probs":[0.9744627475738525,0.0020176272373646498,0.01643430069088936,0.007085318211466074],"argmax_index":0,"argmax_id":"A_C","top2_margin":0.9580284468829632,"near_tie":false,"gold":"A_C","gold_correct":true},{"question_id":"feasible","type":"noul","question_span":[370,388],"option_spans":[[399,417],[423,441]],"option_ids":["true","false"],"logits":[-2.237995147705078,0.946277379989624],"probs":[0.03976188600063324,0.9602381587028503],"argmax_index":1,"argmax_id":"false","top2_margin":0.9204762727022171,"near_tie":false,"gold":"false","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"room":{"type":"choice","choice":"B","confidence":0.9195,"probabilities":{"A":0.0173,"B":0.9195,"C":0.01,"none":0.0532}},"slot_10":{"type":"choice","choice":"A_C","confidence":0.9745,"probabilities":{"A_C":0.9745,"B":0.002,"all":0.0164,"none":0.0071}},"feasible":{"type":"noul","noul":0.0398}},"usage":{"input_tokens":463,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"own_j12","arm":"text","set":"fixture","source":"own_json","tokens":562,"n_image_tokens":0,"state_tokens":103,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,57746,64331,1163,3147,895,56868,189628,2129,6352,2135,322,763,18,13,23,35607,1163,3147,4920,23505,209239,241308,2129,6352,2135,322,763,21,13,20,35607,1163,3147,6205,57622,273,2129,6352,2135,322,763,24,13,17,92,27573,37031,21624,1787,3147,55359,2705,45,16,15,2129,23650,763,16,15,1288,1826,42880,3147,17,15,17,21,12,15,24,12,18,15,56658,10726,21624,8656,3147,216604,35702,2129,591,3147,41,167864,17151,7746,56658,29129,3147,17,15,17,21,12,16,15,12,15,18,8934,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,23076,8078,198,9222,25,307,10644,198,669,55530,25,220,760,23076,628,1990,381,1429,3242,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,2702,43832,198,9222,25,5576,198,669,55530,25,220,3710,369,279,13621,2702,1518,866,10766,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16,15,310,220,17,15,35858,2129,1979,816,3147,16,15,2270,62,17,15,8934,198,32929,220,17,25,220,4754,4532,3147,17,15,310,220,18,15,35858,2129,1979,816,3147,17,15,2270,62,18,15,8934,198,32929,220,18,25,220,4754,4532,3147,1853,220,18,15,35858,2129,1979,816,3147,1908,62,18,15,8934,198,32929,220,19,25,220,4754,4532,3147,15771,220,16,15,35858,2129,1979,816,3147,7744,62,16,15,8934,198,4537,38945,198,198,27806,220,18,198,887,25,41334,11789,198,9222,25,5576,198,669,55530,25,220,4199,1599,3470,513,68990,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,6487,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,4537,38945,198,198,27806,220,19,198,887,25,13621,2291,198,9222,25,5284,198,669,55530,25,220,4199,3349,369,279,13621,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,8732,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,25252,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,16275,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"coupon_valid","type":"noul","question_span":[165,173],"option_spans":[[184,202],[208,226]],"option_ids":["true","false"],"logits":[-2.288350820541382,1.6444902420043945],"probs":[0.019211627542972565,0.9807883501052856],"argmax_index":1,"argmax_id":"false","top2_margin":0.9615767225623131,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"total_band","type":"choice","question_span":[248,257],"option_spans":[[268,289],[295,316],[322,339],[345,362]],"option_ids":["10_to_20","20_to_30","over_30","under_10"],"logits":[-0.6268253326416016,-2.2298240661621094,-3.797109365463257,-2.77079176902771],"probs":[0.7350385785102844,0.14795738458633423,0.030865497887134552,0.08613850176334381],"argmax_index":0,"argmax_id":"10_to_20","top2_margin":0.5870811939239502,"near_tie":false,"gold":"10_to_20","gold_correct":true},{"question_id":"dessert_items","type":"choice","question_span":[384,390],"option_spans":[[401,411],[417,427],[433,443],[449,459]],"option_ids":["0","1","2","3"],"logits":[-2.8830647468566895,-0.4769333004951477,-0.8407855033874512,-2.935763359069824],"probs":[0.04819788783788681,0.5345614552497864,0.3715169429779053,0.04572369158267975],"argmax_index":1,"argmax_id":"1","top2_margin":0.1630445122718811,"near_tie":false,"gold":"1","gold_correct":true},{"question_id":"basket_size","type":"score","question_span":[481,487],"option_spans":[[498,508],[514,524],[530,540]],"option_ids":["0","1","2"],"logits":[-0.5066636800765991,1.0791609287261963,-2.7026002407073975],"probs":[0.16681760549545288,0.8146231770515442,0.018559178337454796],"argmax_index":1,"argmax_id":"1","top2_margin":0.6478055715560913,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"coupon_valid":{"type":"noul","noul":0.0192},"total_band":{"type":"choice","choice":"10_to_20","confidence":0.735,"probabilities":{"under_10":0.0861,"10_to_20":0.735,"20_to_30":0.148,"over_30":0.0309}},"dessert_items":{"type":"choice","choice":"1","confidence":0.5346,"probabilities":{"0":0.0482,"1":0.5346,"2":0.3715,"3":0.0457}},"basket_size":{"type":"score","score":0.8517,"confidence":0.8146,"legend":{"0":"small","1":"medium","2":"large"},"probabilities":{"0":0.1668,"1":0.8146,"2":0.0186}}},"usage":{"input_tokens":562,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_01","arm":"g256","set":"fixture","source":"own_image","tokens":430,"n_image_tokens":64,"state_tokens":8,"token_offset":36,"image_files":["shapes_224.png"],"image_size_in":[224,224],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"9039285a1afdc13d757d2b9b19c352103f9b2b600dd868f1277b250a2aae8811","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,2427,279,20182,303,279,12037,6588,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,24557,198,9222,25,5576,198,669,55530,25,220,4199,1599,24557,513,1017,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,20,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,21,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,17,198,887,25,777,648,71735,198,9222,25,307,10644,198,669,55530,25,220,3733,513,777,24557,1056,30308,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,52010,198,9222,25,5284,198,669,55530,25,220,4199,36947,369,279,6588,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,94834,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,50061,9497,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"circles","type":"choice","question_span":[135,141],"option_spans":[[152,162],[168,178],[184,194],[200,210],[216,226],[232,242]],"option_ids":["1","2","3","4","5","6"],"logits":[-4.81542444229126,-4.424631595611572,-0.18891417980194092,-4.6017680168151855,-4.395053386688232,-4.439544200897217],"probs":[0.009186779148876667,0.013579458929598331,0.9384931325912476,0.011375043541193008,0.01398711372166872,0.013378456234931946],"argmax_index":2,"argmax_id":"3","top2_margin":0.9245060188695788,"near_tie":false,"gold":"3","gold_correct":true},{"question_id":"more_circles","type":"noul","question_span":[266,273],"option_spans":[[284,302],[308,326]],"option_ids":["true","false"],"logits":[-0.42368438839912415,-1.846826195716858],"probs":[0.8058304786682129,0.1941695213317871],"argmax_index":0,"argmax_id":"true","top2_margin":0.6116609573364258,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"clutter","type":"score","question_span":[347,353],"option_spans":[[364,374],[380,391],[397,408]],"option_ids":["0","1","2"],"logits":[0.6887646913528442,0.6843016147613525,-2.4599757194519043],"probs":[0.4905681312084198,0.48838356137275696,0.02104838564991951],"argmax_index":0,"argmax_id":"0","top2_margin":0.002184569835662842,"near_tie":true,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"circles":{"type":"choice","choice":"3","confidence":0.9385,"probabilities":{"1":0.0092,"2":0.0136,"3":0.9385,"4":0.0114,"5":0.014,"6":0.0134}},"more_circles":{"type":"noul","noul":0.8058},"clutter":{"type":"score","score":0.5305,"confidence":0.4906,"legend":{"0":"sparse","1":"moderate","2":"crowded"},"probabilities":{"0":0.4906,"1":0.4884,"2":0.021}}},"usage":{"input_tokens":430,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_01","arm":"g448","set":"fixture","source":"own_image","tokens":562,"n_image_tokens":196,"state_tokens":8,"token_offset":36,"image_files":["shapes_224.png"],"image_size_in":[224,224],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"9ccc6687b34e522cc5a6ab5c4000b2c147495588644201834414a94f37496677","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,2427,279,20182,303,279,12037,6588,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,24557,198,9222,25,5576,198,669,55530,25,220,4199,1599,24557,513,1017,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,20,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,21,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,17,198,887,25,777,648,71735,198,9222,25,307,10644,198,669,55530,25,220,3733,513,777,24557,1056,30308,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,52010,198,9222,25,5284,198,669,55530,25,220,4199,36947,369,279,6588,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,94834,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,50061,9497,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"circles","type":"choice","question_span":[267,273],"option_spans":[[284,294],[300,310],[316,326],[332,342],[348,358],[364,374]],"option_ids":["1","2","3","4","5","6"],"logits":[-4.849889755249023,-4.484442234039307,-0.2675701975822449,-4.569883823394775,-4.411252498626709,-4.467963695526123],"probs":[0.00956746842712164,0.01378824096173048,0.935132622718811,0.012659077532589436,0.014835246838629246,0.014017333276569843],"argmax_index":2,"argmax_id":"3","top2_margin":0.9202973758801818,"near_tie":false,"gold":"3","gold_correct":true},{"question_id":"more_circles","type":"noul","question_span":[398,405],"option_spans":[[416,434],[440,458]],"option_ids":["true","false"],"logits":[-0.5503659248352051,-1.910875678062439],"probs":[0.7958424687385559,0.20415745675563812],"argmax_index":0,"argmax_id":"true","top2_margin":0.5916850119829178,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"clutter","type":"score","question_span":[479,485],"option_spans":[[496,506],[512,523],[529,540]],"option_ids":["0","1","2"],"logits":[0.2282777726650238,0.8444536924362183,-2.1631546020507812],"probs":[0.33975109457969666,0.6291621923446655,0.031086716800928116],"argmax_index":1,"argmax_id":"1","top2_margin":0.28941109776496887,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"circles":{"type":"choice","choice":"3","confidence":0.9351,"probabilities":{"1":0.0096,"2":0.0138,"3":0.9351,"4":0.0127,"5":0.0148,"6":0.014}},"more_circles":{"type":"noul","noul":0.7958},"clutter":{"type":"score","score":0.6913,"confidence":0.6292,"legend":{"0":"sparse","1":"moderate","2":"crowded"},"probabilities":{"0":0.3398,"1":0.6292,"2":0.0311}}},"usage":{"input_tokens":562,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_01","arm":"native","set":"fixture","source":"own_image","tokens":430,"n_image_tokens":64,"state_tokens":8,"token_offset":36,"image_files":["shapes_224.png"],"image_size_in":[224,224],"image_size_fed":[224,224],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"9039285a1afdc13d757d2b9b19c352103f9b2b600dd868f1277b250a2aae8811","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,2427,279,20182,303,279,12037,6588,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,24557,198,9222,25,5576,198,669,55530,25,220,4199,1599,24557,513,1017,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,20,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,21,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,17,198,887,25,777,648,71735,198,9222,25,307,10644,198,669,55530,25,220,3733,513,777,24557,1056,30308,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,52010,198,9222,25,5284,198,669,55530,25,220,4199,36947,369,279,6588,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,94834,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,50061,9497,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"circles","type":"choice","question_span":[135,141],"option_spans":[[152,162],[168,178],[184,194],[200,210],[216,226],[232,242]],"option_ids":["1","2","3","4","5","6"],"logits":[-4.81542444229126,-4.424631595611572,-0.18891417980194092,-4.6017680168151855,-4.395053386688232,-4.439544200897217],"probs":[0.009186779148876667,0.013579458929598331,0.9384931325912476,0.011375043541193008,0.01398711372166872,0.013378456234931946],"argmax_index":2,"argmax_id":"3","top2_margin":0.9245060188695788,"near_tie":false,"gold":"3","gold_correct":true},{"question_id":"more_circles","type":"noul","question_span":[266,273],"option_spans":[[284,302],[308,326]],"option_ids":["true","false"],"logits":[-0.42368438839912415,-1.846826195716858],"probs":[0.8058304786682129,0.1941695213317871],"argmax_index":0,"argmax_id":"true","top2_margin":0.6116609573364258,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"clutter","type":"score","question_span":[347,353],"option_spans":[[364,374],[380,391],[397,408]],"option_ids":["0","1","2"],"logits":[0.6887646913528442,0.6843016147613525,-2.4599757194519043],"probs":[0.4905681312084198,0.48838356137275696,0.02104838564991951],"argmax_index":0,"argmax_id":"0","top2_margin":0.002184569835662842,"near_tie":true,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"circles":{"type":"choice","choice":"3","confidence":0.9385,"probabilities":{"1":0.0092,"2":0.0136,"3":0.9385,"4":0.0114,"5":0.014,"6":0.0134}},"more_circles":{"type":"noul","noul":0.8058},"clutter":{"type":"score","score":0.5305,"confidence":0.4906,"legend":{"0":"sparse","1":"moderate","2":"crowded"},"probabilities":{"0":0.4906,"1":0.4884,"2":0.021}}},"usage":{"input_tokens":430,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_02","arm":"g256","set":"fixture","source":"own_image","tokens":466,"n_image_tokens":64,"state_tokens":26,"token_offset":36,"image_files":["bars_640x480.png"],"image_size_in":[640,480],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"3c06ec3b20b0c66d854f6d5f743993af621ca6d3aaa4ef520dff3e20cb6e8835","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,24160,3147,2282,310,1245,25,2438,11,18016,11,13358,11,5983,11,6105,2129,7945,3147,4274,279,12037,3498,9207,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,78802,198,9222,25,5576,198,669,55530,25,220,22365,3498,369,279,78802,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,6105,3498,2129,1979,816,3147,11855,8934,198,32929,220,17,25,220,4754,4532,3147,760,5983,3498,2129,1979,816,3147,12865,8934,198,32929,220,18,25,220,4754,4532,3147,760,18016,3498,2129,1979,816,3147,33034,8934,198,32929,220,19,25,220,4754,4532,3147,760,2438,3498,2129,1979,816,3147,1114,8934,198,32929,220,20,25,220,4754,4532,3147,760,13358,3498,2129,1979,816,3147,26964,8934,198,4537,38945,198,198,27806,220,17,198,887,25,5983,14980,8469,818,198,9222,25,307,10644,198,669,55530,25,220,760,5983,3498,369,48230,1056,279,18016,3498,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,8597,198,9222,25,5284,198,669,55530,25,220,4199,58302,513,279,3498,34124,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,8823,5951,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,54842,77699,58302,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,2792,1002,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,1164,58302,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"tallest","type":"choice","question_span":[153,159],"option_spans":[[170,182],[188,200],[206,218],[224,236],[242,254]],"option_ids":["blue","green","orange","red","yellow"],"logits":[-4.874417304992676,-0.458612859249115,-4.732632637023926,-4.9626593589782715,-4.951841354370117],"probs":[0.01152847521007061,0.9539625644683838,0.013284590095281601,0.010554770939052105,0.010669572278857231],"argmax_index":1,"argmax_id":"green","top2_margin":0.9406779743731022,"near_tie":false,"gold":"green","gold_correct":true},{"question_id":"green_over_orange","type":"noul","question_span":[279,289],"option_spans":[[300,318],[324,342]],"option_ids":["true","false"],"logits":[-1.6133511066436768,-4.524852275848389],"probs":[0.9484120607376099,0.05158793926239014],"argmax_index":0,"argmax_id":"true","top2_margin":0.8968241214752197,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"spread","type":"score","question_span":[363,370],"option_spans":[[381,392],[398,410],[416,427],[433,444]],"option_ids":["0","1","2","3"],"logits":[-4.090166091918945,-2.588817834854126,-1.753161907196045,-2.677049398422241],"probs":[0.05013371258974075,0.22498683631420135,0.5188930630683899,0.20598644018173218],"argmax_index":2,"argmax_id":"2","top2_margin":0.29390622675418854,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"tallest":{"type":"choice","choice":"green","confidence":0.954,"probabilities":{"red":0.0106,"orange":0.0133,"yellow":0.0107,"green":0.954,"blue":0.0115}},"green_over_orange":{"type":"noul","noul":0.9484},"spread":{"type":"score","score":1.8807,"confidence":0.5189,"legend":{"0":"about equal","1":"somewhat uneven","2":"uneven","3":"very uneven"},"probabilities":{"0":0.0501,"1":0.225,"2":0.5189,"3":0.206}}},"usage":{"input_tokens":466,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_02","arm":"g448","set":"fixture","source":"own_image","tokens":598,"n_image_tokens":196,"state_tokens":26,"token_offset":36,"image_files":["bars_640x480.png"],"image_size_in":[640,480],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"a5f7d5f6e9c68cc6881d6008d9face706def9a31aac9ef67c4e4ad00f460bc24","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,24160,3147,2282,310,1245,25,2438,11,18016,11,13358,11,5983,11,6105,2129,7945,3147,4274,279,12037,3498,9207,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,78802,198,9222,25,5576,198,669,55530,25,220,22365,3498,369,279,78802,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,6105,3498,2129,1979,816,3147,11855,8934,198,32929,220,17,25,220,4754,4532,3147,760,5983,3498,2129,1979,816,3147,12865,8934,198,32929,220,18,25,220,4754,4532,3147,760,18016,3498,2129,1979,816,3147,33034,8934,198,32929,220,19,25,220,4754,4532,3147,760,2438,3498,2129,1979,816,3147,1114,8934,198,32929,220,20,25,220,4754,4532,3147,760,13358,3498,2129,1979,816,3147,26964,8934,198,4537,38945,198,198,27806,220,17,198,887,25,5983,14980,8469,818,198,9222,25,307,10644,198,669,55530,25,220,760,5983,3498,369,48230,1056,279,18016,3498,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,8597,198,9222,25,5284,198,669,55530,25,220,4199,58302,513,279,3498,34124,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,8823,5951,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,54842,77699,58302,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,2792,1002,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,1164,58302,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"tallest","type":"choice","question_span":[285,291],"option_spans":[[302,314],[320,332],[338,350],[356,368],[374,386]],"option_ids":["blue","green","orange","red","yellow"],"logits":[-4.878697872161865,-0.48188817501068115,-4.41578483581543,-4.970155715942383,-4.9223222732543945],"probs":[0.011675418354570866,0.9479438066482544,0.01854868046939373,0.010654984973371029,0.011177035048604012],"argmax_index":1,"argmax_id":"green","top2_margin":0.9293951261788607,"near_tie":false,"gold":"green","gold_correct":true},{"question_id":"green_over_orange","type":"noul","question_span":[411,421],"option_spans":[[432,450],[456,474]],"option_ids":["true","false"],"logits":[-1.5903263092041016,-4.365880489349365],"probs":[0.9413403868675232,0.05865956470370293],"argmax_index":0,"argmax_id":"true","top2_margin":0.8826808221638203,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"spread","type":"score","question_span":[495,502],"option_spans":[[513,524],[530,542],[548,559],[565,576]],"option_ids":["0","1","2","3"],"logits":[-3.6903655529022217,-2.0530762672424316,-1.1753017902374268,-2.0516958236694336],"probs":[0.04227091372013092,0.21732382476329803,0.5227811932563782,0.21762403845787048],"argmax_index":2,"argmax_id":"2","top2_margin":0.3051571547985077,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"tallest":{"type":"choice","choice":"green","confidence":0.9479,"probabilities":{"red":0.0107,"orange":0.0185,"yellow":0.0112,"green":0.9479,"blue":0.0117}},"green_over_orange":{"type":"noul","noul":0.9413},"spread":{"type":"score","score":1.9158,"confidence":0.5228,"legend":{"0":"about equal","1":"somewhat uneven","2":"uneven","3":"very uneven"},"probabilities":{"0":0.0423,"1":0.2173,"2":0.5228,"3":0.2176}}},"usage":{"input_tokens":598,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_02","arm":"native","set":"fixture","source":"own_image","tokens":702,"n_image_tokens":300,"state_tokens":26,"token_offset":36,"image_files":["bars_640x480.png"],"image_size_in":[640,480],"image_size_fed":[640,480],"resized_hw":[480,640],"grid_thw":[1,30,40],"merged_hw":[15,20],"rope_shift_amount":280,"pixel_values_shape":[1200,1536],"pixel_values_sha256":"f20a42e843813d1d549680437e342907f761285b4fbeb29b6b307525a0ba04c8","image_embeds_shape":[300,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,24160,3147,2282,310,1245,25,2438,11,18016,11,13358,11,5983,11,6105,2129,7945,3147,4274,279,12037,3498,9207,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,78802,198,9222,25,5576,198,669,55530,25,220,22365,3498,369,279,78802,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,6105,3498,2129,1979,816,3147,11855,8934,198,32929,220,17,25,220,4754,4532,3147,760,5983,3498,2129,1979,816,3147,12865,8934,198,32929,220,18,25,220,4754,4532,3147,760,18016,3498,2129,1979,816,3147,33034,8934,198,32929,220,19,25,220,4754,4532,3147,760,2438,3498,2129,1979,816,3147,1114,8934,198,32929,220,20,25,220,4754,4532,3147,760,13358,3498,2129,1979,816,3147,26964,8934,198,4537,38945,198,198,27806,220,17,198,887,25,5983,14980,8469,818,198,9222,25,307,10644,198,669,55530,25,220,760,5983,3498,369,48230,1056,279,18016,3498,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,8597,198,9222,25,5284,198,669,55530,25,220,4199,58302,513,279,3498,34124,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,8823,5951,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,54842,77699,58302,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,2792,1002,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,1164,58302,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"tallest","type":"choice","question_span":[389,395],"option_spans":[[406,418],[424,436],[442,454],[460,472],[478,490]],"option_ids":["blue","green","orange","red","yellow"],"logits":[-4.8624091148376465,-0.3616864085197449,-4.580626487731934,-4.933384895324707,-4.916268825531006],"probs":[0.010605943389236927,0.9554069638252258,0.014058077707886696,0.009879270568490028,0.010049820877611637],"argmax_index":1,"argmax_id":"green","top2_margin":0.9413488861173391,"near_tie":false,"gold":"green","gold_correct":true},{"question_id":"green_over_orange","type":"noul","question_span":[515,525],"option_spans":[[536,554],[560,578]],"option_ids":["true","false"],"logits":[-1.5574009418487549,-4.331693649291992],"probs":[0.9412707686424255,0.05872926488518715],"argmax_index":0,"argmax_id":"true","top2_margin":0.8825415037572384,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"spread","type":"score","question_span":[599,606],"option_spans":[[617,628],[634,646],[652,663],[669,680]],"option_ids":["0","1","2","3"],"logits":[-3.4475796222686768,-1.689032793045044,-0.801769495010376,-1.6871901750564575],"probs":[0.0374341756105423,0.21726784110069275,0.5276294350624084,0.2176685631275177],"argmax_index":2,"argmax_id":"2","top2_margin":0.30996087193489075,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"tallest":{"type":"choice","choice":"green","confidence":0.9554,"probabilities":{"red":0.0099,"orange":0.0141,"yellow":0.01,"green":0.9554,"blue":0.0106}},"green_over_orange":{"type":"noul","noul":0.9413},"spread":{"type":"score","score":1.9255,"confidence":0.5276,"legend":{"0":"about equal","1":"somewhat uneven","2":"uneven","3":"very uneven"},"probabilities":{"0":0.0374,"1":0.2173,"2":0.5276,"3":0.2177}}},"usage":{"input_tokens":702,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_03","arm":"g256","set":"fixture","source":"own_image","tokens":495,"n_image_tokens":64,"state_tokens":9,"token_offset":36,"image_files":["sevenseg_47_300x200.png"],"image_size_in":[300,200],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"f341fb030a0f803bc6f06abb70d5bd8427e114fb84bd1f9ecd9cf6578505439c","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,760,12037,2099,4774,264,1330,46730,2940,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,21330,46790,198,9222,25,5576,198,669,55530,25,220,3710,369,279,2047,15262,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,15,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,16,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,17,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,18,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,19,2129,1979,816,3147,19,8934,198,32929,220,21,25,220,4754,4532,3147,20,2129,1979,816,3147,20,8934,198,32929,220,22,25,220,4754,4532,3147,21,2129,1979,816,3147,21,8934,198,32929,220,23,25,220,4754,4532,3147,22,2129,1979,816,3147,22,8934,198,32929,220,24,25,220,4754,4532,3147,23,2129,1979,816,3147,23,8934,198,32929,220,16,15,25,220,4754,4532,3147,24,2129,1979,816,3147,24,8934,198,4537,38945,198,198,27806,220,17,198,887,25,1442,198,9222,25,307,10644,198,669,55530,25,220,760,12234,1324,369,1442,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,88341,198,9222,25,5284,198,669,55530,25,220,4199,3999,369,279,2940,310,1301,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,18140,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,43518,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"tens_digit","type":"choice","question_span":[137,143],"option_spans":[[154,164],[170,180],[186,196],[202,212],[218,228],[234,244],[250,260],[266,276],[282,292],[299,309]],"option_ids":["0","1","2","3","4","5","6","7","8","9"],"logits":[-4.251860618591309,-3.7797398567199707,-3.839890480041504,-3.5281877517700195,1.0920077562332153,-3.778508186340332,-3.9994988441467285,-4.000050067901611,-4.100477695465088,-3.8250679969787598],"probs":[0.004496998153626919,0.007210446521639824,0.006789519917219877,0.00927277747541666,0.9413154721260071,0.0072193327359855175,0.00578791368752718,0.005784723907709122,0.00523199699819088,0.006890906952321529],"argmax_index":4,"argmax_id":"4","top2_margin":0.9320426946505904,"near_tie":false,"gold":"4","gold_correct":true},{"question_id":"even","type":"noul","question_span":[331,337],"option_spans":[[348,366],[372,390]],"option_ids":["true","false"],"logits":[-0.8250815868377686,-0.6434381604194641],"probs":[0.4547136127948761,0.5452864170074463],"argmax_index":1,"argmax_id":"false","top2_margin":0.09057280421257019,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"readability","type":"score","question_span":[411,419],"option_spans":[[430,440],[446,457],[463,473]],"option_ids":["0","1","2"],"logits":[-2.2722296714782715,-1.3909504413604736,-0.04496648907661438],"probs":[0.07881192862987518,0.19025087356567383,0.7309371829032898],"argmax_index":2,"argmax_id":"2","top2_margin":0.540686309337616,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"tens_digit":{"type":"choice","choice":"4","confidence":0.9413,"probabilities":{"0":0.0045,"1":0.0072,"2":0.0068,"3":0.0093,"4":0.9413,"5":0.0072,"6":0.0058,"7":0.0058,"8":0.0052,"9":0.0069}},"even":{"type":"noul","noul":0.4547},"readability":{"type":"score","score":1.6521,"confidence":0.7309,"legend":{"0":"hard","1":"moderate","2":"easy"},"probabilities":{"0":0.0788,"1":0.1903,"2":0.7309}}},"usage":{"input_tokens":495,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_03","arm":"g448","set":"fixture","source":"own_image","tokens":627,"n_image_tokens":196,"state_tokens":9,"token_offset":36,"image_files":["sevenseg_47_300x200.png"],"image_size_in":[300,200],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"0cecb39c288eb5e96ac20400b29b96050535cc081aea8fd9997cdbc771124081","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,760,12037,2099,4774,264,1330,46730,2940,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,21330,46790,198,9222,25,5576,198,669,55530,25,220,3710,369,279,2047,15262,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,15,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,16,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,17,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,18,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,19,2129,1979,816,3147,19,8934,198,32929,220,21,25,220,4754,4532,3147,20,2129,1979,816,3147,20,8934,198,32929,220,22,25,220,4754,4532,3147,21,2129,1979,816,3147,21,8934,198,32929,220,23,25,220,4754,4532,3147,22,2129,1979,816,3147,22,8934,198,32929,220,24,25,220,4754,4532,3147,23,2129,1979,816,3147,23,8934,198,32929,220,16,15,25,220,4754,4532,3147,24,2129,1979,816,3147,24,8934,198,4537,38945,198,198,27806,220,17,198,887,25,1442,198,9222,25,307,10644,198,669,55530,25,220,760,12234,1324,369,1442,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,88341,198,9222,25,5284,198,669,55530,25,220,4199,3999,369,279,2940,310,1301,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,18140,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,43518,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"tens_digit","type":"choice","question_span":[269,275],"option_spans":[[286,296],[302,312],[318,328],[334,344],[350,360],[366,376],[382,392],[398,408],[414,424],[431,441]],"option_ids":["0","1","2","3","4","5","6","7","8","9"],"logits":[-4.925123691558838,-4.799170017242432,-4.942065238952637,-4.860466003417969,0.43050479888916016,-4.677213191986084,-4.980784893035889,-4.899515628814697,-4.987296104431152,-4.997246265411377],"probs":[0.0045227468945086,0.005129833705723286,0.004446770064532757,0.004824838601052761,0.9579044580459595,0.0057952008210122585,0.004277883563190699,0.004640061408281326,0.004250119440257549,0.004208039958029985],"argmax_index":4,"argmax_id":"4","top2_margin":0.9521092572249472,"near_tie":false,"gold":"4","gold_correct":true},{"question_id":"even","type":"noul","question_span":[463,469],"option_spans":[[480,498],[504,522]],"option_ids":["true","false"],"logits":[-3.8456902503967285,-0.8749568462371826],"probs":[0.04876568913459778,0.9512342810630798],"argmax_index":1,"argmax_id":"false","top2_margin":0.9024685919284821,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"readability","type":"score","question_span":[543,551],"option_spans":[[562,572],[578,589],[595,605]],"option_ids":["0","1","2"],"logits":[-4.168938159942627,-3.5297465324401855,-1.5850207805633545],"probs":[0.06194295361638069,0.11737870424985886,0.8206783533096313],"argmax_index":2,"argmax_id":"2","top2_margin":0.7032996490597725,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"tens_digit":{"type":"choice","choice":"4","confidence":0.9579,"probabilities":{"0":0.0045,"1":0.0051,"2":0.0044,"3":0.0048,"4":0.9579,"5":0.0058,"6":0.0043,"7":0.0046,"8":0.0043,"9":0.0042}},"even":{"type":"noul","noul":0.0488},"readability":{"type":"score","score":1.7587,"confidence":0.8207,"legend":{"0":"hard","1":"moderate","2":"easy"},"probabilities":{"0":0.0619,"1":0.1174,"2":0.8207}}},"usage":{"input_tokens":627,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_03","arm":"native","set":"fixture","source":"own_image","tokens":501,"n_image_tokens":70,"state_tokens":9,"token_offset":36,"image_files":["sevenseg_47_300x200.png"],"image_size_in":[300,200],"image_size_fed":[300,200],"resized_hw":[224,320],"grid_thw":[1,14,20],"merged_hw":[7,10],"rope_shift_amount":60,"pixel_values_shape":[280,1536],"pixel_values_sha256":"f3520ebe3fc075c302ab8255247fc0d601dfc3525027607441c0712ac60024d0","image_embeds_shape":[70,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,760,12037,2099,4774,264,1330,46730,2940,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,21330,46790,198,9222,25,5576,198,669,55530,25,220,3710,369,279,2047,15262,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,15,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,16,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,17,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,18,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,19,2129,1979,816,3147,19,8934,198,32929,220,21,25,220,4754,4532,3147,20,2129,1979,816,3147,20,8934,198,32929,220,22,25,220,4754,4532,3147,21,2129,1979,816,3147,21,8934,198,32929,220,23,25,220,4754,4532,3147,22,2129,1979,816,3147,22,8934,198,32929,220,24,25,220,4754,4532,3147,23,2129,1979,816,3147,23,8934,198,32929,220,16,15,25,220,4754,4532,3147,24,2129,1979,816,3147,24,8934,198,4537,38945,198,198,27806,220,17,198,887,25,1442,198,9222,25,307,10644,198,669,55530,25,220,760,12234,1324,369,1442,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,88341,198,9222,25,5284,198,669,55530,25,220,4199,3999,369,279,2940,310,1301,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,18140,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,43518,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"tens_digit","type":"choice","question_span":[143,149],"option_spans":[[160,170],[176,186],[192,202],[208,218],[224,234],[240,250],[256,266],[272,282],[288,298],[305,315]],"option_ids":["0","1","2","3","4","5","6","7","8","9"],"logits":[-4.898802757263184,-4.778792381286621,-4.898293972015381,-4.830085277557373,0.512681245803833,-4.679737567901611,-4.954373359680176,-4.863940238952637,-4.9539408683776855,-4.945536136627197],"probs":[0.004286513198167086,0.004833080805838108,0.004288692958652973,0.004591426346451044,0.9600232243537903,0.00533633166924119,0.004054806660860777,0.004438587464392185,0.004056559409946203,0.004090797156095505],"argmax_index":4,"argmax_id":"4","top2_margin":0.9546868926845491,"near_tie":false,"gold":"4","gold_correct":true},{"question_id":"even","type":"noul","question_span":[337,343],"option_spans":[[354,372],[378,396]],"option_ids":["true","false"],"logits":[-3.7363767623901367,-1.2663562297821045],"probs":[0.07798675447702408,0.9220132231712341],"argmax_index":1,"argmax_id":"false","top2_margin":0.84402646869421,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"readability","type":"score","question_span":[417,425],"option_spans":[[436,446],[452,463],[469,479]],"option_ids":["0","1","2"],"logits":[-4.0457000732421875,-3.2824435234069824,-1.5370104312896729],"probs":[0.064791738986969,0.13899452984333038,0.7962137460708618],"argmax_index":2,"argmax_id":"2","top2_margin":0.6572192162275314,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"tens_digit":{"type":"choice","choice":"4","confidence":0.96,"probabilities":{"0":0.0043,"1":0.0048,"2":0.0043,"3":0.0046,"4":0.96,"5":0.0053,"6":0.0041,"7":0.0044,"8":0.0041,"9":0.0041}},"even":{"type":"noul","noul":0.078},"readability":{"type":"score","score":1.7314,"confidence":0.7962,"legend":{"0":"hard","1":"moderate","2":"easy"},"probabilities":{"0":0.0648,"1":0.139,"2":0.7962}}},"usage":{"input_tokens":501,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_04","arm":"g256","set":"fixture","source":"own_image","tokens":558,"n_image_tokens":64,"state_tokens":9,"token_offset":36,"image_files":["receipt_1024x768.png"],"image_size_in":[1024,768],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"878c1888d05e6f24590743de0685dec01aa7e9af670fd1e13cacbb0bcf2cd434","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,7945,3147,4274,279,12037,21867,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,2702,198,9222,25,5576,198,669,55530,25,220,3710,369,279,21867,2702,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,20,13,22,20,2129,1979,816,3147,20,13,22,20,8934,198,32929,220,17,25,220,4754,4532,3147,21,13,20,22,2129,1979,816,3147,21,13,20,22,8934,198,32929,220,18,25,220,4754,4532,3147,21,13,22,20,2129,1979,816,3147,21,13,22,20,8934,198,32929,220,19,25,220,4754,4532,3147,22,13,21,20,2129,1979,816,3147,22,13,21,20,8934,198,4537,38945,198,198,27806,220,17,198,887,25,6945,62190,198,9222,25,307,10644,198,669,55530,25,220,760,7384,557,6945,303,8091,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,3470,198,9222,25,5576,198,669,55530,25,220,4199,1599,2086,3470,998,10471,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,20,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,21,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,19,198,887,25,2392,3047,198,9222,25,5284,198,669,55530,25,220,4199,2392,1196,369,279,21867,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,319,1871,1196,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,4332,391,2392,1196,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,3518,2392,1196,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"total","type":"choice","question_span":[136,142],"option_spans":[[153,169],[175,191],[197,213],[219,235]],"option_ids":["5.75","6.57","6.75","7.65"],"logits":[-3.67553973197937,-4.113386631011963,0.3759382367134094,-4.089323043823242],"probs":[0.016725508496165276,0.010795052163302898,0.9614214897155762,0.011057970114052296],"argmax_index":2,"argmax_id":"6.75","top2_margin":0.9446959812194109,"near_tie":false,"gold":"6.75","gold_correct":true},{"question_id":"paid_cash","type":"noul","question_span":[258,265],"option_spans":[[276,294],[300,318]],"option_ids":["true","false"],"logits":[-0.9275957942008972,-4.184687614440918],"probs":[0.9629271030426025,0.037072889506816864],"argmax_index":0,"argmax_id":"true","top2_margin":0.9258542135357857,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"items","type":"choice","question_span":[339,346],"option_spans":[[357,367],[373,383],[389,399],[405,415],[421,431],[437,447]],"option_ids":["1","2","3","4","5","6"],"logits":[-5.040473461151123,-4.750730514526367,-0.10806792974472046,-4.464332580566406,-4.983133792877197,-5.073338031768799],"probs":[0.00690346909686923,0.009223613888025284,0.957599401473999,0.01228237058967352,0.007310880348086357,0.0066802771762013435],"argmax_index":2,"argmax_id":"3","top2_margin":0.9453170308843255,"near_tie":false,"gold":"3","gold_correct":true},{"question_id":"legibility","type":"score","question_span":[469,476],"option_spans":[[487,499],[505,518],[524,536]],"option_ids":["0","1","2"],"logits":[-3.9303293228149414,-3.3579535484313965,-0.5415071845054626],"probs":[0.030860861763358116,0.05470004677772522,0.9144390821456909],"argmax_index":2,"argmax_id":"2","top2_margin":0.8597390353679657,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"total":{"type":"choice","choice":"6.75","confidence":0.9614,"probabilities":{"5.75":0.0167,"6.57":0.0108,"6.75":0.9614,"7.65":0.0111}},"paid_cash":{"type":"noul","noul":0.9629},"items":{"type":"choice","choice":"3","confidence":0.9576,"probabilities":{"1":0.0069,"2":0.0092,"3":0.9576,"4":0.0123,"5":0.0073,"6":0.0067}},"legibility":{"type":"score","score":1.8836,"confidence":0.9144,"legend":{"0":"illegible","1":"partly legible","2":"fully legible"},"probabilities":{"0":0.0309,"1":0.0547,"2":0.9144}}},"usage":{"input_tokens":558,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_04","arm":"g448","set":"fixture","source":"own_image","tokens":690,"n_image_tokens":196,"state_tokens":9,"token_offset":36,"image_files":["receipt_1024x768.png"],"image_size_in":[1024,768],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"1e976f27ef17b5ffffc8ecb7f24ae226749336a08e6776d79aa746bbef049d39","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,7945,3147,4274,279,12037,21867,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,2702,198,9222,25,5576,198,669,55530,25,220,3710,369,279,21867,2702,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,20,13,22,20,2129,1979,816,3147,20,13,22,20,8934,198,32929,220,17,25,220,4754,4532,3147,21,13,20,22,2129,1979,816,3147,21,13,20,22,8934,198,32929,220,18,25,220,4754,4532,3147,21,13,22,20,2129,1979,816,3147,21,13,22,20,8934,198,32929,220,19,25,220,4754,4532,3147,22,13,21,20,2129,1979,816,3147,22,13,21,20,8934,198,4537,38945,198,198,27806,220,17,198,887,25,6945,62190,198,9222,25,307,10644,198,669,55530,25,220,760,7384,557,6945,303,8091,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,3470,198,9222,25,5576,198,669,55530,25,220,4199,1599,2086,3470,998,10471,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,20,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,21,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,19,198,887,25,2392,3047,198,9222,25,5284,198,669,55530,25,220,4199,2392,1196,369,279,21867,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,319,1871,1196,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,4332,391,2392,1196,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,3518,2392,1196,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"total","type":"choice","question_span":[268,274],"option_spans":[[285,301],[307,323],[329,345],[351,367]],"option_ids":["5.75","6.57","6.75","7.65"],"logits":[-3.9721519947052,-4.2291107177734375,0.06417685747146606,-4.346233367919922],"probs":[0.016926366835832596,0.013090851716697216,0.9583387970924377,0.011643999256193638],"argmax_index":2,"argmax_id":"6.75","top2_margin":0.9414124302566051,"near_tie":false,"gold":"6.75","gold_correct":true},{"question_id":"paid_cash","type":"noul","question_span":[390,397],"option_spans":[[408,426],[432,450]],"option_ids":["true","false"],"logits":[-0.7267496585845947,-4.138790130615234],"probs":[0.9680786728858948,0.031921278685331345],"argmax_index":0,"argmax_id":"true","top2_margin":0.9361573942005634,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"items","type":"choice","question_span":[471,478],"option_spans":[[489,499],[505,515],[521,531],[537,547],[553,563],[569,579]],"option_ids":["1","2","3","4","5","6"],"logits":[-5.0609517097473145,-4.733673095703125,0.021243274211883545,-4.673628807067871,-5.0713677406311035,-5.113415241241455],"probs":[0.005990676116198301,0.008310193195939064,0.9652615189552307,0.008824458345770836,0.005928601138293743,0.005684486124664545],"argmax_index":2,"argmax_id":"3","top2_margin":0.9564370606094599,"near_tie":false,"gold":"3","gold_correct":true},{"question_id":"legibility","type":"score","question_span":[601,608],"option_spans":[[619,631],[637,650],[656,668]],"option_ids":["0","1","2"],"logits":[-3.575402021408081,-3.0210297107696533,-0.6964157819747925],"probs":[0.048692427575588226,0.08476611971855164,0.8665414452552795],"argmax_index":2,"argmax_id":"2","top2_margin":0.7817753255367279,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"total":{"type":"choice","choice":"6.75","confidence":0.9583,"probabilities":{"5.75":0.0169,"6.57":0.0131,"6.75":0.9583,"7.65":0.0116}},"paid_cash":{"type":"noul","noul":0.9681},"items":{"type":"choice","choice":"3","confidence":0.9653,"probabilities":{"1":0.006,"2":0.0083,"3":0.9653,"4":0.0088,"5":0.0059,"6":0.0057}},"legibility":{"type":"score","score":1.8178,"confidence":0.8665,"legend":{"0":"illegible","1":"partly legible","2":"fully legible"},"probabilities":{"0":0.0487,"1":0.0848,"2":0.8665}}},"usage":{"input_tokens":690,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_04","arm":"native","set":"fixture","source":"own_image","tokens":1262,"n_image_tokens":768,"state_tokens":9,"token_offset":36,"image_files":["receipt_1024x768.png"],"image_size_in":[1024,768],"image_size_fed":[1024,768],"resized_hw":[768,1024],"grid_thw":[1,48,64],"merged_hw":[24,32],"rope_shift_amount":736,"pixel_values_shape":[3072,1536],"pixel_values_sha256":"0721f059e58b5f9648fbe46f05e02e69b00c6a6bb19df753f6f8978d0aa45428","image_embeds_shape":[768,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,7945,3147,4274,279,12037,21867,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,2702,198,9222,25,5576,198,669,55530,25,220,3710,369,279,21867,2702,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,20,13,22,20,2129,1979,816,3147,20,13,22,20,8934,198,32929,220,17,25,220,4754,4532,3147,21,13,20,22,2129,1979,816,3147,21,13,20,22,8934,198,32929,220,18,25,220,4754,4532,3147,21,13,22,20,2129,1979,816,3147,21,13,22,20,8934,198,32929,220,19,25,220,4754,4532,3147,22,13,21,20,2129,1979,816,3147,22,13,21,20,8934,198,4537,38945,198,198,27806,220,17,198,887,25,6945,62190,198,9222,25,307,10644,198,669,55530,25,220,760,7384,557,6945,303,8091,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,3470,198,9222,25,5576,198,669,55530,25,220,4199,1599,2086,3470,998,10471,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,20,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,21,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,19,198,887,25,2392,3047,198,9222,25,5284,198,669,55530,25,220,4199,2392,1196,369,279,21867,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,319,1871,1196,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,4332,391,2392,1196,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,3518,2392,1196,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"total","type":"choice","question_span":[840,846],"option_spans":[[857,873],[879,895],[901,917],[923,939]],"option_ids":["5.75","6.57","6.75","7.65"],"logits":[-3.8432247638702393,-4.254848003387451,0.475582480430603,-4.287176609039307],"probs":[0.01291938591748476,0.00856004562228918,0.9702328443527222,0.008287736214697361],"argmax_index":2,"argmax_id":"6.75","top2_margin":0.9573134584352374,"near_tie":false,"gold":"6.75","gold_correct":true},{"question_id":"paid_cash","type":"noul","question_span":[962,969],"option_spans":[[980,998],[1004,1022]],"option_ids":["true","false"],"logits":[-1.148495078086853,-4.244000434875488],"probs":[0.9567070007324219,0.04329304397106171],"argmax_index":0,"argmax_id":"true","top2_margin":0.9134139567613602,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"items","type":"choice","question_span":[1043,1050],"option_spans":[[1061,1071],[1077,1087],[1093,1103],[1109,1119],[1125,1135],[1141,1151]],"option_ids":["1","2","3","4","5","6"],"logits":[-5.030008792877197,-4.7750043869018555,-0.07114660739898682,-4.64773416519165,-4.982321262359619,-5.0464301109313965],"probs":[0.006746713537722826,0.008706414140760899,0.9609456658363342,0.009888081811368465,0.0070762429386377335,0.006636828649789095],"argmax_index":2,"argmax_id":"3","top2_margin":0.9510575840249658,"near_tie":false,"gold":"3","gold_correct":true},{"question_id":"legibility","type":"score","question_span":[1173,1180],"option_spans":[[1191,1203],[1209,1222],[1228,1240]],"option_ids":["0","1","2"],"logits":[-3.5260887145996094,-3.1360116004943848,-0.5566095113754272],"probs":[0.045539695769548416,0.06726644188165665,0.8871938586235046],"argmax_index":2,"argmax_id":"2","top2_margin":0.819927416741848,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"total":{"type":"choice","choice":"6.75","confidence":0.9702,"probabilities":{"5.75":0.0129,"6.57":0.0086,"6.75":0.9702,"7.65":0.0083}},"paid_cash":{"type":"noul","noul":0.9567},"items":{"type":"choice","choice":"3","confidence":0.9609,"probabilities":{"1":0.0067,"2":0.0087,"3":0.9609,"4":0.0099,"5":0.0071,"6":0.0066}},"legibility":{"type":"score","score":1.8417,"confidence":0.8872,"legend":{"0":"illegible","1":"partly legible","2":"fully legible"},"probabilities":{"0":0.0455,"1":0.0673,"2":0.8872}}},"usage":{"input_tokens":1262,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_05","arm":"g256","set":"fixture","source":"own_image","tokens":464,"n_image_tokens":64,"state_tokens":8,"token_offset":36,"image_files":["signal_green_256x240.png"],"image_size_in":[256,240],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"5ae3bf3c20038b8d1fe4db2e58fd57e2591cb983ba700a56c8116d66f1a9814d","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,2523,513,9550,321,5313,411,8028,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,1852,198,9222,25,5576,198,669,55530,25,220,3710,1220,279,5404,635,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,81427,2129,1979,816,3147,3237,8934,198,32929,220,17,25,220,4754,4532,3147,56328,1441,321,10236,310,2842,2129,1979,816,3147,34045,8934,198,32929,220,18,25,220,4754,4532,3147,10358,2129,1979,816,3147,9215,8934,198,4537,38945,198,198,27806,220,17,198,887,25,12647,6504,198,9222,25,5576,198,669,55530,25,220,22365,27056,369,12647,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,18978,2129,1979,816,3147,12865,8934,198,32929,220,17,25,220,4754,4532,3147,5844,2129,1979,816,3147,1114,8934,198,32929,220,18,25,220,4754,4532,3147,46110,2129,1979,816,3147,26964,8934,198,4537,38945,198,198,27806,220,18,198,887,25,1189,2459,4485,198,9222,25,307,10644,198,669,55530,25,220,760,5404,1189,10046,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,26926,198,9222,25,5284,198,669,55530,25,220,4199,1691,26926,369,4221,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,53532,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,14265,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,64,2609,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"action","type":"choice","question_span":[135,141],"option_spans":[[152,162],[168,183],[189,199]],"option_ids":["go","slow","stop"],"logits":[-0.6721516847610474,-4.512298107147217,-4.917428493499756],"probs":[0.9654166102409363,0.020747238770127296,0.01383612398058176],"argmax_index":0,"argmax_id":"go","top2_margin":0.944669371470809,"near_tie":false,"gold":"go","gold_correct":true},{"question_id":"lit_color","type":"choice","question_span":[221,226],"option_spans":[[237,247],[253,263],[269,279]],"option_ids":["green","red","yellow"],"logits":[-0.27824705839157104,-5.045816898345947,-5.072797775268555],"probs":[0.9835010766983032,0.008360754698514938,0.00813819095492363],"argmax_index":0,"argmax_id":"green","top2_margin":0.9751403219997883,"near_tie":false,"gold":"green","gold_correct":true},{"question_id":"may_proceed","type":"noul","question_span":[303,308],"option_spans":[[319,337],[343,361]],"option_ids":["true","false"],"logits":[-2.1505064964294434,-4.861174583435059],"probs":[0.9376532435417175,0.06234678626060486],"argmax_index":0,"argmax_id":"true","top2_margin":0.8753064572811127,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"caution","type":"score","question_span":[382,388],"option_spans":[[399,409],[415,425],[431,442]],"option_ids":["0","1","2"],"logits":[-0.9645805954933167,-3.2149195671081543,-4.473725318908691],"probs":[0.8808353543281555,0.09280788898468018,0.026356790214776993],"argmax_index":0,"argmax_id":"0","top2_margin":0.7880274653434753,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"action":{"type":"choice","choice":"go","confidence":0.9654,"probabilities":{"stop":0.0138,"go":0.9654,"slow":0.0207}},"lit_color":{"type":"choice","choice":"green","confidence":0.9835,"probabilities":{"red":0.0084,"yellow":0.0081,"green":0.9835}},"may_proceed":{"type":"noul","noul":0.9377},"caution":{"type":"score","score":0.1455,"confidence":0.8808,"legend":{"0":"little","1":"some","2":"a lot"},"probabilities":{"0":0.8808,"1":0.0928,"2":0.0264}}},"usage":{"input_tokens":464,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_05","arm":"g448","set":"fixture","source":"own_image","tokens":596,"n_image_tokens":196,"state_tokens":8,"token_offset":36,"image_files":["signal_green_256x240.png"],"image_size_in":[256,240],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"1d98ed6453b0b996fb18110c37768e914d5be11a7dfc4a08d896bac09a8ac71b","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,2523,513,9550,321,5313,411,8028,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,1852,198,9222,25,5576,198,669,55530,25,220,3710,1220,279,5404,635,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,81427,2129,1979,816,3147,3237,8934,198,32929,220,17,25,220,4754,4532,3147,56328,1441,321,10236,310,2842,2129,1979,816,3147,34045,8934,198,32929,220,18,25,220,4754,4532,3147,10358,2129,1979,816,3147,9215,8934,198,4537,38945,198,198,27806,220,17,198,887,25,12647,6504,198,9222,25,5576,198,669,55530,25,220,22365,27056,369,12647,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,18978,2129,1979,816,3147,12865,8934,198,32929,220,17,25,220,4754,4532,3147,5844,2129,1979,816,3147,1114,8934,198,32929,220,18,25,220,4754,4532,3147,46110,2129,1979,816,3147,26964,8934,198,4537,38945,198,198,27806,220,18,198,887,25,1189,2459,4485,198,9222,25,307,10644,198,669,55530,25,220,760,5404,1189,10046,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,26926,198,9222,25,5284,198,669,55530,25,220,4199,1691,26926,369,4221,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,53532,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,14265,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,64,2609,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"action","type":"choice","question_span":[267,273],"option_spans":[[284,294],[300,315],[321,331]],"option_ids":["go","slow","stop"],"logits":[-0.6354588866233826,-4.5518479347229,-4.916301727294922],"probs":[0.9673576354980469,0.01926286704838276,0.01337952446192503],"argmax_index":0,"argmax_id":"go","top2_margin":0.9480947684496641,"near_tie":false,"gold":"go","gold_correct":true},{"question_id":"lit_color","type":"choice","question_span":[353,358],"option_spans":[[369,379],[385,395],[401,411]],"option_ids":["green","red","yellow"],"logits":[-0.2465057671070099,-5.033967018127441,-5.05131721496582],"probs":[0.983744740486145,0.00819812249392271,0.008057110011577606],"argmax_index":0,"argmax_id":"green","top2_margin":0.9755466179922223,"near_tie":false,"gold":"green","gold_correct":true},{"question_id":"may_proceed","type":"noul","question_span":[435,440],"option_spans":[[451,469],[475,493]],"option_ids":["true","false"],"logits":[-2.1533946990966797,-4.84774923324585],"probs":[0.9366927146911621,0.06330730766057968],"argmax_index":0,"argmax_id":"true","top2_margin":0.8733854070305824,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"caution","type":"score","question_span":[514,520],"option_spans":[[531,541],[547,557],[563,574]],"option_ids":["0","1","2"],"logits":[-0.8071925640106201,-3.2636871337890625,-4.470290184020996],"probs":[0.8997759222984314,0.07714225351810455,0.023081885650753975],"argmax_index":0,"argmax_id":"0","top2_margin":0.8226336687803268,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"action":{"type":"choice","choice":"go","confidence":0.9674,"probabilities":{"stop":0.0134,"go":0.9674,"slow":0.0193}},"lit_color":{"type":"choice","choice":"green","confidence":0.9837,"probabilities":{"red":0.0082,"yellow":0.0081,"green":0.9837}},"may_proceed":{"type":"noul","noul":0.9367},"caution":{"type":"score","score":0.1233,"confidence":0.8998,"legend":{"0":"little","1":"some","2":"a lot"},"probabilities":{"0":0.8998,"1":0.0771,"2":0.0231}}},"usage":{"input_tokens":596,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_05","arm":"native","set":"fixture","source":"own_image","tokens":464,"n_image_tokens":64,"state_tokens":8,"token_offset":36,"image_files":["signal_green_256x240.png"],"image_size_in":[256,240],"image_size_fed":[256,240],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"5ae3bf3c20038b8d1fe4db2e58fd57e2591cb983ba700a56c8116d66f1a9814d","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,2523,513,9550,321,5313,411,8028,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,1852,198,9222,25,5576,198,669,55530,25,220,3710,1220,279,5404,635,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,81427,2129,1979,816,3147,3237,8934,198,32929,220,17,25,220,4754,4532,3147,56328,1441,321,10236,310,2842,2129,1979,816,3147,34045,8934,198,32929,220,18,25,220,4754,4532,3147,10358,2129,1979,816,3147,9215,8934,198,4537,38945,198,198,27806,220,17,198,887,25,12647,6504,198,9222,25,5576,198,669,55530,25,220,22365,27056,369,12647,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,18978,2129,1979,816,3147,12865,8934,198,32929,220,17,25,220,4754,4532,3147,5844,2129,1979,816,3147,1114,8934,198,32929,220,18,25,220,4754,4532,3147,46110,2129,1979,816,3147,26964,8934,198,4537,38945,198,198,27806,220,18,198,887,25,1189,2459,4485,198,9222,25,307,10644,198,669,55530,25,220,760,5404,1189,10046,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,26926,198,9222,25,5284,198,669,55530,25,220,4199,1691,26926,369,4221,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,53532,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,14265,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,64,2609,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"action","type":"choice","question_span":[135,141],"option_spans":[[152,162],[168,183],[189,199]],"option_ids":["go","slow","stop"],"logits":[-0.6721516847610474,-4.512298107147217,-4.917428493499756],"probs":[0.9654166102409363,0.020747238770127296,0.01383612398058176],"argmax_index":0,"argmax_id":"go","top2_margin":0.944669371470809,"near_tie":false,"gold":"go","gold_correct":true},{"question_id":"lit_color","type":"choice","question_span":[221,226],"option_spans":[[237,247],[253,263],[269,279]],"option_ids":["green","red","yellow"],"logits":[-0.27824705839157104,-5.045816898345947,-5.072797775268555],"probs":[0.9835010766983032,0.008360754698514938,0.00813819095492363],"argmax_index":0,"argmax_id":"green","top2_margin":0.9751403219997883,"near_tie":false,"gold":"green","gold_correct":true},{"question_id":"may_proceed","type":"noul","question_span":[303,308],"option_spans":[[319,337],[343,361]],"option_ids":["true","false"],"logits":[-2.1505064964294434,-4.861174583435059],"probs":[0.9376532435417175,0.06234678626060486],"argmax_index":0,"argmax_id":"true","top2_margin":0.8753064572811127,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"caution","type":"score","question_span":[382,388],"option_spans":[[399,409],[415,425],[431,442]],"option_ids":["0","1","2"],"logits":[-0.9645805954933167,-3.2149195671081543,-4.473725318908691],"probs":[0.8808353543281555,0.09280788898468018,0.026356790214776993],"argmax_index":0,"argmax_id":"0","top2_margin":0.7880274653434753,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"action":{"type":"choice","choice":"go","confidence":0.9654,"probabilities":{"stop":0.0138,"go":0.9654,"slow":0.0207}},"lit_color":{"type":"choice","choice":"green","confidence":0.9835,"probabilities":{"red":0.0084,"yellow":0.0081,"green":0.9835}},"may_proceed":{"type":"noul","noul":0.9377},"caution":{"type":"score","score":0.1455,"confidence":0.8808,"legend":{"0":"little","1":"some","2":"a lot"},"probabilities":{"0":0.8808,"1":0.0928,"2":0.0264}}},"usage":{"input_tokens":464,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_06","arm":"g256","set":"fixture","source":"own_image","tokens":465,"n_image_tokens":64,"state_tokens":17,"token_offset":36,"image_files":["gauge_300x200.png"],"image_size_in":[300,200],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"1915d8711f4b5d4b3ef0f8141cfce35d9a551a697cf9de772827d5d6dc5993c2","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,64130,3147,726,5590,7035,29877,2129,1114,27753,83562,3625,28906,763,22,20,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,303,25225,27753,198,9222,25,307,10644,198,669,55530,25,220,760,29321,369,303,279,2438,9842,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,5122,198,9222,25,5576,198,669,55530,25,220,8791,1503,279,29321,1406,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,15,310,220,17,20,3161,2129,1979,816,3147,15,62,17,20,8934,198,32929,220,17,25,220,4754,4532,3147,17,20,310,220,20,15,3161,2129,1979,816,3147,17,20,62,20,15,8934,198,32929,220,18,25,220,4754,4532,3147,20,15,310,220,22,20,3161,2129,1979,816,3147,20,15,62,22,20,8934,198,32929,220,19,25,220,4754,4532,3147,22,20,310,220,16,15,15,3161,2129,1979,816,3147,22,20,62,16,15,15,8934,198,4537,38945,198,198,27806,220,18,198,887,25,16136,198,9222,25,5284,198,669,55530,25,220,3710,16136,2119,17759,279,5122,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,6487,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,327,2595,661,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,18357,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,57534,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"in_red_zone","type":"noul","question_span":[147,155],"option_spans":[[166,184],[190,208]],"option_ids":["true","false"],"logits":[-0.5656671524047852,-0.9104589223861694],"probs":[0.5853540301322937,0.4146459698677063],"argmax_index":0,"argmax_id":"true","top2_margin":0.1707080602645874,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"reading","type":"choice","question_span":[229,235],"option_spans":[[246,264],[270,290],[296,316],[322,344]],"option_ids":["0_25","25_50","50_75","75_100"],"logits":[-3.8498482704162598,-2.6872732639312744,-0.41771256923675537,-0.2742588520050049],"probs":[0.01411303598433733,0.04513571783900261,0.43669480085372925,0.504056453704834],"argmax_index":3,"argmax_id":"75_100","top2_margin":0.06736165285110474,"near_tie":false,"gold":"75_100","gold_correct":true},{"question_id":"alarm","type":"score","question_span":[365,372],"option_spans":[[383,393],[399,411],[417,427],[433,443]],"option_ids":["0","1","2","3"],"logits":[-1.8591291904449463,-0.9550069570541382,-0.10328108072280884,-2.0619850158691406],"probs":[0.0992603674530983,0.2451496124267578,0.5745542645454407,0.08103576302528381],"argmax_index":2,"argmax_id":"2","top2_margin":0.32940465211868286,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"in_red_zone":{"type":"noul","noul":0.5854},"reading":{"type":"choice","choice":"75_100","confidence":0.5041,"probabilities":{"0_25":0.0141,"25_50":0.0451,"50_75":0.4367,"75_100":0.5041}},"alarm":{"type":"score","score":1.6374,"confidence":0.5746,"legend":{"0":"none","1":"advisory","2":"warning","3":"shutdown"},"probabilities":{"0":0.0993,"1":0.2451,"2":0.5746,"3":0.081}}},"usage":{"input_tokens":465,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_06","arm":"g448","set":"fixture","source":"own_image","tokens":597,"n_image_tokens":196,"state_tokens":17,"token_offset":36,"image_files":["gauge_300x200.png"],"image_size_in":[300,200],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"e01d0fd37df39fc0ed57dc9b2859d23dfa3e1b5bce88c537bfcda71c1d7ab4c4","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,64130,3147,726,5590,7035,29877,2129,1114,27753,83562,3625,28906,763,22,20,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,303,25225,27753,198,9222,25,307,10644,198,669,55530,25,220,760,29321,369,303,279,2438,9842,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,5122,198,9222,25,5576,198,669,55530,25,220,8791,1503,279,29321,1406,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,15,310,220,17,20,3161,2129,1979,816,3147,15,62,17,20,8934,198,32929,220,17,25,220,4754,4532,3147,17,20,310,220,20,15,3161,2129,1979,816,3147,17,20,62,20,15,8934,198,32929,220,18,25,220,4754,4532,3147,20,15,310,220,22,20,3161,2129,1979,816,3147,20,15,62,22,20,8934,198,32929,220,19,25,220,4754,4532,3147,22,20,310,220,16,15,15,3161,2129,1979,816,3147,22,20,62,16,15,15,8934,198,4537,38945,198,198,27806,220,18,198,887,25,16136,198,9222,25,5284,198,669,55530,25,220,3710,16136,2119,17759,279,5122,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,6487,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,327,2595,661,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,18357,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,57534,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"in_red_zone","type":"noul","question_span":[279,287],"option_spans":[[298,316],[322,340]],"option_ids":["true","false"],"logits":[-0.5633763074874878,-0.5984160304069519],"probs":[0.5087590217590332,0.4912409484386444],"argmax_index":0,"argmax_id":"true","top2_margin":0.017518073320388794,"near_tie":true,"gold":"true","gold_correct":true},{"question_id":"reading","type":"choice","question_span":[361,367],"option_spans":[[378,396],[402,422],[428,448],[454,476]],"option_ids":["0_25","25_50","50_75","75_100"],"logits":[-3.5948352813720703,-2.681025505065918,-0.25926411151885986,-0.14977610111236572],"probs":[0.01588982157409191,0.039626117795705795,0.44641542434692383,0.4980686604976654],"argmax_index":3,"argmax_id":"75_100","top2_margin":0.05165323615074158,"near_tie":false,"gold":"75_100","gold_correct":true},{"question_id":"alarm","type":"score","question_span":[497,504],"option_spans":[[515,525],[531,543],[549,559],[565,575]],"option_ids":["0","1","2","3"],"logits":[-1.7170072793960571,-0.7966828346252441,0.03230470418930054,-1.8957470655441284],"probs":[0.09903861582279205,0.24859726428985596,0.5695358514785767,0.08282832056283951],"argmax_index":2,"argmax_id":"2","top2_margin":0.3209385871887207,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"in_red_zone":{"type":"noul","noul":0.5088},"reading":{"type":"choice","choice":"75_100","confidence":0.4981,"probabilities":{"0_25":0.0159,"25_50":0.0396,"50_75":0.4464,"75_100":0.4981}},"alarm":{"type":"score","score":1.6362,"confidence":0.5695,"legend":{"0":"none","1":"advisory","2":"warning","3":"shutdown"},"probabilities":{"0":0.099,"1":0.2486,"2":0.5695,"3":0.0828}}},"usage":{"input_tokens":597,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_06","arm":"native","set":"fixture","source":"own_image","tokens":471,"n_image_tokens":70,"state_tokens":17,"token_offset":36,"image_files":["gauge_300x200.png"],"image_size_in":[300,200],"image_size_fed":[300,200],"resized_hw":[224,320],"grid_thw":[1,14,20],"merged_hw":[7,10],"rope_shift_amount":60,"pixel_values_shape":[280,1536],"pixel_values_sha256":"a423a1eb3bbf6ce7d7ff178753cb596a2283c928f63c748b2cdb82419fd8a90c","image_embeds_shape":[70,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,64130,3147,726,5590,7035,29877,2129,1114,27753,83562,3625,28906,763,22,20,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,303,25225,27753,198,9222,25,307,10644,198,669,55530,25,220,760,29321,369,303,279,2438,9842,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,5122,198,9222,25,5576,198,669,55530,25,220,8791,1503,279,29321,1406,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,15,310,220,17,20,3161,2129,1979,816,3147,15,62,17,20,8934,198,32929,220,17,25,220,4754,4532,3147,17,20,310,220,20,15,3161,2129,1979,816,3147,17,20,62,20,15,8934,198,32929,220,18,25,220,4754,4532,3147,20,15,310,220,22,20,3161,2129,1979,816,3147,20,15,62,22,20,8934,198,32929,220,19,25,220,4754,4532,3147,22,20,310,220,16,15,15,3161,2129,1979,816,3147,22,20,62,16,15,15,8934,198,4537,38945,198,198,27806,220,18,198,887,25,16136,198,9222,25,5284,198,669,55530,25,220,3710,16136,2119,17759,279,5122,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,6487,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,327,2595,661,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,18357,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,57534,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"in_red_zone","type":"noul","question_span":[153,161],"option_spans":[[172,190],[196,214]],"option_ids":["true","false"],"logits":[-0.7577743530273438,-1.3586825132369995],"probs":[0.6458640694618225,0.3541359603404999],"argmax_index":0,"argmax_id":"true","top2_margin":0.29172810912132263,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"reading","type":"choice","question_span":[235,241],"option_spans":[[252,270],[276,296],[302,322],[328,350]],"option_ids":["0_25","25_50","50_75","75_100"],"logits":[-4.009824275970459,-3.054624557495117,-0.8524785041809082,-0.17773574590682983],"probs":[0.013648868538439274,0.03547599911689758,0.3208591639995575,0.6300159096717834],"argmax_index":3,"argmax_id":"75_100","top2_margin":0.30915674567222595,"near_tie":false,"gold":"75_100","gold_correct":true},{"question_id":"alarm","type":"score","question_span":[371,378],"option_spans":[[389,399],[405,417],[423,433],[439,449]],"option_ids":["0","1","2","3"],"logits":[-2.3995957374572754,-1.3581929206848145,-0.330084890127182,-2.3906049728393555],"probs":[0.07835044711828232,0.22198161482810974,0.6206098198890686,0.07905805110931396],"argmax_index":2,"argmax_id":"2","top2_margin":0.39862820506095886,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"in_red_zone":{"type":"noul","noul":0.6459},"reading":{"type":"choice","choice":"75_100","confidence":0.63,"probabilities":{"0_25":0.0136,"25_50":0.0355,"50_75":0.3209,"75_100":0.63}},"alarm":{"type":"score","score":1.7004,"confidence":0.6206,"legend":{"0":"none","1":"advisory","2":"warning","3":"shutdown"},"probabilities":{"0":0.0784,"1":0.222,"2":0.6206,"3":0.0791}}},"usage":{"input_tokens":471,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_07","arm":"g256","set":"fixture","source":"own_image","tokens":426,"n_image_tokens":64,"state_tokens":9,"token_offset":36,"image_files":["arrows_640x480.png"],"image_size_in":[640,480],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"5d4b1163bcfddd74998d037ff1013dbe23aefdc4c95066cbb9671011be49c238","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,9592,279,10016,799,680,4082,279,35732,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,10016,24633,198,9222,25,5576,198,669,55530,25,220,22365,1560,1503,279,10016,17385,1406,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,4308,2129,1979,816,3147,2829,8934,198,32929,220,17,25,220,4754,4532,3147,5247,2129,1979,816,3147,2282,8934,198,32929,220,18,25,220,4754,4532,3147,5791,2129,1979,816,3147,1246,8934,198,32929,220,19,25,220,4754,4532,3147,2248,2129,1979,816,3147,446,8934,198,4537,38945,198,198,27806,220,17,198,887,25,10016,9373,198,9222,25,5576,198,669,55530,25,220,22365,17385,369,279,10016,799,11,24217,494,279,2047,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1719,1118,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,1719,2018,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,1719,4688,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,1719,11400,2129,1979,816,3147,19,8934,198,4537,38945,198,198,27806,220,18,198,887,25,660,32462,198,9222,25,307,10644,198,669,55530,25,220,2324,35732,1406,279,1788,1560,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"odd_direction","type":"choice","question_span":[137,145],"option_spans":[[156,166],[172,182],[188,198],[204,214]],"option_ids":["down","left","right","up"],"logits":[-3.3205482959747314,0.9951578378677368,-1.1873530149459839,-3.086421251296997],"probs":[0.011686054058372974,0.8748937249183655,0.09865131229162216,0.014768893830478191],"argmax_index":1,"argmax_id":"left","top2_margin":0.7762424126267433,"near_tie":false,"gold":"up","gold_correct":false},{"question_id":"odd_position","type":"choice","question_span":[236,248],"option_spans":[[259,270],[276,287],[293,304],[310,321]],"option_ids":["1","2","3","4"],"logits":[0.4614751636981964,-0.20208501815795898,-1.1038810014724731,-1.844350814819336],"probs":[0.5483344793319702,0.2824002206325531,0.11460934579372406,0.054656028747558594],"argmax_index":0,"argmax_id":"1","top2_margin":0.2659342586994171,"near_tie":false,"gold":"2","gold_correct":false},{"question_id":"all_same","type":"noul","question_span":[344,351],"option_spans":[[362,380],[386,404]],"option_ids":["true","false"],"logits":[-0.7864351272583008,1.1590898036956787],"probs":[0.1250421404838562,0.8749578595161438],"argmax_index":1,"argmax_id":"false","top2_margin":0.7499157190322876,"near_tie":false,"gold":"false","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"odd_direction":{"type":"choice","choice":"left","confidence":0.8749,"probabilities":{"up":0.0148,"down":0.0117,"left":0.8749,"right":0.0987}},"odd_position":{"type":"choice","choice":"1","confidence":0.5483,"probabilities":{"1":0.5483,"2":0.2824,"3":0.1146,"4":0.0547}},"all_same":{"type":"noul","noul":0.125}},"usage":{"input_tokens":426,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_07","arm":"g448","set":"fixture","source":"own_image","tokens":558,"n_image_tokens":196,"state_tokens":9,"token_offset":36,"image_files":["arrows_640x480.png"],"image_size_in":[640,480],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"f0e848bd844308aa426dbcf7972377b61b436297941f370acc47cec5c198cba4","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,9592,279,10016,799,680,4082,279,35732,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,10016,24633,198,9222,25,5576,198,669,55530,25,220,22365,1560,1503,279,10016,17385,1406,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,4308,2129,1979,816,3147,2829,8934,198,32929,220,17,25,220,4754,4532,3147,5247,2129,1979,816,3147,2282,8934,198,32929,220,18,25,220,4754,4532,3147,5791,2129,1979,816,3147,1246,8934,198,32929,220,19,25,220,4754,4532,3147,2248,2129,1979,816,3147,446,8934,198,4537,38945,198,198,27806,220,17,198,887,25,10016,9373,198,9222,25,5576,198,669,55530,25,220,22365,17385,369,279,10016,799,11,24217,494,279,2047,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1719,1118,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,1719,2018,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,1719,4688,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,1719,11400,2129,1979,816,3147,19,8934,198,4537,38945,198,198,27806,220,18,198,887,25,660,32462,198,9222,25,307,10644,198,669,55530,25,220,2324,35732,1406,279,1788,1560,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"odd_direction","type":"choice","question_span":[269,277],"option_spans":[[288,298],[304,314],[320,330],[336,346]],"option_ids":["down","left","right","up"],"logits":[-4.644463539123535,-4.166157245635986,-4.522230625152588,-0.038750529289245605],"probs":[0.009634129703044891,0.01554312277585268,0.010886733420193195,0.9639360308647156],"argmax_index":3,"argmax_id":"up","top2_margin":0.9483929080888629,"near_tie":false,"gold":"up","gold_correct":true},{"question_id":"odd_position","type":"choice","question_span":[368,380],"option_spans":[[391,402],[408,419],[425,436],[442,453]],"option_ids":["1","2","3","4"],"logits":[-3.5871903896331787,-1.0160481929779053,-3.610060930252075,-4.167004108428955],"probs":[0.06402807682752609,0.8375357389450073,0.06258033961057663,0.03585589677095413],"argmax_index":1,"argmax_id":"2","top2_margin":0.7735076621174812,"near_tie":false,"gold":"2","gold_correct":true},{"question_id":"all_same","type":"noul","question_span":[476,483],"option_spans":[[494,512],[518,536]],"option_ids":["true","false"],"logits":[-2.6470413208007812,1.0941311120986938],"probs":[0.02317638508975506,0.9768236875534058],"argmax_index":1,"argmax_id":"false","top2_margin":0.9536473024636507,"near_tie":false,"gold":"false","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"odd_direction":{"type":"choice","choice":"up","confidence":0.9639,"probabilities":{"up":0.9639,"down":0.0096,"left":0.0155,"right":0.0109}},"odd_position":{"type":"choice","choice":"2","confidence":0.8375,"probabilities":{"1":0.064,"2":0.8375,"3":0.0626,"4":0.0359}},"all_same":{"type":"noul","noul":0.0232}},"usage":{"input_tokens":558,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_07","arm":"native","set":"fixture","source":"own_image","tokens":662,"n_image_tokens":300,"state_tokens":9,"token_offset":36,"image_files":["arrows_640x480.png"],"image_size_in":[640,480],"image_size_fed":[640,480],"resized_hw":[480,640],"grid_thw":[1,30,40],"merged_hw":[15,20],"rope_shift_amount":280,"pixel_values_shape":[1200,1536],"pixel_values_sha256":"6fecb5b3e0430dfc27c9b1b41dc80347dd14b7c2a061e957e129fcdb285017d8","image_embeds_shape":[300,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,9592,279,10016,799,680,4082,279,35732,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,10016,24633,198,9222,25,5576,198,669,55530,25,220,22365,1560,1503,279,10016,17385,1406,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,4308,2129,1979,816,3147,2829,8934,198,32929,220,17,25,220,4754,4532,3147,5247,2129,1979,816,3147,2282,8934,198,32929,220,18,25,220,4754,4532,3147,5791,2129,1979,816,3147,1246,8934,198,32929,220,19,25,220,4754,4532,3147,2248,2129,1979,816,3147,446,8934,198,4537,38945,198,198,27806,220,17,198,887,25,10016,9373,198,9222,25,5576,198,669,55530,25,220,22365,17385,369,279,10016,799,11,24217,494,279,2047,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1719,1118,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,1719,2018,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,1719,4688,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,1719,11400,2129,1979,816,3147,19,8934,198,4537,38945,198,198,27806,220,18,198,887,25,660,32462,198,9222,25,307,10644,198,669,55530,25,220,2324,35732,1406,279,1788,1560,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"odd_direction","type":"choice","question_span":[373,381],"option_spans":[[392,402],[408,418],[424,434],[440,450]],"option_ids":["down","left","right","up"],"logits":[-5.028372287750244,-4.791247367858887,-4.8876752853393555,-0.1987958550453186],"probs":[0.007777463179081678,0.009858709760010242,0.008952450007200241,0.9734113812446594],"argmax_index":3,"argmax_id":"up","top2_margin":0.9635526714846492,"near_tie":false,"gold":"up","gold_correct":true},{"question_id":"odd_position","type":"choice","question_span":[472,484],"option_spans":[[495,506],[512,523],[529,540],[546,557]],"option_ids":["1","2","3","4"],"logits":[-4.167104721069336,-1.2124003171920776,-4.101622581481934,-4.2872819900512695],"probs":[0.04514573886990547,0.8666196465492249,0.04820091649889946,0.040033578872680664],"argmax_index":1,"argmax_id":"2","top2_margin":0.8184187300503254,"near_tie":false,"gold":"2","gold_correct":true},{"question_id":"all_same","type":"noul","question_span":[580,587],"option_spans":[[598,616],[622,640]],"option_ids":["true","false"],"logits":[-3.1151516437530518,1.1064088344573975],"probs":[0.014463462866842747,0.985536515712738],"argmax_index":1,"argmax_id":"false","top2_margin":0.9710730528458953,"near_tie":false,"gold":"false","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"odd_direction":{"type":"choice","choice":"up","confidence":0.9734,"probabilities":{"up":0.9734,"down":0.0078,"left":0.0099,"right":0.009}},"odd_position":{"type":"choice","choice":"2","confidence":0.8666,"probabilities":{"1":0.0451,"2":0.8666,"3":0.0482,"4":0.04}},"all_same":{"type":"noul","noul":0.0145}},"usage":{"input_tokens":662,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_08","arm":"g256","set":"fixture","source":"own_image","tokens":522,"n_image_tokens":64,"state_tokens":12,"token_offset":36,"image_files":["floorplan_1024x768.png"],"image_size_in":[1024,768],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"d6fab13dd6413d91e6528a74ceb6cba188857be4f73335a80db1580a28e5dc93","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,7945,3147,15666,4602,883,279,12037,6220,3019,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,7526,24211,198,9222,25,5576,198,669,55530,25,220,22365,2956,369,279,7526,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,33,75171,2129,1979,816,3147,65,572,8934,198,32929,220,17,25,220,4754,4532,3147,15225,2871,2129,1979,816,3147,2634,8934,198,32929,220,18,25,220,4754,4532,3147,81663,2129,1979,816,3147,74,7219,8934,198,32929,220,19,25,220,4754,4532,3147,43027,2956,2129,1979,816,3147,61041,8934,198,4537,38945,198,198,27806,220,17,198,887,25,11683,198,9222,25,5576,198,669,55530,25,220,4199,1599,58205,12258,513,1017,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,17,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,18,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,19,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,20,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,18,198,887,25,8617,10930,2270,86012,198,9222,25,307,10644,198,669,55530,25,220,760,14422,369,1727,310,279,13433,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,22373,198,9222,25,5284,198,669,55530,25,220,4199,6150,369,279,6576,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,22230,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,22523,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"largest_room","type":"choice","question_span":[140,146],"option_spans":[[157,169],[175,186],[192,203],[209,220]],"option_ids":["bath","bed","kitchen","living"],"logits":[-4.689757347106934,-3.5719997882843018,-3.945399761199951,-0.31737780570983887],"probs":[0.011710288003087044,0.035809922963380814,0.024651188403367996,0.9278286099433899],"argmax_index":3,"argmax_id":"living","top2_margin":0.8920186869800091,"near_tie":false,"gold":"living","gold_correct":true},{"question_id":"rooms","type":"choice","question_span":[241,248],"option_spans":[[259,269],[275,285],[291,301],[307,317],[323,333]],"option_ids":["2","3","4","5","6"],"logits":[-4.230262756347656,-3.1470680236816406,-1.6863163709640503,-0.17727130651474,-2.4015002250671387],"probs":[0.012425524182617664,0.036706265062093735,0.1581745147705078,0.7153301239013672,0.0773635134100914],"argmax_index":3,"argmax_id":"5","top2_margin":0.5571556091308594,"near_tie":false,"gold":"5","gold_correct":true},{"question_id":"bath_next_to_bed","type":"noul","question_span":[358,366],"option_spans":[[377,395],[401,419]],"option_ids":["true","false"],"logits":[-0.9627602100372314,-0.12860530614852905],"probs":[0.30276724696159363,0.697232723236084],"argmax_index":1,"argmax_id":"false","top2_margin":0.39446547627449036,"near_tie":false,"gold":"true","gold_correct":false},{"question_id":"complexity","type":"score","question_span":[440,446],"option_spans":[[457,467],[473,484],[490,500]],"option_ids":["0","1","2"],"logits":[-0.8032771348953247,0.40570342540740967,-2.0746684074401855],"probs":[0.21595898270606995,0.7234771847724915,0.06056378409266472],"argmax_index":1,"argmax_id":"1","top2_margin":0.5075182020664215,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"largest_room":{"type":"choice","choice":"living","confidence":0.9278,"probabilities":{"living":0.9278,"kitchen":0.0247,"bed":0.0358,"bath":0.0117}},"rooms":{"type":"choice","choice":"5","confidence":0.7153,"probabilities":{"2":0.0124,"3":0.0367,"4":0.1582,"5":0.7153,"6":0.0774}},"bath_next_to_bed":{"type":"noul","noul":0.3028},"complexity":{"type":"score","score":0.8446,"confidence":0.7235,"legend":{"0":"simple","1":"moderate","2":"complex"},"probabilities":{"0":0.216,"1":0.7235,"2":0.0606}}},"usage":{"input_tokens":522,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_08","arm":"g448","set":"fixture","source":"own_image","tokens":654,"n_image_tokens":196,"state_tokens":12,"token_offset":36,"image_files":["floorplan_1024x768.png"],"image_size_in":[1024,768],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"3b6f6fdccbbd1aab7d2ac10635a2bf6492104dc207345ef2384260e625fe4b28","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,7945,3147,15666,4602,883,279,12037,6220,3019,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,7526,24211,198,9222,25,5576,198,669,55530,25,220,22365,2956,369,279,7526,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,33,75171,2129,1979,816,3147,65,572,8934,198,32929,220,17,25,220,4754,4532,3147,15225,2871,2129,1979,816,3147,2634,8934,198,32929,220,18,25,220,4754,4532,3147,81663,2129,1979,816,3147,74,7219,8934,198,32929,220,19,25,220,4754,4532,3147,43027,2956,2129,1979,816,3147,61041,8934,198,4537,38945,198,198,27806,220,17,198,887,25,11683,198,9222,25,5576,198,669,55530,25,220,4199,1599,58205,12258,513,1017,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,17,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,18,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,19,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,20,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,18,198,887,25,8617,10930,2270,86012,198,9222,25,307,10644,198,669,55530,25,220,760,14422,369,1727,310,279,13433,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,22373,198,9222,25,5284,198,669,55530,25,220,4199,6150,369,279,6576,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,22230,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,22523,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"largest_room","type":"choice","question_span":[272,278],"option_spans":[[289,301],[307,318],[324,335],[341,352]],"option_ids":["bath","bed","kitchen","living"],"logits":[-4.704373359680176,-3.661036729812622,-4.081873893737793,-0.6408155560493469],"probs":[0.015653572976589203,0.04443535953760147,0.02917167916893959,0.9107393622398376],"argmax_index":3,"argmax_id":"living","top2_margin":0.8663040027022362,"near_tie":false,"gold":"living","gold_correct":true},{"question_id":"rooms","type":"choice","question_span":[373,380],"option_spans":[[391,401],[407,417],[423,433],[439,449],[455,465]],"option_ids":["2","3","4","5","6"],"logits":[-4.570162773132324,-3.8160345554351807,-2.5107262134552,-0.26239013671875,-2.680215358734131],"probs":[0.010885989293456078,0.02314097434282303,0.08536303788423538,0.8085554242134094,0.07205459475517273],"argmax_index":3,"argmax_id":"5","top2_margin":0.723192386329174,"near_tie":false,"gold":"5","gold_correct":true},{"question_id":"bath_next_to_bed","type":"noul","question_span":[490,498],"option_spans":[[509,527],[533,551]],"option_ids":["true","false"],"logits":[-1.1055936813354492,-0.14903241395950317],"probs":[0.27756720781326294,0.7224327921867371],"argmax_index":1,"argmax_id":"false","top2_margin":0.4448655843734741,"near_tie":false,"gold":"true","gold_correct":false},{"question_id":"complexity","type":"score","question_span":[572,578],"option_spans":[[589,599],[605,616],[622,632]],"option_ids":["0","1","2"],"logits":[-0.9052980542182922,0.2961692214012146,-2.2906696796417236],"probs":[0.2185685932636261,0.7267388105392456,0.0546925850212574],"argmax_index":1,"argmax_id":"1","top2_margin":0.5081702172756195,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"largest_room":{"type":"choice","choice":"living","confidence":0.9107,"probabilities":{"living":0.9107,"kitchen":0.0292,"bed":0.0444,"bath":0.0157}},"rooms":{"type":"choice","choice":"5","confidence":0.8086,"probabilities":{"2":0.0109,"3":0.0231,"4":0.0854,"5":0.8086,"6":0.0721}},"bath_next_to_bed":{"type":"noul","noul":0.2776},"complexity":{"type":"score","score":0.8361,"confidence":0.7267,"legend":{"0":"simple","1":"moderate","2":"complex"},"probabilities":{"0":0.2186,"1":0.7267,"2":0.0547}}},"usage":{"input_tokens":654,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_08","arm":"native","set":"fixture","source":"own_image","tokens":1226,"n_image_tokens":768,"state_tokens":12,"token_offset":36,"image_files":["floorplan_1024x768.png"],"image_size_in":[1024,768],"image_size_fed":[1024,768],"resized_hw":[768,1024],"grid_thw":[1,48,64],"merged_hw":[24,32],"rope_shift_amount":736,"pixel_values_shape":[3072,1536],"pixel_values_sha256":"115a19535fe9b15870ca778580f55fa27b44b2c1c89067d5cdf96d02b2c927eb","image_embeds_shape":[768,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,7945,3147,15666,4602,883,279,12037,6220,3019,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,7526,24211,198,9222,25,5576,198,669,55530,25,220,22365,2956,369,279,7526,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,33,75171,2129,1979,816,3147,65,572,8934,198,32929,220,17,25,220,4754,4532,3147,15225,2871,2129,1979,816,3147,2634,8934,198,32929,220,18,25,220,4754,4532,3147,81663,2129,1979,816,3147,74,7219,8934,198,32929,220,19,25,220,4754,4532,3147,43027,2956,2129,1979,816,3147,61041,8934,198,4537,38945,198,198,27806,220,17,198,887,25,11683,198,9222,25,5576,198,669,55530,25,220,4199,1599,58205,12258,513,1017,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,17,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,18,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,19,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,20,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,18,198,887,25,8617,10930,2270,86012,198,9222,25,307,10644,198,669,55530,25,220,760,14422,369,1727,310,279,13433,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,22373,198,9222,25,5284,198,669,55530,25,220,4199,6150,369,279,6576,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,22230,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,22523,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"largest_room","type":"choice","question_span":[844,850],"option_spans":[[861,873],[879,890],[896,907],[913,924]],"option_ids":["bath","bed","kitchen","living"],"logits":[-4.625689506530762,-3.524610996246338,-3.987359046936035,-0.2288021743297577],"probs":[0.011481259949505329,0.03452882915735245,0.02173766680061817,0.9322522282600403],"argmax_index":3,"argmax_id":"living","top2_margin":0.8977233991026878,"near_tie":false,"gold":"living","gold_correct":true},{"question_id":"rooms","type":"choice","question_span":[945,952],"option_spans":[[963,973],[979,989],[995,1005],[1011,1021],[1027,1037]],"option_ids":["2","3","4","5","6"],"logits":[-4.142745018005371,-2.786344528198242,-0.2320374846458435,-0.552047848701477,-2.609405994415283],"probs":[0.010448291897773743,0.04056229814887047,0.5217278003692627,0.37884822487831116,0.04841344431042671],"argmax_index":2,"argmax_id":"4","top2_margin":0.14287957549095154,"near_tie":false,"gold":"5","gold_correct":false},{"question_id":"bath_next_to_bed","type":"noul","question_span":[1062,1070],"option_spans":[[1081,1099],[1105,1123]],"option_ids":["true","false"],"logits":[-0.8999065160751343,-0.08833837509155273],"probs":[0.3075564503669739,0.6924436092376709],"argmax_index":1,"argmax_id":"false","top2_margin":0.384887158870697,"near_tie":false,"gold":"true","gold_correct":false},{"question_id":"complexity","type":"score","question_span":[1144,1150],"option_spans":[[1161,1171],[1177,1188],[1194,1204]],"option_ids":["0","1","2"],"logits":[-0.483359694480896,0.4852072298526764,-2.0150461196899414],"probs":[0.25971749424934387,0.6841391921043396,0.05614333227276802],"argmax_index":1,"argmax_id":"1","top2_margin":0.4244216978549957,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"largest_room":{"type":"choice","choice":"living","confidence":0.9323,"probabilities":{"living":0.9323,"kitchen":0.0217,"bed":0.0345,"bath":0.0115}},"rooms":{"type":"choice","choice":"4","confidence":0.5217,"probabilities":{"2":0.0104,"3":0.0406,"4":0.5217,"5":0.3788,"6":0.0484}},"bath_next_to_bed":{"type":"noul","noul":0.3076},"complexity":{"type":"score","score":0.7964,"confidence":0.6841,"legend":{"0":"simple","1":"moderate","2":"complex"},"probabilities":{"0":0.2597,"1":0.6841,"2":0.0561}}},"usage":{"input_tokens":1226,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_09","arm":"g256","set":"fixture","source":"own_image","tokens":460,"n_image_tokens":64,"state_tokens":4,"token_offset":36,"image_files":["circles_640x480.png"],"image_size_in":[640,480],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"c5326c92abc8314fe72a02f67245fe0885271396b40d5c6c0db22eae08870fd6","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,26487,279,24557,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,7526,198,9222,25,5576,198,669,55530,25,220,3710,1829,369,279,7526,12307,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,10025,2129,1979,816,3147,11855,8934,198,32929,220,17,25,220,4754,4532,3147,18978,2129,1979,816,3147,12865,8934,198,32929,220,18,25,220,4754,4532,3147,5844,2129,1979,816,3147,1114,8934,198,4537,38945,198,198,27806,220,17,198,887,25,23856,198,9222,25,5576,198,669,55530,25,220,3710,1829,369,279,23856,12307,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,10025,2129,1979,816,3147,11855,8934,198,32929,220,17,25,220,4754,4532,3147,18978,2129,1979,816,3147,12865,8934,198,32929,220,18,25,220,4754,4532,3147,5844,2129,1979,816,3147,1114,8934,198,4537,38945,198,198,27806,220,18,198,887,25,1788,2291,198,9222,25,307,10644,198,669,55530,25,220,2324,24557,513,279,1788,1331,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,8597,198,9222,25,5284,198,669,55530,25,220,4199,2086,513,279,11930,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,62116,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,54842,77699,2086,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,1164,2086,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"largest","type":"choice","question_span":[131,138],"option_spans":[[149,159],[165,175],[181,191]],"option_ids":["blue","green","red"],"logits":[-4.8918681144714355,-4.952845573425293,-0.3727675676345825],"probs":[0.010673053562641144,0.010041683912277222,0.9792851805686951],"argmax_index":2,"argmax_id":"red","top2_margin":0.9686121270060539,"near_tie":false,"gold":"red","gold_correct":true},{"question_id":"smallest","type":"choice","question_span":[212,219],"option_spans":[[230,240],[246,256],[262,272]],"option_ids":["blue","green","red"],"logits":[-0.7329745292663574,-4.7116780281066895,-4.932882308959961],"probs":[0.9673922061920166,0.018099794164299965,0.014507941901683807],"argmax_index":0,"argmax_id":"blue","top2_margin":0.9492924120277166,"near_tie":false,"gold":"blue","gold_correct":true},{"question_id":"same_size","type":"noul","question_span":[295,302],"option_spans":[[313,331],[337,355]],"option_ids":["true","false"],"logits":[-2.6914029121398926,1.7342467308044434],"probs":[0.011824932880699635,0.9881750345230103],"argmax_index":1,"argmax_id":"false","top2_margin":0.9763501016423106,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"spread","type":"score","question_span":[376,382],"option_spans":[[393,403],[409,421],[427,438]],"option_ids":["0","1","2"],"logits":[-3.8201637268066406,-2.5559260845184326,-1.5901036262512207],"probs":[0.07225006073713303,0.25579360127449036,0.6719563007354736],"argmax_index":2,"argmax_id":"2","top2_margin":0.4161626994609833,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"largest":{"type":"choice","choice":"red","confidence":0.9793,"probabilities":{"blue":0.0107,"red":0.9793,"green":0.01}},"smallest":{"type":"choice","choice":"blue","confidence":0.9674,"probabilities":{"blue":0.9674,"red":0.0145,"green":0.0181}},"same_size":{"type":"noul","noul":0.0118},"spread":{"type":"score","score":1.5997,"confidence":0.672,"legend":{"0":"similar","1":"somewhat different","2":"very different"},"probabilities":{"0":0.0723,"1":0.2558,"2":0.672}}},"usage":{"input_tokens":460,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_09","arm":"g448","set":"fixture","source":"own_image","tokens":592,"n_image_tokens":196,"state_tokens":4,"token_offset":36,"image_files":["circles_640x480.png"],"image_size_in":[640,480],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"8da551e9d166bb466ff12a50e34f68ccce9c99ceaa1d9c5c41f6243ee88e136c","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,26487,279,24557,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,7526,198,9222,25,5576,198,669,55530,25,220,3710,1829,369,279,7526,12307,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,10025,2129,1979,816,3147,11855,8934,198,32929,220,17,25,220,4754,4532,3147,18978,2129,1979,816,3147,12865,8934,198,32929,220,18,25,220,4754,4532,3147,5844,2129,1979,816,3147,1114,8934,198,4537,38945,198,198,27806,220,17,198,887,25,23856,198,9222,25,5576,198,669,55530,25,220,3710,1829,369,279,23856,12307,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,10025,2129,1979,816,3147,11855,8934,198,32929,220,17,25,220,4754,4532,3147,18978,2129,1979,816,3147,12865,8934,198,32929,220,18,25,220,4754,4532,3147,5844,2129,1979,816,3147,1114,8934,198,4537,38945,198,198,27806,220,18,198,887,25,1788,2291,198,9222,25,307,10644,198,669,55530,25,220,2324,24557,513,279,1788,1331,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,8597,198,9222,25,5284,198,669,55530,25,220,4199,2086,513,279,11930,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,62116,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,54842,77699,2086,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,1164,2086,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"largest","type":"choice","question_span":[263,270],"option_spans":[[281,291],[297,307],[313,323]],"option_ids":["blue","green","red"],"logits":[-4.875720977783203,-4.947685241699219,-0.4143667221069336],"probs":[0.011294933035969734,0.010510658845305443,0.9781944155693054],"argmax_index":2,"argmax_id":"red","top2_margin":0.9668994825333357,"near_tie":false,"gold":"red","gold_correct":true},{"question_id":"smallest","type":"choice","question_span":[344,351],"option_spans":[[362,372],[378,388],[394,404]],"option_ids":["blue","green","red"],"logits":[-0.7487223148345947,-4.592153072357178,-4.8898773193359375],"probs":[0.9640185832977295,0.020649267360568047,0.015332202427089214],"argmax_index":0,"argmax_id":"blue","top2_margin":0.9433693159371614,"near_tie":false,"gold":"blue","gold_correct":true},{"question_id":"same_size","type":"noul","question_span":[427,434],"option_spans":[[445,463],[469,487]],"option_ids":["true","false"],"logits":[-2.6189701557159424,1.7054522037506104],"probs":[0.01306815817952156,0.9869318604469299],"argmax_index":1,"argmax_id":"false","top2_margin":0.9738637022674084,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"spread","type":"score","question_span":[508,514],"option_spans":[[525,535],[541,553],[559,570]],"option_ids":["0","1","2"],"logits":[-3.7238123416900635,-2.4780335426330566,-1.4577646255493164],"probs":[0.07083727419376373,0.2462048977613449,0.6829577684402466],"argmax_index":2,"argmax_id":"2","top2_margin":0.4367528706789017,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"largest":{"type":"choice","choice":"red","confidence":0.9782,"probabilities":{"blue":0.0113,"red":0.9782,"green":0.0105}},"smallest":{"type":"choice","choice":"blue","confidence":0.964,"probabilities":{"blue":0.964,"red":0.0153,"green":0.0206}},"same_size":{"type":"noul","noul":0.0131},"spread":{"type":"score","score":1.6121,"confidence":0.683,"legend":{"0":"similar","1":"somewhat different","2":"very different"},"probabilities":{"0":0.0708,"1":0.2462,"2":0.683}}},"usage":{"input_tokens":592,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_09","arm":"native","set":"fixture","source":"own_image","tokens":696,"n_image_tokens":300,"state_tokens":4,"token_offset":36,"image_files":["circles_640x480.png"],"image_size_in":[640,480],"image_size_fed":[640,480],"resized_hw":[480,640],"grid_thw":[1,30,40],"merged_hw":[15,20],"rope_shift_amount":280,"pixel_values_shape":[1200,1536],"pixel_values_sha256":"0e8e3c5552a20e22f245dee080dce185276d783abcd4e1bdbb231cf45070f074","image_embeds_shape":[300,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,26487,279,24557,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,7526,198,9222,25,5576,198,669,55530,25,220,3710,1829,369,279,7526,12307,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,10025,2129,1979,816,3147,11855,8934,198,32929,220,17,25,220,4754,4532,3147,18978,2129,1979,816,3147,12865,8934,198,32929,220,18,25,220,4754,4532,3147,5844,2129,1979,816,3147,1114,8934,198,4537,38945,198,198,27806,220,17,198,887,25,23856,198,9222,25,5576,198,669,55530,25,220,3710,1829,369,279,23856,12307,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,10025,2129,1979,816,3147,11855,8934,198,32929,220,17,25,220,4754,4532,3147,18978,2129,1979,816,3147,12865,8934,198,32929,220,18,25,220,4754,4532,3147,5844,2129,1979,816,3147,1114,8934,198,4537,38945,198,198,27806,220,18,198,887,25,1788,2291,198,9222,25,307,10644,198,669,55530,25,220,2324,24557,513,279,1788,1331,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,8597,198,9222,25,5284,198,669,55530,25,220,4199,2086,513,279,11930,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,62116,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,54842,77699,2086,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,1164,2086,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"largest","type":"choice","question_span":[367,374],"option_spans":[[385,395],[401,411],[417,427]],"option_ids":["blue","green","red"],"logits":[-4.907917022705078,-5.0047221183776855,-0.4810889959335327],"probs":[0.011685882695019245,0.010607660748064518,0.9777064323425293],"argmax_index":2,"argmax_id":"red","top2_margin":0.96602054964751,"near_tie":false,"gold":"red","gold_correct":true},{"question_id":"smallest","type":"choice","question_span":[448,455],"option_spans":[[466,476],[482,492],[498,508]],"option_ids":["blue","green","red"],"logits":[-0.8806982040405273,-4.626491069793701,-4.912041664123535],"probs":[0.9602758884429932,0.022678736597299576,0.017045345157384872],"argmax_index":0,"argmax_id":"blue","top2_margin":0.9375971518456936,"near_tie":false,"gold":"blue","gold_correct":true},{"question_id":"same_size","type":"noul","question_span":[531,538],"option_spans":[[549,567],[573,591]],"option_ids":["true","false"],"logits":[-2.574556589126587,1.7182151079177856],"probs":[0.013482722453773022,0.9865173101425171],"argmax_index":1,"argmax_id":"false","top2_margin":0.9730345876887441,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"spread","type":"score","question_span":[612,618],"option_spans":[[629,639],[645,657],[663,674]],"option_ids":["0","1","2"],"logits":[-3.3314170837402344,-2.0668208599090576,-1.2893787622451782],"probs":[0.08164621889591217,0.28916335105895996,0.6291903853416443],"argmax_index":2,"argmax_id":"2","top2_margin":0.3400270342826843,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"largest":{"type":"choice","choice":"red","confidence":0.9777,"probabilities":{"blue":0.0117,"red":0.9777,"green":0.0106}},"smallest":{"type":"choice","choice":"blue","confidence":0.9603,"probabilities":{"blue":0.9603,"red":0.017,"green":0.0227}},"same_size":{"type":"noul","noul":0.0135},"spread":{"type":"score","score":1.5475,"confidence":0.6292,"legend":{"0":"similar","1":"somewhat different","2":"very different"},"probabilities":{"0":0.0816,"1":0.2892,"2":0.6292}}},"usage":{"input_tokens":696,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_10","arm":"g256","set":"fixture","source":"own_image","tokens":405,"n_image_tokens":64,"state_tokens":8,"token_offset":36,"image_files":["clock_3oclock_256x240.png"],"image_size_in":[256,240],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"33a9636b492ae80ac964bb11fc549a76f3eb6f313b59616278d03d15fc6cdeb6","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,760,12037,2099,4774,264,6784,8598,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,6258,198,9222,25,5576,198,669,55530,25,220,3710,6258,1503,279,8598,1420,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,15103,20559,2129,1979,816,3147,16,17,8934,198,32929,220,17,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,18,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,32929,220,19,25,220,4754,4532,3147,90012,2129,1979,816,3147,24,8934,198,4537,38945,198,198,27806,220,17,198,887,25,9104,22470,7980,198,9222,25,307,10644,198,669,55530,25,220,760,9104,1375,3387,7435,685,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,11791,198,9222,25,5284,198,669,55530,25,220,4199,9077,628,279,854,381,1301,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1299,1493,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,39551,391,2708,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,1164,2708,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"hour","type":"choice","question_span":[135,142],"option_spans":[[153,165],[171,181],[187,197],[203,213]],"option_ids":["12","3","6","9"],"logits":[-4.363167762756348,-0.2576790452003479,-5.0719146728515625,-4.807971477508545],"probs":[0.015922145918011665,0.9660347104072571,0.00783784594386816,0.010205300524830818],"argmax_index":1,"argmax_id":"3","top2_margin":0.9501125644892454,"near_tie":false,"gold":"3","gold_correct":true},{"question_id":"minute_hand_up","type":"noul","question_span":[237,244],"option_spans":[[255,273],[279,297]],"option_ids":["true","false"],"logits":[-1.2069377899169922,-4.117978096008301],"probs":[0.9483895301818848,0.051610492169857025],"argmax_index":0,"argmax_id":"true","top2_margin":0.8967790380120277,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"confidence","type":"score","question_span":[318,326],"option_spans":[[337,348],[354,366],[372,383]],"option_ids":["0","1","2"],"logits":[-3.6747701168060303,-1.9879026412963867,0.12337544560432434],"probs":[0.01959981396794319,0.10588859021663666,0.8745115995407104],"argmax_index":2,"argmax_id":"2","top2_margin":0.7686230093240738,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"hour":{"type":"choice","choice":"3","confidence":0.966,"probabilities":{"12":0.0159,"3":0.966,"6":0.0078,"9":0.0102}},"minute_hand_up":{"type":"noul","noul":0.9484},"confidence":{"type":"score","score":1.8549,"confidence":0.8745,"legend":{"0":"unclear","1":"fairly clear","2":"very clear"},"probabilities":{"0":0.0196,"1":0.1059,"2":0.8745}}},"usage":{"input_tokens":405,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_10","arm":"g448","set":"fixture","source":"own_image","tokens":537,"n_image_tokens":196,"state_tokens":8,"token_offset":36,"image_files":["clock_3oclock_256x240.png"],"image_size_in":[256,240],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"76eeaaacd5fc57f149972e42d3e822df242f96461747c683e3318e426956336e","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,760,12037,2099,4774,264,6784,8598,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,6258,198,9222,25,5576,198,669,55530,25,220,3710,6258,1503,279,8598,1420,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,15103,20559,2129,1979,816,3147,16,17,8934,198,32929,220,17,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,18,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,32929,220,19,25,220,4754,4532,3147,90012,2129,1979,816,3147,24,8934,198,4537,38945,198,198,27806,220,17,198,887,25,9104,22470,7980,198,9222,25,307,10644,198,669,55530,25,220,760,9104,1375,3387,7435,685,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,11791,198,9222,25,5284,198,669,55530,25,220,4199,9077,628,279,854,381,1301,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1299,1493,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,39551,391,2708,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,1164,2708,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"hour","type":"choice","question_span":[267,274],"option_spans":[[285,297],[303,313],[319,329],[335,345]],"option_ids":["12","3","6","9"],"logits":[-2.9581761360168457,0.9523033499717712,-4.615214824676514,-3.8521289825439453],"probs":[0.019408948719501495,0.9689507484436035,0.0037013425026088953,0.00793895311653614],"argmax_index":1,"argmax_id":"3","top2_margin":0.949541799724102,"near_tie":false,"gold":"3","gold_correct":true},{"question_id":"minute_hand_up","type":"noul","question_span":[369,376],"option_spans":[[387,405],[411,429]],"option_ids":["true","false"],"logits":[-0.5792127251625061,-3.5911097526550293],"probs":[0.9531087279319763,0.046891286969184875],"argmax_index":0,"argmax_id":"true","top2_margin":0.9062174409627914,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"confidence","type":"score","question_span":[450,458],"option_spans":[[469,480],[486,498],[504,515]],"option_ids":["0","1","2"],"logits":[-3.4698781967163086,-1.6298296451568604,0.5980119109153748],"probs":[0.015213661827147007,0.095798060297966,0.8889882564544678],"argmax_index":2,"argmax_id":"2","top2_margin":0.7931901961565018,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"hour":{"type":"choice","choice":"3","confidence":0.969,"probabilities":{"12":0.0194,"3":0.969,"6":0.0037,"9":0.0079}},"minute_hand_up":{"type":"noul","noul":0.9531},"confidence":{"type":"score","score":1.8738,"confidence":0.889,"legend":{"0":"unclear","1":"fairly clear","2":"very clear"},"probabilities":{"0":0.0152,"1":0.0958,"2":0.889}}},"usage":{"input_tokens":537,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"img_10","arm":"native","set":"fixture","source":"own_image","tokens":405,"n_image_tokens":64,"state_tokens":8,"token_offset":36,"image_files":["clock_3oclock_256x240.png"],"image_size_in":[256,240],"image_size_fed":[256,240],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"a0d9464bbcde0c84ea3c349e7601a37ac259cb8900109318611ff99cf3e3d5d4","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,760,12037,2099,4774,264,6784,8598,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,6258,198,9222,25,5576,198,669,55530,25,220,3710,6258,1503,279,8598,1420,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,15103,20559,2129,1979,816,3147,16,17,8934,198,32929,220,17,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,18,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,32929,220,19,25,220,4754,4532,3147,90012,2129,1979,816,3147,24,8934,198,4537,38945,198,198,27806,220,17,198,887,25,9104,22470,7980,198,9222,25,307,10644,198,669,55530,25,220,760,9104,1375,3387,7435,685,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,11791,198,9222,25,5284,198,669,55530,25,220,4199,9077,628,279,854,381,1301,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1299,1493,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,39551,391,2708,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,1164,2708,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"hour","type":"choice","question_span":[135,142],"option_spans":[[153,165],[171,181],[187,197],[203,213]],"option_ids":["12","3","6","9"],"logits":[-4.361988544464111,-0.25623369216918945,-5.071813583374023,-4.807557582855225],"probs":[0.0159183107316494,0.9660590887069702,0.00782751478254795,0.01019503828138113],"argmax_index":1,"argmax_id":"3","top2_margin":0.9501407779753208,"near_tie":false,"gold":"3","gold_correct":true},{"question_id":"minute_hand_up","type":"noul","question_span":[237,244],"option_spans":[[255,273],[279,297]],"option_ids":["true","false"],"logits":[-1.2058193683624268,-4.1172566413879395],"probs":[0.9484089612960815,0.051591068506240845],"argmax_index":0,"argmax_id":"true","top2_margin":0.8968178927898407,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"confidence","type":"score","question_span":[318,326],"option_spans":[[337,348],[354,366],[372,383]],"option_ids":["0","1","2"],"logits":[-3.673570156097412,-1.9855611324310303,0.1257806122303009],"probs":[0.019576799124479294,0.10588505864143372,0.8745381236076355],"argmax_index":2,"argmax_id":"2","top2_margin":0.7686530649662018,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"hour":{"type":"choice","choice":"3","confidence":0.9661,"probabilities":{"12":0.0159,"3":0.9661,"6":0.0078,"9":0.0102}},"minute_hand_up":{"type":"noul","noul":0.9484},"confidence":{"type":"score","score":1.855,"confidence":0.8745,"legend":{"0":"unclear","1":"fairly clear","2":"very clear"},"probabilities":{"0":0.0196,"1":0.1059,"2":0.8745}}},"usage":{"input_tokens":405,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"photo_01","arm":"g256","set":"fixture","source":"cc0_photo","tokens":556,"n_image_tokens":64,"state_tokens":13,"token_offset":36,"image_files":["photo_fruit.png"],"image_size_in":[960,1280],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"32438e7f47f8c6053bca1ed753946a52d6b9f06de5bd8c5fc76b6870e6292783","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,7945,3147,3840,279,13383,18632,303,279,12037,6345,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,324,1672,198,9222,25,5576,198,669,55530,25,220,4199,1599,324,1672,513,303,279,18632,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,20,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,21,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,17,198,887,25,682,853,276,3253,198,9222,25,307,10644,198,669,55530,25,220,3733,369,264,41674,303,279,18632,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,12461,198,9222,25,5576,198,669,55530,25,220,4199,1599,2086,12461,314,13383,513,1017,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,17,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,18,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,19,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,20,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,19,198,887,25,94827,198,9222,25,5284,198,669,55530,25,220,4199,7478,1503,279,13383,1353,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,79999,2112,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,35788,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,68887,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"limes","type":"choice","question_span":[141,150],"option_spans":[[161,171],[177,187],[193,203],[209,219],[225,235],[241,251]],"option_ids":["1","2","3","4","5","6"],"logits":[-4.269002437591553,-3.5498600006103516,-1.2577426433563232,-0.602066159248352,-2.4343507289886475,-3.4118099212646484],"probs":[0.014061340130865574,0.028863323852419853,0.28562769293785095,0.5502461194992065,0.08806552737951279,0.033136047422885895],"argmax_index":3,"argmax_id":"4","top2_margin":0.2646184265613556,"near_tie":false,"gold":"3","gold_correct":false},{"question_id":"has_banana","type":"noul","question_span":[276,284],"option_spans":[[295,313],[319,337]],"option_ids":["true","false"],"logits":[-3.292261838912964,2.0979056358337402],"probs":[0.004540500231087208,0.9954594969749451],"argmax_index":1,"argmax_id":"false","top2_margin":0.9909189967438579,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"kinds","type":"choice","question_span":[358,367],"option_spans":[[378,388],[394,404],[410,420],[426,436],[442,452]],"option_ids":["2","3","4","5","6"],"logits":[-3.6479969024658203,-2.064840078353882,-0.8699471950531006,-0.8463967442512512,-2.3221044540405273],"probs":[0.023699797689914703,0.11542528122663498,0.3812732696533203,0.39035898447036743,0.08924264460802078],"argmax_index":3,"argmax_id":"5","top2_margin":0.00908571481704712,"near_tie":true,"gold":"4","gold_correct":false},{"question_id":"freshness","type":"score","question_span":[473,480],"option_spans":[[491,502],[508,518],[524,534]],"option_ids":["0","1","2"],"logits":[-3.558563709259033,-0.5118257999420166,0.7997377514839172],"probs":[0.009982937015593052,0.21010668575763702,0.7799103260040283],"argmax_index":2,"argmax_id":"2","top2_margin":0.5698036402463913,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"limes":{"type":"choice","choice":"4","confidence":0.5502,"probabilities":{"1":0.0141,"2":0.0289,"3":0.2856,"4":0.5502,"5":0.0881,"6":0.0331}},"has_banana":{"type":"noul","noul":0.0045},"kinds":{"type":"choice","choice":"5","confidence":0.3904,"probabilities":{"2":0.0237,"3":0.1154,"4":0.3813,"5":0.3904,"6":0.0892}},"freshness":{"type":"score","score":1.7699,"confidence":0.7799,"legend":{"0":"spoiled","1":"acceptable","2":"fresh"},"probabilities":{"0":0.01,"1":0.2101,"2":0.7799}}},"usage":{"input_tokens":556,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"photo_01","arm":"g448","set":"fixture","source":"cc0_photo","tokens":688,"n_image_tokens":196,"state_tokens":13,"token_offset":36,"image_files":["photo_fruit.png"],"image_size_in":[960,1280],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"8c8858aa751aa5443cb2d49500855d4a19bbd6507551b7b00622bf52499e8c24","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,7945,3147,3840,279,13383,18632,303,279,12037,6345,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,324,1672,198,9222,25,5576,198,669,55530,25,220,4199,1599,324,1672,513,303,279,18632,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,20,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,21,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,17,198,887,25,682,853,276,3253,198,9222,25,307,10644,198,669,55530,25,220,3733,369,264,41674,303,279,18632,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,12461,198,9222,25,5576,198,669,55530,25,220,4199,1599,2086,12461,314,13383,513,1017,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,17,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,18,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,19,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,20,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,19,198,887,25,94827,198,9222,25,5284,198,669,55530,25,220,4199,7478,1503,279,13383,1353,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,79999,2112,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,35788,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,68887,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"limes","type":"choice","question_span":[273,282],"option_spans":[[293,303],[309,319],[325,335],[341,351],[357,367],[373,383]],"option_ids":["1","2","3","4","5","6"],"logits":[-4.210570812225342,-3.1271424293518066,-0.829371988773346,-0.6987099051475525,-2.791368007659912,-3.6267971992492676],"probs":[0.013736645691096783,0.04058893397450447,0.403939813375473,0.4603227972984314,0.056784871965646744,0.02462693303823471],"argmax_index":3,"argmax_id":"4","top2_margin":0.056382983922958374,"near_tie":false,"gold":"3","gold_correct":false},{"question_id":"has_banana","type":"noul","question_span":[408,416],"option_spans":[[427,445],[451,469]],"option_ids":["true","false"],"logits":[-3.545872449874878,2.023212432861328],"probs":[0.003799476893618703,0.9962005019187927],"argmax_index":1,"argmax_id":"false","top2_margin":0.992401025025174,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"kinds","type":"choice","question_span":[490,499],"option_spans":[[510,520],[526,536],[542,552],[558,568],[574,584]],"option_ids":["2","3","4","5","6"],"logits":[-3.609628438949585,-1.848745346069336,-0.5697962045669556,-1.1877957582473755,-2.746285915374756],"probs":[0.02417978085577488,0.1406676173210144,0.5053993463516235,0.2724212110042572,0.057332009077072144],"argmax_index":2,"argmax_id":"4","top2_margin":0.23297813534736633,"near_tie":false,"gold":"4","gold_correct":true},{"question_id":"freshness","type":"score","question_span":[605,612],"option_spans":[[623,634],[640,650],[656,666]],"option_ids":["0","1","2"],"logits":[-3.7598350048065186,-0.8429338335990906,0.7358361482620239],"probs":[0.009164875373244286,0.1694028377532959,0.8214322924613953],"argmax_index":2,"argmax_id":"2","top2_margin":0.6520294547080994,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"limes":{"type":"choice","choice":"4","confidence":0.4603,"probabilities":{"1":0.0137,"2":0.0406,"3":0.4039,"4":0.4603,"5":0.0568,"6":0.0246}},"has_banana":{"type":"noul","noul":0.0038},"kinds":{"type":"choice","choice":"4","confidence":0.5054,"probabilities":{"2":0.0242,"3":0.1407,"4":0.5054,"5":0.2724,"6":0.0573}},"freshness":{"type":"score","score":1.8123,"confidence":0.8214,"legend":{"0":"spoiled","1":"acceptable","2":"fresh"},"probabilities":{"0":0.0092,"1":0.1694,"2":0.8214}}},"usage":{"input_tokens":688,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"photo_01","arm":"native","set":"fixture","source":"cc0_photo","tokens":1692,"n_image_tokens":1200,"state_tokens":13,"token_offset":36,"image_files":["photo_fruit.png"],"image_size_in":[960,1280],"image_size_fed":[960,1280],"resized_hw":[1280,960],"grid_thw":[1,80,60],"merged_hw":[40,30],"rope_shift_amount":1160,"pixel_values_shape":[4800,1536],"pixel_values_sha256":"0725fc1e2d408bd015e5dd2cc15e2d634f06ebffb7bac9fe5bab1be2b89d092c","image_embeds_shape":[1200,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,7945,3147,3840,279,13383,18632,303,279,12037,6345,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,324,1672,198,9222,25,5576,198,669,55530,25,220,4199,1599,324,1672,513,303,279,18632,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,20,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,21,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,17,198,887,25,682,853,276,3253,198,9222,25,307,10644,198,669,55530,25,220,3733,369,264,41674,303,279,18632,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,12461,198,9222,25,5576,198,669,55530,25,220,4199,1599,2086,12461,314,13383,513,1017,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,17,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,18,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,19,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,20,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,19,198,887,25,94827,198,9222,25,5284,198,669,55530,25,220,4199,7478,1503,279,13383,1353,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,79999,2112,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,35788,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,68887,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":false,"excluded_why":"1,200 image rows (40 x 30 merged) exceed the decoder graph's 1,024-row image buffer; not in any decoder gate (oracle only)","questions":[{"question_id":"limes","type":"choice","question_span":[1277,1286],"option_spans":[[1297,1307],[1313,1323],[1329,1339],[1345,1355],[1361,1371],[1377,1387]],"option_ids":["1","2","3","4","5","6"],"logits":[-4.336813449859619,-3.5668997764587402,-0.9083899259567261,-0.5581082105636597,-2.8696959018707275,-3.713104009628296],"probs":[0.011911926791071892,0.025724755600094795,0.3672208786010742,0.5212580561637878,0.05165863782167435,0.022225702181458473],"argmax_index":3,"argmax_id":"4","top2_margin":0.15403717756271362,"near_tie":false,"gold":"3","gold_correct":false},{"question_id":"has_banana","type":"noul","question_span":[1412,1420],"option_spans":[[1431,1449],[1455,1473]],"option_ids":["true","false"],"logits":[-3.5593929290771484,2.073511838912964],"probs":[0.0035654078237712383,0.9964345693588257],"argmax_index":1,"argmax_id":"false","top2_margin":0.9928691615350544,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"kinds","type":"choice","question_span":[1494,1503],"option_spans":[[1514,1524],[1530,1540],[1546,1556],[1562,1572],[1578,1588]],"option_ids":["2","3","4","5","6"],"logits":[-3.992788791656494,-2.2860426902770996,-0.6290273666381836,-1.7702744007110596,-3.2984838485717773],"probs":[0.021440129727125168,0.1181565523147583,0.6195700764656067,0.1979033499956131,0.042929936200380325],"argmax_index":2,"argmax_id":"4","top2_margin":0.4216667264699936,"near_tie":false,"gold":"4","gold_correct":true},{"question_id":"freshness","type":"score","question_span":[1609,1616],"option_spans":[[1627,1638],[1644,1654],[1660,1670]],"option_ids":["0","1","2"],"logits":[-3.91516375541687,-0.9899078607559204,0.78485506772995],"probs":[0.007716749329119921,0.14383246004581451,0.848450779914856],"argmax_index":2,"argmax_id":"2","top2_margin":0.7046183198690414,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"limes":{"type":"choice","choice":"4","confidence":0.5213,"probabilities":{"1":0.0119,"2":0.0257,"3":0.3672,"4":0.5213,"5":0.0517,"6":0.0222}},"has_banana":{"type":"noul","noul":0.0036},"kinds":{"type":"choice","choice":"4","confidence":0.6196,"probabilities":{"2":0.0214,"3":0.1182,"4":0.6196,"5":0.1979,"6":0.0429}},"freshness":{"type":"score","score":1.8407,"confidence":0.8485,"legend":{"0":"spoiled","1":"acceptable","2":"fresh"},"probabilities":{"0":0.0077,"1":0.1438,"2":0.8485}}},"usage":{"input_tokens":1692,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"photo_02","arm":"g256","set":"fixture","source":"cc0_photo","tokens":499,"n_image_tokens":64,"state_tokens":9,"token_offset":36,"image_files":["photo_cat.png"],"image_size_in":[960,640],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"db031114e4a4f365510c0101d8b0e2ca0982611c0f86a3b7fa104a7629440990","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,27382,1386,279,9572,303,279,12037,6345,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,9572,198,9222,25,5576,198,669,55530,25,220,22365,9572,369,6625,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,32,7993,2129,1979,816,3147,4466,8934,198,32929,220,17,25,220,4754,4532,3147,32,5388,2129,1979,816,3147,17907,8934,198,32929,220,18,25,220,4754,4532,3147,32,37550,2129,1979,816,3147,14576,8934,198,32929,220,19,25,220,4754,4532,3147,32,37442,2129,1979,816,3147,83362,8934,198,4537,38945,198,198,27806,220,17,198,887,25,7662,6504,198,9222,25,5576,198,669,55530,25,220,3710,1829,513,1141,6213,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,10025,2129,1979,816,3147,11855,8934,198,32929,220,17,25,220,4754,4532,3147,46776,2129,1979,816,3147,62284,8934,198,32929,220,18,25,220,4754,4532,3147,18978,2129,1979,816,3147,12865,8934,198,32929,220,19,25,220,4754,4532,3147,46110,466,65326,2129,1979,816,3147,26964,8934,198,4537,38945,198,198,27806,220,18,198,887,25,3870,198,9222,25,5284,198,669,55530,25,220,4199,12655,369,279,3870,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,20139,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,54842,77699,12655,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,75778,2129,1979,816,3147,17,8934,198,4537,38945,198,198,27806,220,19,198,887,25,65311,198,9222,25,307,10644,198,669,55530,25,220,760,9572,579,17699,682,52850,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"animal","type":"choice","question_span":[136,141],"option_spans":[[152,163],[169,180],[186,197],[203,214]],"option_ids":["cat","dog","fox","rabbit"],"logits":[-0.5377101302146912,-4.966231346130371,-4.2339348793029785,-4.991822242736816],"probs":[0.9538528323173523,0.011381489224731922,0.02367180585861206,0.011093921959400177],"argmax_index":0,"argmax_id":"cat","top2_margin":0.9301810264587402,"near_tie":false,"gold":"cat","gold_correct":true},{"question_id":"eye_color","type":"choice","question_span":[236,242],"option_spans":[[253,263],[269,279],[285,295],[301,313]],"option_ids":["blue","brown","green","yellow"],"logits":[-5.038334369659424,-4.717514991760254,-4.409954071044922,-0.34385067224502563],"probs":[0.008803190663456917,0.012133055366575718,0.01650221273303032,0.9625614881515503],"argmax_index":3,"argmax_id":"yellow","top2_margin":0.94605927541852,"near_tie":false,"gold":"yellow","gold_correct":true},{"question_id":"background","type":"score","question_span":[334,340],"option_spans":[[351,361],[367,379],[385,395]],"option_ids":["0","1","2"],"logits":[-1.2122198343276978,-2.522596597671509,-3.601806163787842],"probs":[0.7345455884933472,0.19812045991420746,0.06733401864767075],"argmax_index":0,"argmax_id":"0","top2_margin":0.5364251285791397,"near_tie":false,"gold":null,"gold_correct":null},{"question_id":"striped","type":"noul","question_span":[417,424],"option_spans":[[435,453],[459,477]],"option_ids":["true","false"],"logits":[-1.7583361864089966,-4.289011478424072],"probs":[0.9262645244598389,0.07373550534248352],"argmax_index":0,"argmax_id":"true","top2_margin":0.8525290191173553,"near_tie":false,"gold":"true","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"animal":{"type":"choice","choice":"cat","confidence":0.9539,"probabilities":{"cat":0.9539,"dog":0.0114,"rabbit":0.0111,"fox":0.0237}},"eye_color":{"type":"choice","choice":"yellow","confidence":0.9626,"probabilities":{"blue":0.0088,"green":0.0165,"yellow":0.9626,"brown":0.0121}},"background":{"type":"score","score":0.3328,"confidence":0.7345,"legend":{"0":"plain","1":"somewhat busy","2":"busy"},"probabilities":{"0":0.7345,"1":0.1981,"2":0.0673}},"striped":{"type":"noul","noul":0.9263}},"usage":{"input_tokens":499,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"photo_02","arm":"g448","set":"fixture","source":"cc0_photo","tokens":631,"n_image_tokens":196,"state_tokens":9,"token_offset":36,"image_files":["photo_cat.png"],"image_size_in":[960,640],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"181662b11d55b465bc9f0825dc24762f1d7c322fd2cd8a1068dfdf7c55b3ed4a","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,27382,1386,279,9572,303,279,12037,6345,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,9572,198,9222,25,5576,198,669,55530,25,220,22365,9572,369,6625,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,32,7993,2129,1979,816,3147,4466,8934,198,32929,220,17,25,220,4754,4532,3147,32,5388,2129,1979,816,3147,17907,8934,198,32929,220,18,25,220,4754,4532,3147,32,37550,2129,1979,816,3147,14576,8934,198,32929,220,19,25,220,4754,4532,3147,32,37442,2129,1979,816,3147,83362,8934,198,4537,38945,198,198,27806,220,17,198,887,25,7662,6504,198,9222,25,5576,198,669,55530,25,220,3710,1829,513,1141,6213,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,10025,2129,1979,816,3147,11855,8934,198,32929,220,17,25,220,4754,4532,3147,46776,2129,1979,816,3147,62284,8934,198,32929,220,18,25,220,4754,4532,3147,18978,2129,1979,816,3147,12865,8934,198,32929,220,19,25,220,4754,4532,3147,46110,466,65326,2129,1979,816,3147,26964,8934,198,4537,38945,198,198,27806,220,18,198,887,25,3870,198,9222,25,5284,198,669,55530,25,220,4199,12655,369,279,3870,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,20139,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,54842,77699,12655,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,75778,2129,1979,816,3147,17,8934,198,4537,38945,198,198,27806,220,19,198,887,25,65311,198,9222,25,307,10644,198,669,55530,25,220,760,9572,579,17699,682,52850,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"animal","type":"choice","question_span":[268,273],"option_spans":[[284,295],[301,312],[318,329],[335,346]],"option_ids":["cat","dog","fox","rabbit"],"logits":[-0.45860207080841064,-4.990456581115723,-4.444618225097656,-5.007287979125977],"probs":[0.9616167545318604,0.010347669944167137,0.017860645428299904,0.010174961760640144],"argmax_index":0,"argmax_id":"cat","top2_margin":0.9437561091035604,"near_tie":false,"gold":"cat","gold_correct":true},{"question_id":"eye_color","type":"choice","question_span":[368,374],"option_spans":[[385,395],[401,411],[417,427],[433,445]],"option_ids":["blue","brown","green","yellow"],"logits":[-5.1110920906066895,-4.570823669433594,-4.009984493255615,-0.19795376062393188],"probs":[0.00705269630998373,0.012105724774301052,0.02121095173060894,0.9596306085586548],"argmax_index":3,"argmax_id":"yellow","top2_margin":0.9384196568280458,"near_tie":false,"gold":"yellow","gold_correct":true},{"question_id":"background","type":"score","question_span":[466,472],"option_spans":[[483,493],[499,511],[517,527]],"option_ids":["0","1","2"],"logits":[-1.17232084274292,-2.560537099838257,-3.63852596282959],"probs":[0.7493856549263,0.18698669970035553,0.0636276826262474],"argmax_index":0,"argmax_id":"0","top2_margin":0.5623989552259445,"near_tie":false,"gold":null,"gold_correct":null},{"question_id":"striped","type":"noul","question_span":[549,556],"option_spans":[[567,585],[591,609]],"option_ids":["true","false"],"logits":[-1.7425130605697632,-4.358412265777588],"probs":[0.931877851486206,0.06812216341495514],"argmax_index":0,"argmax_id":"true","top2_margin":0.8637556880712509,"near_tie":false,"gold":"true","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"animal":{"type":"choice","choice":"cat","confidence":0.9616,"probabilities":{"cat":0.9616,"dog":0.0103,"rabbit":0.0102,"fox":0.0179}},"eye_color":{"type":"choice","choice":"yellow","confidence":0.9596,"probabilities":{"blue":0.0071,"green":0.0212,"yellow":0.9596,"brown":0.0121}},"background":{"type":"score","score":0.3142,"confidence":0.7494,"legend":{"0":"plain","1":"somewhat busy","2":"busy"},"probabilities":{"0":0.7494,"1":0.187,"2":0.0636}},"striped":{"type":"noul","noul":0.9319}},"usage":{"input_tokens":631,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"photo_02","arm":"native","set":"fixture","source":"cc0_photo","tokens":1035,"n_image_tokens":600,"state_tokens":9,"token_offset":36,"image_files":["photo_cat.png"],"image_size_in":[960,640],"image_size_fed":[960,640],"resized_hw":[640,960],"grid_thw":[1,40,60],"merged_hw":[20,30],"rope_shift_amount":570,"pixel_values_shape":[2400,1536],"pixel_values_sha256":"7ca47c9e0393ab1bea2a232db27271fb77816ea64fd33accfb55d367fc612323","image_embeds_shape":[600,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,27382,1386,279,9572,303,279,12037,6345,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,9572,198,9222,25,5576,198,669,55530,25,220,22365,9572,369,6625,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,32,7993,2129,1979,816,3147,4466,8934,198,32929,220,17,25,220,4754,4532,3147,32,5388,2129,1979,816,3147,17907,8934,198,32929,220,18,25,220,4754,4532,3147,32,37550,2129,1979,816,3147,14576,8934,198,32929,220,19,25,220,4754,4532,3147,32,37442,2129,1979,816,3147,83362,8934,198,4537,38945,198,198,27806,220,17,198,887,25,7662,6504,198,9222,25,5576,198,669,55530,25,220,3710,1829,513,1141,6213,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,10025,2129,1979,816,3147,11855,8934,198,32929,220,17,25,220,4754,4532,3147,46776,2129,1979,816,3147,62284,8934,198,32929,220,18,25,220,4754,4532,3147,18978,2129,1979,816,3147,12865,8934,198,32929,220,19,25,220,4754,4532,3147,46110,466,65326,2129,1979,816,3147,26964,8934,198,4537,38945,198,198,27806,220,18,198,887,25,3870,198,9222,25,5284,198,669,55530,25,220,4199,12655,369,279,3870,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,20139,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,54842,77699,12655,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,75778,2129,1979,816,3147,17,8934,198,4537,38945,198,198,27806,220,19,198,887,25,65311,198,9222,25,307,10644,198,669,55530,25,220,760,9572,579,17699,682,52850,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"animal","type":"choice","question_span":[672,677],"option_spans":[[688,699],[705,716],[722,733],[739,750]],"option_ids":["cat","dog","fox","rabbit"],"logits":[-0.31955432891845703,-4.984421730041504,-4.394810199737549,-5.024158477783203],"probs":[0.9657526016235352,0.009097869507968426,0.01640607789158821,0.008743438869714737],"argmax_index":0,"argmax_id":"cat","top2_margin":0.949346523731947,"near_tie":false,"gold":"cat","gold_correct":true},{"question_id":"eye_color","type":"choice","question_span":[772,778],"option_spans":[[789,799],[805,815],[821,831],[837,849]],"option_ids":["blue","brown","green","yellow"],"logits":[-5.109397888183594,-4.427114486694336,-3.8244681358337402,-0.15661633014678955],"probs":[0.006749413441866636,0.013352972455322742,0.024395175278186798,0.9555023908615112],"argmax_index":3,"argmax_id":"yellow","top2_margin":0.9311072155833244,"near_tie":false,"gold":"yellow","gold_correct":true},{"question_id":"background","type":"score","question_span":[870,876],"option_spans":[[887,897],[903,915],[921,931]],"option_ids":["0","1","2"],"logits":[-1.0357372760772705,-2.586081027984619,-3.6802170276641846],"probs":[0.7792912125587463,0.16534613072872162,0.05536263808608055],"argmax_index":0,"argmax_id":"0","top2_margin":0.6139450818300247,"near_tie":false,"gold":null,"gold_correct":null},{"question_id":"striped","type":"noul","question_span":[953,960],"option_spans":[[971,989],[995,1013]],"option_ids":["true","false"],"logits":[-1.7174402475357056,-4.383325099945068],"probs":[0.9349833726882935,0.06501667946577072],"argmax_index":0,"argmax_id":"true","top2_margin":0.8699666932225227,"near_tie":false,"gold":"true","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"animal":{"type":"choice","choice":"cat","confidence":0.9658,"probabilities":{"cat":0.9658,"dog":0.0091,"rabbit":0.0087,"fox":0.0164}},"eye_color":{"type":"choice","choice":"yellow","confidence":0.9555,"probabilities":{"blue":0.0067,"green":0.0244,"yellow":0.9555,"brown":0.0134}},"background":{"type":"score","score":0.2761,"confidence":0.7793,"legend":{"0":"plain","1":"somewhat busy","2":"busy"},"probabilities":{"0":0.7793,"1":0.1653,"2":0.0554}},"striped":{"type":"noul","noul":0.935}},"usage":{"input_tokens":1035,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"photo_03","arm":"g256","set":"fixture","source":"cc0_photo","tokens":505,"n_image_tokens":64,"state_tokens":12,"token_offset":36,"image_files":["photo_lake.png"],"image_size_in":[960,640],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"76239c69ce005d792a06e323418c3bc343b94de43d7faafcb648c56d5ca1e3af","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,7945,3147,15666,4602,883,279,12037,17865,6345,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,11456,33229,198,9222,25,307,10644,198,669,55530,25,220,60187,369,9155,303,279,6345,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,21172,198,9222,25,307,10644,198,669,55530,25,220,760,2919,25133,279,17865,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,12515,198,9222,25,5576,198,669,55530,25,220,3710,369,279,12515,1040,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,13606,321,38383,2129,1979,816,3147,7187,8934,198,32929,220,17,25,220,4754,4532,3147,48739,2129,1979,816,3147,9011,8934,198,32929,220,18,25,220,4754,4532,3147,1853,3701,2129,1979,816,3147,1908,3701,8934,198,32929,220,19,25,220,4754,4532,3147,49,1988,2129,1979,816,3147,28142,8934,198,4537,38945,198,198,27806,220,19,198,887,25,8121,2022,198,9222,25,5284,198,669,55530,25,220,4199,8121,369,279,1925,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,57062,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,1899,57062,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,81,4038,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,65584,27636,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"snow_visible","type":"noul","question_span":[141,148],"option_spans":[[159,177],[183,201]],"option_ids":["true","false"],"logits":[-2.0455543994903564,-4.485628128051758],"probs":[0.9198325276374817,0.08016747236251831],"argmax_index":0,"argmax_id":"true","top2_margin":0.8396650552749634,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"reflection","type":"noul","question_span":[223,229],"option_spans":[[240,258],[264,282]],"option_ids":["true","false"],"logits":[-1.475717306137085,-4.458216667175293],"probs":[0.9517771601676941,0.04822278022766113],"argmax_index":0,"argmax_id":"true","top2_margin":0.903554379940033,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"sky","type":"choice","question_span":[303,309],"option_spans":[[320,332],[338,348],[354,366],[372,383]],"option_ids":["clear","night","overcast","rain"],"logits":[-4.785508632659912,-4.588129997253418,-1.0164293050765991,-4.825441837310791],"probs":[0.021496502682566643,0.026187149807810783,0.9316613674163818,0.020654991269111633],"argmax_index":2,"argmax_id":"overcast","top2_margin":0.905474217608571,"near_tie":false,"gold":"overcast","gold_correct":true},{"question_id":"wildness","type":"score","question_span":[405,411],"option_spans":[[422,432],[438,449],[455,466],[472,483]],"option_ids":["0","1","2","3"],"logits":[-5.090063095092773,-5.039792537689209,-3.500098943710327,-0.7843558192253113],"probs":[0.012333991006016731,0.012969876639544964,0.06048063933849335,0.9142155647277832],"argmax_index":3,"argmax_id":"3","top2_margin":0.8537349253892899,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"snow_visible":{"type":"noul","noul":0.9198},"reflection":{"type":"noul","noul":0.9518},"sky":{"type":"choice","choice":"overcast","confidence":0.9317,"probabilities":{"clear":0.0215,"overcast":0.9317,"rain":0.0207,"night":0.0262}},"wildness":{"type":"score","score":2.8766,"confidence":0.9142,"legend":{"0":"urban","1":"suburban","2":"rural","3":"wilderness"},"probabilities":{"0":0.0123,"1":0.013,"2":0.0605,"3":0.9142}}},"usage":{"input_tokens":505,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"photo_03","arm":"g448","set":"fixture","source":"cc0_photo","tokens":637,"n_image_tokens":196,"state_tokens":12,"token_offset":36,"image_files":["photo_lake.png"],"image_size_in":[960,640],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"dc7bc0191f13a7358f975b7512fb56a098b2cce4b3fe5e2701f1b48856fe32d5","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,7945,3147,15666,4602,883,279,12037,17865,6345,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,11456,33229,198,9222,25,307,10644,198,669,55530,25,220,60187,369,9155,303,279,6345,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,21172,198,9222,25,307,10644,198,669,55530,25,220,760,2919,25133,279,17865,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,12515,198,9222,25,5576,198,669,55530,25,220,3710,369,279,12515,1040,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,13606,321,38383,2129,1979,816,3147,7187,8934,198,32929,220,17,25,220,4754,4532,3147,48739,2129,1979,816,3147,9011,8934,198,32929,220,18,25,220,4754,4532,3147,1853,3701,2129,1979,816,3147,1908,3701,8934,198,32929,220,19,25,220,4754,4532,3147,49,1988,2129,1979,816,3147,28142,8934,198,4537,38945,198,198,27806,220,19,198,887,25,8121,2022,198,9222,25,5284,198,669,55530,25,220,4199,8121,369,279,1925,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,57062,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,1899,57062,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,81,4038,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,65584,27636,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"snow_visible","type":"noul","question_span":[273,280],"option_spans":[[291,309],[315,333]],"option_ids":["true","false"],"logits":[-2.135958194732666,-4.59937858581543],"probs":[0.9215373396873474,0.07846266776323318],"argmax_index":0,"argmax_id":"true","top2_margin":0.8430746719241142,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"reflection","type":"noul","question_span":[355,361],"option_spans":[[372,390],[396,414]],"option_ids":["true","false"],"logits":[-1.604405403137207,-4.552488327026367],"probs":[0.9501727819442749,0.04982719570398331],"argmax_index":0,"argmax_id":"true","top2_margin":0.9003455862402916,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"sky","type":"choice","question_span":[435,441],"option_spans":[[452,464],[470,480],[486,498],[504,515]],"option_ids":["clear","night","overcast","rain"],"logits":[-4.8338623046875,-4.627209186553955,-0.970078706741333,-4.866893291473389],"probs":[0.019668642431497574,0.024183696135878563,0.9371179938316345,0.019029581919312477],"argmax_index":2,"argmax_id":"overcast","top2_margin":0.912934297695756,"near_tie":false,"gold":"overcast","gold_correct":true},{"question_id":"wildness","type":"score","question_span":[537,543],"option_spans":[[554,564],[570,581],[587,598],[604,615]],"option_ids":["0","1","2","3"],"logits":[-5.087391376495361,-5.037034034729004,-3.564222574234009,-0.7367212176322937],"probs":[0.011880886740982533,0.012494496069848537,0.05449450761079788,0.9211300611495972],"argmax_index":3,"argmax_id":"3","top2_margin":0.8666355535387993,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"snow_visible":{"type":"noul","noul":0.9215},"reflection":{"type":"noul","noul":0.9502},"sky":{"type":"choice","choice":"overcast","confidence":0.9371,"probabilities":{"clear":0.0197,"overcast":0.9371,"rain":0.019,"night":0.0242}},"wildness":{"type":"score","score":2.8849,"confidence":0.9211,"legend":{"0":"urban","1":"suburban","2":"rural","3":"wilderness"},"probabilities":{"0":0.0119,"1":0.0125,"2":0.0545,"3":0.9211}}},"usage":{"input_tokens":637,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"photo_03","arm":"native","set":"fixture","source":"cc0_photo","tokens":1041,"n_image_tokens":600,"state_tokens":12,"token_offset":36,"image_files":["photo_lake.png"],"image_size_in":[960,640],"image_size_fed":[960,640],"resized_hw":[640,960],"grid_thw":[1,40,60],"merged_hw":[20,30],"rope_shift_amount":570,"pixel_values_shape":[2400,1536],"pixel_values_sha256":"5d352fdc784e56a9ba7d8fa58b9270954433b14515d833b1d6278d010f3af1f6","image_embeds_shape":[600,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,7945,3147,15666,4602,883,279,12037,17865,6345,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,11456,33229,198,9222,25,307,10644,198,669,55530,25,220,60187,369,9155,303,279,6345,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,21172,198,9222,25,307,10644,198,669,55530,25,220,760,2919,25133,279,17865,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,12515,198,9222,25,5576,198,669,55530,25,220,3710,369,279,12515,1040,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,13606,321,38383,2129,1979,816,3147,7187,8934,198,32929,220,17,25,220,4754,4532,3147,48739,2129,1979,816,3147,9011,8934,198,32929,220,18,25,220,4754,4532,3147,1853,3701,2129,1979,816,3147,1908,3701,8934,198,32929,220,19,25,220,4754,4532,3147,49,1988,2129,1979,816,3147,28142,8934,198,4537,38945,198,198,27806,220,19,198,887,25,8121,2022,198,9222,25,5284,198,669,55530,25,220,4199,8121,369,279,1925,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,57062,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,1899,57062,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,81,4038,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,65584,27636,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"snow_visible","type":"noul","question_span":[677,684],"option_spans":[[695,713],[719,737]],"option_ids":["true","false"],"logits":[-1.9916656017303467,-4.681529521942139],"probs":[0.9364258646965027,0.06357411295175552],"argmax_index":0,"argmax_id":"true","top2_margin":0.8728517517447472,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"reflection","type":"noul","question_span":[759,765],"option_spans":[[776,794],[800,818]],"option_ids":["true","false"],"logits":[-1.4946279525756836,-4.561967372894287],"probs":[0.955525279045105,0.044474758207798004],"argmax_index":0,"argmax_id":"true","top2_margin":0.911050520837307,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"sky","type":"choice","question_span":[839,845],"option_spans":[[856,868],[874,884],[890,902],[908,919]],"option_ids":["clear","night","overcast","rain"],"logits":[-4.75921630859375,-4.6404128074646,-0.9220372438430786,-4.825642108917236],"probs":[0.020219869911670685,0.022770578041672707,0.93808913230896,0.01892038621008396],"argmax_index":2,"argmax_id":"overcast","top2_margin":0.9153185542672873,"near_tie":false,"gold":"overcast","gold_correct":true},{"question_id":"wildness","type":"score","question_span":[941,947],"option_spans":[[958,968],[974,985],[991,1002],[1008,1019]],"option_ids":["0","1","2","3"],"logits":[-5.013962268829346,-4.941922187805176,-3.3938934803009033,-0.7754532694816589],"probs":[0.013083424419164658,0.014060735702514648,0.06611628830432892,0.9067395925521851],"argmax_index":3,"argmax_id":"3","top2_margin":0.8406233042478561,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"snow_visible":{"type":"noul","noul":0.9364},"reflection":{"type":"noul","noul":0.9555},"sky":{"type":"choice","choice":"overcast","confidence":0.9381,"probabilities":{"clear":0.0202,"overcast":0.9381,"rain":0.0189,"night":0.0228}},"wildness":{"type":"score","score":2.8665,"confidence":0.9067,"legend":{"0":"urban","1":"suburban","2":"rural","3":"wilderness"},"probabilities":{"0":0.0131,"1":0.0141,"2":0.0661,"3":0.9067}}},"usage":{"input_tokens":1041,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"photo_04","arm":"g256","set":"fixture","source":"cc0_photo","tokens":502,"n_image_tokens":64,"state_tokens":8,"token_offset":36,"image_files":["photo_coffee.png"],"image_size_in":[960,635],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"37b607e4ee4660119f77af593cdbec50c0103038d58fad37e94802ef19e2aea7","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,3710,369,6625,303,279,12037,6345,30,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,6808,198,9222,25,5576,198,669,55530,25,220,22365,6808,369,303,279,10212,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,79440,2129,1979,816,3147,75440,8934,198,32929,220,17,25,220,4754,4532,3147,60494,543,2129,1979,816,3147,8350,543,8934,198,32929,220,18,25,220,4754,4532,3147,44,32062,2129,1979,816,3147,76,32062,8934,198,32929,220,19,25,220,4754,4532,3147,6430,64,2129,1979,816,3147,78781,8934,198,4537,38945,198,198,27806,220,17,198,887,25,13671,36406,198,9222,25,307,10644,198,669,55530,25,220,44,32062,682,978,3568,310,279,6808,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,10212,6504,198,9222,25,5576,198,669,55530,25,220,3710,1829,369,279,10212,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,14004,2129,1979,816,3147,11124,8934,198,32929,220,17,25,220,4754,4532,3147,10025,2129,1979,816,3147,11855,8934,198,32929,220,18,25,220,4754,4532,3147,5844,2129,1979,816,3147,1114,8934,198,32929,220,19,25,220,4754,4532,3147,13672,2129,1979,816,3147,5603,8934,198,4537,38945,198,198,27806,220,19,198,887,25,26034,198,9222,25,5284,198,669,55530,25,220,4199,1599,10482,26034,513,9155,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,6487,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,64,2342,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,33431,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"drink","type":"choice","question_span":[135,142],"option_spans":[[153,163],[169,181],[187,199],[205,216]],"option_ids":["coffee","juice","milk","tea"],"logits":[-0.5158005952835083,-5.033682346343994,-5.055849075317383,-4.6105499267578125],"probs":[0.9631638526916504,0.010510153137147427,0.010279739275574684,0.016046231612563133],"argmax_index":0,"argmax_id":"coffee","top2_margin":0.9471176210790873,"near_tie":false,"gold":"coffee","gold_correct":true},{"question_id":"milk_added","type":"noul","question_span":[239,248],"option_spans":[[259,277],[283,301]],"option_ids":["true","false"],"logits":[-3.465205192565918,1.6965525150299072],"probs":[0.0056989542208611965,0.9943010807037354],"argmax_index":1,"argmax_id":"false","top2_margin":0.9886021264828742,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"cup_color","type":"choice","question_span":[323,329],"option_spans":[[340,350],[356,366],[372,382],[388,398]],"option_ids":["black","blue","red","white"],"logits":[-5.198818206787109,-5.223084449768066,-5.160698413848877,-0.3309723734855652],"probs":[0.0075156670063734055,0.0073354849591851234,0.0078076934441924095,0.9773411750793457],"argmax_index":3,"argmax_id":"white","top2_margin":0.9695334816351533,"near_tie":false,"gold":"white","gold_correct":true},{"question_id":"beans","type":"score","question_span":[419,426],"option_spans":[[437,447],[453,464],[470,480]],"option_ids":["0","1","2"],"logits":[-4.405829429626465,-3.1962311267852783,0.005625240504741669],"probs":[0.01152852550148964,0.03864520415663719,0.9498262405395508],"argmax_index":2,"argmax_id":"2","top2_margin":0.9111810363829136,"near_tie":false,"gold":"2","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"drink":{"type":"choice","choice":"coffee","confidence":0.9632,"probabilities":{"tea":0.016,"coffee":0.9632,"milk":0.0103,"juice":0.0105}},"milk_added":{"type":"noul","noul":0.0057},"cup_color":{"type":"choice","choice":"white","confidence":0.9773,"probabilities":{"white":0.9773,"black":0.0075,"red":0.0078,"blue":0.0073}},"beans":{"type":"score","score":1.9383,"confidence":0.9498,"legend":{"0":"none","1":"a few","2":"many"},"probabilities":{"0":0.0115,"1":0.0386,"2":0.9498}}},"usage":{"input_tokens":502,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"photo_04","arm":"g448","set":"fixture","source":"cc0_photo","tokens":634,"n_image_tokens":196,"state_tokens":8,"token_offset":36,"image_files":["photo_coffee.png"],"image_size_in":[960,635],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"293a9c25a46b7ad0180a649556ddb43a97a59ea48701feb859c425620f253add","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,3710,369,6625,303,279,12037,6345,30,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,6808,198,9222,25,5576,198,669,55530,25,220,22365,6808,369,303,279,10212,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,79440,2129,1979,816,3147,75440,8934,198,32929,220,17,25,220,4754,4532,3147,60494,543,2129,1979,816,3147,8350,543,8934,198,32929,220,18,25,220,4754,4532,3147,44,32062,2129,1979,816,3147,76,32062,8934,198,32929,220,19,25,220,4754,4532,3147,6430,64,2129,1979,816,3147,78781,8934,198,4537,38945,198,198,27806,220,17,198,887,25,13671,36406,198,9222,25,307,10644,198,669,55530,25,220,44,32062,682,978,3568,310,279,6808,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,10212,6504,198,9222,25,5576,198,669,55530,25,220,3710,1829,369,279,10212,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,14004,2129,1979,816,3147,11124,8934,198,32929,220,17,25,220,4754,4532,3147,10025,2129,1979,816,3147,11855,8934,198,32929,220,18,25,220,4754,4532,3147,5844,2129,1979,816,3147,1114,8934,198,32929,220,19,25,220,4754,4532,3147,13672,2129,1979,816,3147,5603,8934,198,4537,38945,198,198,27806,220,19,198,887,25,26034,198,9222,25,5284,198,669,55530,25,220,4199,1599,10482,26034,513,9155,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,6487,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,64,2342,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,33431,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"drink","type":"choice","question_span":[267,274],"option_spans":[[285,295],[301,313],[319,331],[337,348]],"option_ids":["coffee","juice","milk","tea"],"logits":[-0.48188140988349915,-5.043992042541504,-5.034241676330566,-4.7030348777771],"probs":[0.9655641317367554,0.010080492123961449,0.010179261676967144,0.014176125638186932],"argmax_index":0,"argmax_id":"coffee","top2_margin":0.9513880060985684,"near_tie":false,"gold":"coffee","gold_correct":true},{"question_id":"milk_added","type":"noul","question_span":[371,380],"option_spans":[[391,409],[415,433]],"option_ids":["true","false"],"logits":[-3.5533573627471924,1.6821911334991455],"probs":[0.005295709706842899,0.9947043061256409],"argmax_index":1,"argmax_id":"false","top2_margin":0.989408596418798,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"cup_color","type":"choice","question_span":[455,461],"option_spans":[[472,482],[488,498],[504,514],[520,530]],"option_ids":["black","blue","red","white"],"logits":[-5.226492881774902,-5.22794771194458,-5.161438941955566,-0.3222389221191406],"probs":[0.007250157650560141,0.007239617872983217,0.0077374884858727455,0.9777726531028748],"argmax_index":3,"argmax_id":"white","top2_margin":0.970035164617002,"near_tie":false,"gold":"white","gold_correct":true},{"question_id":"beans","type":"score","question_span":[551,558],"option_spans":[[569,579],[585,596],[602,612]],"option_ids":["0","1","2"],"logits":[-4.441617488861084,-3.2935285568237305,-0.008819125592708588],"probs":[0.011322620324790478,0.03569074720144272,0.9529866576194763],"argmax_index":2,"argmax_id":"2","top2_margin":0.9172959104180336,"near_tie":false,"gold":"2","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"drink":{"type":"choice","choice":"coffee","confidence":0.9656,"probabilities":{"tea":0.0142,"coffee":0.9656,"milk":0.0102,"juice":0.0101}},"milk_added":{"type":"noul","noul":0.0053},"cup_color":{"type":"choice","choice":"white","confidence":0.9778,"probabilities":{"white":0.9778,"black":0.0073,"red":0.0077,"blue":0.0072}},"beans":{"type":"score","score":1.9417,"confidence":0.953,"legend":{"0":"none","1":"a few","2":"many"},"probabilities":{"0":0.0113,"1":0.0357,"2":0.953}}},"usage":{"input_tokens":634,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"photo_04","arm":"native","set":"fixture","source":"cc0_photo","tokens":1038,"n_image_tokens":600,"state_tokens":8,"token_offset":36,"image_files":["photo_coffee.png"],"image_size_in":[960,635],"image_size_fed":[960,635],"resized_hw":[640,960],"grid_thw":[1,40,60],"merged_hw":[20,30],"rope_shift_amount":570,"pixel_values_shape":[2400,1536],"pixel_values_sha256":"2f4c444ad6508883e62e3608a78211c4d24f346ddb5a38148e0f1751748381b5","image_embeds_shape":[600,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,3710,369,6625,303,279,12037,6345,30,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,6808,198,9222,25,5576,198,669,55530,25,220,22365,6808,369,303,279,10212,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,79440,2129,1979,816,3147,75440,8934,198,32929,220,17,25,220,4754,4532,3147,60494,543,2129,1979,816,3147,8350,543,8934,198,32929,220,18,25,220,4754,4532,3147,44,32062,2129,1979,816,3147,76,32062,8934,198,32929,220,19,25,220,4754,4532,3147,6430,64,2129,1979,816,3147,78781,8934,198,4537,38945,198,198,27806,220,17,198,887,25,13671,36406,198,9222,25,307,10644,198,669,55530,25,220,44,32062,682,978,3568,310,279,6808,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,10212,6504,198,9222,25,5576,198,669,55530,25,220,3710,1829,369,279,10212,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,14004,2129,1979,816,3147,11124,8934,198,32929,220,17,25,220,4754,4532,3147,10025,2129,1979,816,3147,11855,8934,198,32929,220,18,25,220,4754,4532,3147,5844,2129,1979,816,3147,1114,8934,198,32929,220,19,25,220,4754,4532,3147,13672,2129,1979,816,3147,5603,8934,198,4537,38945,198,198,27806,220,19,198,887,25,26034,198,9222,25,5284,198,669,55530,25,220,4199,1599,10482,26034,513,9155,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,6487,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,64,2342,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,33431,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"drink","type":"choice","question_span":[671,678],"option_spans":[[689,699],[705,717],[723,735],[741,752]],"option_ids":["coffee","juice","milk","tea"],"logits":[-0.5188403725624084,-5.031795978546143,-5.036859512329102,-4.761591911315918],"probs":[0.9650231003761292,0.01058244425803423,0.010528994724154472,0.013865454122424126],"argmax_index":0,"argmax_id":"coffee","top2_margin":0.951157646253705,"near_tie":false,"gold":"coffee","gold_correct":true},{"question_id":"milk_added","type":"noul","question_span":[775,784],"option_spans":[[795,813],[819,837]],"option_ids":["true","false"],"logits":[-3.5722806453704834,1.68734610080719],"probs":[0.005170371849089861,0.9948295950889587],"argmax_index":1,"argmax_id":"false","top2_margin":0.9896592232398689,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"cup_color","type":"choice","question_span":[859,865],"option_spans":[[876,886],[892,902],[908,918],[924,934]],"option_ids":["black","blue","red","white"],"logits":[-5.24997091293335,-5.191107749938965,-5.126114845275879,-0.2925410270690918],"probs":[0.006876620464026928,0.0072935507632792,0.007783322595059872,0.9780464768409729],"argmax_index":3,"argmax_id":"white","top2_margin":0.970263154245913,"near_tie":false,"gold":"white","gold_correct":true},{"question_id":"beans","type":"score","question_span":[955,962],"option_spans":[[973,983],[989,1000],[1006,1016]],"option_ids":["0","1","2"],"logits":[-4.378904342651367,-3.3151097297668457,-0.01913376897573471],"probs":[0.012174849398434162,0.03527474403381348,0.95255047082901],"argmax_index":2,"argmax_id":"2","top2_margin":0.9172757267951965,"near_tie":false,"gold":"2","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"drink":{"type":"choice","choice":"coffee","confidence":0.965,"probabilities":{"tea":0.0139,"coffee":0.965,"milk":0.0105,"juice":0.0106}},"milk_added":{"type":"noul","noul":0.0052},"cup_color":{"type":"choice","choice":"white","confidence":0.978,"probabilities":{"white":0.978,"black":0.0069,"red":0.0078,"blue":0.0073}},"beans":{"type":"score","score":1.9404,"confidence":0.9526,"legend":{"0":"none","1":"a few","2":"many"},"probabilities":{"0":0.0122,"1":0.0353,"2":0.9526}}},"usage":{"input_tokens":1038,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_a3f18f3a63d45345942b","arm":"text","set":"fixture","source":"semif_authored144","tokens":185,"n_image_tokens":0,"state_tokens":22,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,3278,12095,11128,13329,33800,13,561,24279,40988,781,599,524,3482,978,27419,310,279,5813,579,27233,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,13329,33800,599,978,27419,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[82,94],"option_spans":[[105,121],[127,143],[149,163]],"option_ids":["contradicted","insufficient","supported"],"logits":[-1.1605620384216309,-4.601136684417725,-4.539452075958252],"probs":[0.9379705190658569,0.030058452859520912,0.03197097405791283],"argmax_index":0,"argmax_id":"contradicted","top2_margin":0.9059995450079441,"near_tie":false,"gold":"contradicted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"contradicted","confidence":0.938,"probabilities":{"supported":0.032,"insufficient":0.0301,"contradicted":0.938}}},"usage":{"input_tokens":185,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_f40beba9088c8db8bbd6","arm":"text","set":"fixture","source":"semif_authored144","tokens":183,"n_image_tokens":0,"state_tokens":22,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,3278,12095,11128,13329,33800,13,561,24279,40988,781,599,524,3482,978,27419,310,279,5813,579,27233,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,13329,33800,998,11128,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[82,92],"option_spans":[[103,119],[125,141],[147,161]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.388455867767334,-3.7669827938079834,-1.1155295372009277],"probs":[0.03418789803981781,0.06364653259515762,0.9021655917167664],"argmax_index":2,"argmax_id":"supported","top2_margin":0.8385190591216087,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.9022,"probabilities":{"supported":0.9022,"insufficient":0.0636,"contradicted":0.0342}}},"usage":{"input_tokens":183,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_0b43ea8e24e74d621f1b","arm":"text","set":"fixture","source":"semif_authored144","tokens":177,"n_image_tokens":0,"state_tokens":14,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,24279,40988,424,27419,279,13329,33800,310,279,5813,579,27233,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,13329,33800,599,978,27419,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[74,86],"option_spans":[[97,113],[119,135],[141,155]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.774107456207275,-4.550616264343262,-1.1849499940872192],"probs":[0.0260050967335701,0.03251767158508301,0.9414771795272827],"argmax_index":2,"argmax_id":"supported","top2_margin":0.9089595079421997,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.9415,"probabilities":{"insufficient":0.0325,"supported":0.9415,"contradicted":0.026}}},"usage":{"input_tokens":177,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_942ec65dac0bfb9874f9","arm":"text","set":"fixture","source":"semif_authored144","tokens":177,"n_image_tokens":0,"state_tokens":14,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,3278,12095,7189,264,27233,21407,694,874,1906,466,25504,2552,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,13329,33800,599,978,27419,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[74,86],"option_spans":[[97,113],[119,135],[141,155]],"option_ids":["contradicted","insufficient","supported"],"logits":[-3.3295066356658936,-1.0609946250915527,-4.478346347808838],"probs":[0.09105799347162247,0.8800762891769409,0.02886578068137169],"argmax_index":1,"argmax_id":"insufficient","top2_margin":0.7890182957053185,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.8801,"probabilities":{"insufficient":0.8801,"supported":0.0289,"contradicted":0.0911}}},"usage":{"input_tokens":177,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_e0c140e9222d1bce5647","arm":"text","set":"fixture","source":"semif_authored144","tokens":189,"n_image_tokens":0,"state_tokens":23,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,8376,13514,11,328,760,14408,2640,51106,513,16713,13,353,7268,25,7478,51106,998,3766,2261,821,4565,1149,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,8376,809,21735,279,14408,579,41198,4952,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[83,98],"option_spans":[[109,125],[131,147],[153,167]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.617707252502441,-4.801997661590576,-1.43197762966156],"probs":[0.03843696787953377,0.031967807561159134,0.9295952320098877],"argmax_index":2,"argmax_id":"supported","top2_margin":0.8911582641303539,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.9296,"probabilities":{"supported":0.9296,"contradicted":0.0384,"insufficient":0.032}}},"usage":{"input_tokens":189,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_8e8c3804a3c15ebb31e3","arm":"text","set":"fixture","source":"semif_authored144","tokens":186,"n_image_tokens":0,"state_tokens":23,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,8376,13514,11,328,760,14408,2640,51106,513,16713,13,353,7268,25,7478,51106,998,3766,2261,821,4565,1149,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,8376,39795,279,41198,4952,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[83,95],"option_spans":[[106,122],[128,144],[150,164]],"option_ids":["contradicted","insufficient","supported"],"logits":[1.0137969255447388,-2.777022123336792,-1.0767312049865723],"probs":[0.8724489212036133,0.019697366282343864,0.10785374790430069],"argmax_index":0,"argmax_id":"contradicted","top2_margin":0.7645951732993126,"near_tie":false,"gold":"contradicted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"contradicted","confidence":0.8724,"probabilities":{"insufficient":0.0197,"supported":0.1079,"contradicted":0.8724}}},"usage":{"input_tokens":186,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_56c419b12cd37cbe790c","arm":"text","set":"fixture","source":"semif_authored144","tokens":188,"n_image_tokens":0,"state_tokens":22,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,8376,13514,11,328,760,14408,2640,51106,513,16713,11,694,6643,998,3766,13,353,24240,421,4952,1149,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,8376,809,21735,279,14408,579,41198,4952,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[82,97],"option_spans":[[108,124],[130,146],[152,166]],"option_ids":["contradicted","insufficient","supported"],"logits":[-1.0719987154006958,-4.650363922119141,-4.391922950744629],"probs":[0.9397817254066467,0.02623993158340454,0.03397833928465843],"argmax_index":0,"argmax_id":"contradicted","top2_margin":0.9058033861219883,"near_tie":false,"gold":"contradicted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"contradicted","confidence":0.9398,"probabilities":{"contradicted":0.9398,"insufficient":0.0262,"supported":0.034}}},"usage":{"input_tokens":188,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_75b470586834df8c568d","arm":"text","set":"fixture","source":"semif_authored144","tokens":181,"n_image_tokens":0,"state_tokens":15,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,8376,39698,279,14408,694,5529,874,3847,883,51106,466,279,4565,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,8376,809,21735,279,14408,579,41198,4952,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[75,90],"option_spans":[[101,117],[123,139],[145,159]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.710535049438477,-1.116910696029663,-4.081366539001465],"probs":[0.02548310160636902,0.9267092943191528,0.04780757427215576],"argmax_index":1,"argmax_id":"insufficient","top2_margin":0.8789017200469971,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.9267,"probabilities":{"supported":0.0478,"contradicted":0.0255,"insufficient":0.9267}}},"usage":{"input_tokens":181,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_834f3b3e10cc61b33d25","arm":"text","set":"fixture","source":"semif_authored144","tokens":175,"n_image_tokens":0,"state_tokens":14,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,51715,2640,14586,54753,369,2767,13,32961,42033,369,20335,25331,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,14586,42033,369,2767,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[74,84],"option_spans":[[95,111],[117,133],[139,153]],"option_ids":["contradicted","insufficient","supported"],"logits":[-0.9267144203186035,-4.847934246063232,-4.750284194946289],"probs":[0.9600001573562622,0.019024234265089035,0.020975681021809578],"argmax_index":0,"argmax_id":"contradicted","top2_margin":0.9390244763344526,"near_tie":false,"gold":"contradicted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"contradicted","confidence":0.96,"probabilities":{"supported":0.021,"insufficient":0.019,"contradicted":0.96}}},"usage":{"input_tokens":175,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_76c7a79b1ff2363a60c5","arm":"text","set":"fixture","source":"semif_authored144","tokens":175,"n_image_tokens":0,"state_tokens":14,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,51715,2640,14586,54753,369,2767,13,32961,42033,369,20335,25331,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,14586,54753,369,2767,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[74,84],"option_spans":[[95,111],[117,133],[139,153]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.950608730316162,-4.905882358551025,-0.5284470319747925],"probs":[0.011720328591763973,0.012256436049938202,0.9760231971740723],"argmax_index":2,"argmax_id":"supported","top2_margin":0.9637667611241341,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.976,"probabilities":{"supported":0.976,"insufficient":0.0123,"contradicted":0.0117}}},"usage":{"input_tokens":175,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_36c71bb649b7d5eccb1f","arm":"text","set":"fixture","source":"semif_authored144","tokens":174,"n_image_tokens":0,"state_tokens":13,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,51715,40988,14586,42033,369,2767,321,14586,54753,369,25331,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,14586,42033,369,2767,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[73,83],"option_spans":[[94,110],[116,132],[138,152]],"option_ids":["contradicted","insufficient","supported"],"logits":[-5.012272357940674,-5.0702362060546875,-0.79993736743927],"probs":[0.014397252351045609,0.013586457818746567,0.9720162153244019],"argmax_index":2,"argmax_id":"supported","top2_margin":0.9576189629733562,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.972,"probabilities":{"supported":0.972,"insufficient":0.0136,"contradicted":0.0144}}},"usage":{"input_tokens":174,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_eac5121a4a1bf391a08c","arm":"text","set":"fixture","source":"semif_authored144","tokens":173,"n_image_tokens":0,"state_tokens":12,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,51715,11140,2107,51201,694,7613,1159,836,3511,2552,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,14586,42033,369,2767,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[72,82],"option_spans":[[93,109],[115,131],[137,151]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.914957046508789,-0.6555237770080566,-4.815057277679443],"probs":[0.013722140341997147,0.9711139798164368,0.015163789503276348],"argmax_index":1,"argmax_id":"insufficient","top2_margin":0.9559501903131604,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.9711,"probabilities":{"contradicted":0.0137,"insufficient":0.9711,"supported":0.0152}}},"usage":{"input_tokens":173,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_44574c44e243ee1b2f52","arm":"text","set":"fixture","source":"semif_authored144","tokens":186,"n_image_tokens":0,"state_tokens":23,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,46922,12830,28058,279,6321,310,5361,13,11116,3049,10579,264,25423,26895,5589,557,4252,321,5361,557,34811,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,34773,2321,369,303,5589,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[83,95],"option_spans":[[106,122],[128,144],[150,164]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.722664833068848,-4.819595813751221,-1.186289668083191],"probs":[0.02758634276688099,0.02503787726163864,0.9473757743835449],"argmax_index":2,"argmax_id":"supported","top2_margin":0.9197894316166639,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.9474,"probabilities":{"insufficient":0.025,"contradicted":0.0276,"supported":0.9474}}},"usage":{"input_tokens":186,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_2608370d01783735c3ea","arm":"text","set":"fixture","source":"semif_authored144","tokens":186,"n_image_tokens":0,"state_tokens":23,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,46922,12830,28058,279,6321,310,5361,13,11116,3049,10579,264,25423,26895,5589,557,4252,321,5361,557,34811,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,34773,2321,369,303,5361,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[83,95],"option_spans":[[106,122],[128,144],[150,164]],"option_ids":["contradicted","insufficient","supported"],"logits":[-1.420650839805603,-4.407536506652832,-4.531802654266357],"probs":[0.9132471680641174,0.046068113297224045,0.040684811770915985],"argmax_index":0,"argmax_id":"contradicted","top2_margin":0.8671790547668934,"near_tie":false,"gold":"contradicted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"contradicted","confidence":0.9132,"probabilities":{"supported":0.0407,"insufficient":0.0461,"contradicted":0.9132}}},"usage":{"input_tokens":186,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_ec94ef909f7918695ab3","arm":"text","set":"fixture","source":"semif_authored144","tokens":180,"n_image_tokens":0,"state_tokens":17,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,46922,3049,4252,82,279,2321,310,5361,321,50742,57409,279,6575,5589,2321,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,34773,2321,369,303,5589,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[77,89],"option_spans":[[100,116],[122,138],[144,158]],"option_ids":["contradicted","insufficient","supported"],"logits":[-1.266447901725769,-4.948559284210205,-4.797476768493652],"probs":[0.9483665823936462,0.023870177567005157,0.027763227000832558],"argmax_index":0,"argmax_id":"contradicted","top2_margin":0.9206033553928137,"near_tie":false,"gold":"contradicted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"contradicted","confidence":0.9484,"probabilities":{"supported":0.0278,"insufficient":0.0239,"contradicted":0.9484}}},"usage":{"input_tokens":180,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_03d919fd0e9b3a7cd1e6","arm":"text","set":"fixture","source":"semif_authored144","tokens":175,"n_image_tokens":0,"state_tokens":12,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,46922,32726,264,2321,25423,694,6486,874,34773,2321,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,34773,2321,369,303,5589,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[72,84],"option_spans":[[95,111],[117,133],[139,153]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.353754997253418,-1.5888071060180664,-4.626830101013184],"probs":[0.05669175833463669,0.9001638293266296,0.043144453316926956],"argmax_index":1,"argmax_id":"insufficient","top2_margin":0.843472070991993,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.9002,"probabilities":{"insufficient":0.9002,"supported":0.0431,"contradicted":0.0567}}},"usage":{"input_tokens":175,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_86a15618ba4cb90a307e","arm":"text","set":"fixture","source":"semif_authored144","tokens":179,"n_image_tokens":0,"state_tokens":17,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,2505,12037,10895,364,45164,279,28350,321,20335,2640,874,45164,682,978,10345,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,28350,682,978,45054,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[77,88],"option_spans":[[99,115],[121,137],[143,157]],"option_ids":["contradicted","insufficient","supported"],"logits":[-1.8707879781723022,-3.282742500305176,-4.1033854484558105],"probs":[0.7402384877204895,0.18037135899066925,0.07939021289348602],"argmax_index":0,"argmax_id":"contradicted","top2_margin":0.5598671287298203,"near_tie":false,"gold":"contradicted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"contradicted","confidence":0.7402,"probabilities":{"contradicted":0.7402,"insufficient":0.1804,"supported":0.0794}}},"usage":{"input_tokens":179,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_3dd9c3ac98f8ddf7bdd3","arm":"text","set":"fixture","source":"semif_authored144","tokens":179,"n_image_tokens":0,"state_tokens":17,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,2505,12037,10895,364,45164,279,28350,321,20335,2640,874,45164,682,978,10345,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,2128,5435,45164,10895,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[77,88],"option_spans":[[99,115],[121,137],[143,157]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.7549285888671875,-4.789936065673828,-1.1235804557800293],"probs":[0.02517034485936165,0.02430444024503231,0.950525164604187],"argmax_index":2,"argmax_id":"supported","top2_margin":0.9253548197448254,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.9505,"probabilities":{"contradicted":0.0252,"supported":0.9505,"insufficient":0.0243}}},"usage":{"input_tokens":179,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_36273dd4777876a26338","arm":"text","set":"fixture","source":"semif_authored144","tokens":175,"n_image_tokens":0,"state_tokens":13,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,2505,10345,279,45164,321,12076,279,28350,579,16384,4528,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,28350,682,978,45054,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[73,84],"option_spans":[[95,111],[117,133],[139,153]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.832609176635742,-4.69189977645874,-0.9909957051277161],"probs":[0.020512109622359276,0.02361128479242325,0.9558766484260559],"argmax_index":2,"argmax_id":"supported","top2_margin":0.9322653636336327,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.9559,"probabilities":{"contradicted":0.0205,"supported":0.9559,"insufficient":0.0236}}},"usage":{"input_tokens":175,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_07ee70cfe4b01330ce89","arm":"text","set":"fixture","source":"semif_authored144","tokens":175,"n_image_tokens":0,"state_tokens":13,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,2505,16229,264,28350,12448,694,874,10018,466,18100,1928,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,28350,682,978,45054,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[73,84],"option_spans":[[95,111],[117,133],[139,153]],"option_ids":["contradicted","insufficient","supported"],"logits":[-3.5717718601226807,-1.4378883838653564,-4.459823131561279],"probs":[0.10142947733402252,0.8568366169929504,0.04173384606838226],"argmax_index":1,"argmax_id":"insufficient","top2_margin":0.7554071396589279,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.8568,"probabilities":{"supported":0.0417,"contradicted":0.1014,"insufficient":0.8568}}},"usage":{"input_tokens":175,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_51ceab3ce622c7c759f1","arm":"text","set":"fixture","source":"semif_authored144","tokens":191,"n_image_tokens":0,"state_tokens":29,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,15542,264,47784,1420,11,264,3542,46094,279,17789,314,264,24309,18075,13,561,5492,5020,5134,411,369,42140,321,874,1865,18075,557,10782,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,18075,17789,557,42140,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[89,100],"option_spans":[[111,127],[133,149],[155,169]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.772977828979492,-4.649282932281494,-0.9766820669174194],"probs":[0.02142815664410591,0.024249611422419548,0.9543222188949585],"argmax_index":2,"argmax_id":"supported","top2_margin":0.930072607472539,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.9543,"probabilities":{"contradicted":0.0214,"insufficient":0.0242,"supported":0.9543}}},"usage":{"input_tokens":191,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_b8091be929b1e489fcc4","arm":"text","set":"fixture","source":"semif_authored144","tokens":191,"n_image_tokens":0,"state_tokens":29,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,15542,264,47784,1420,11,264,3542,46094,279,17789,314,264,24309,18075,13,561,5492,5020,5134,411,369,42140,321,874,1865,18075,557,10782,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,264,1865,18075,557,10782,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[89,100],"option_spans":[[111,127],[133,149],[155,169]],"option_ids":["contradicted","insufficient","supported"],"logits":[-1.349582552909851,-4.512454032897949,-4.600882053375244],"probs":[0.9250455498695374,0.0391332171857357,0.035821329802274704],"argmax_index":0,"argmax_id":"contradicted","top2_margin":0.8859123326838017,"near_tie":false,"gold":"contradicted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"contradicted","confidence":0.925,"probabilities":{"supported":0.0358,"contradicted":0.925,"insufficient":0.0391}}},"usage":{"input_tokens":191,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_c89b86e3f55123bfa506","arm":"text","set":"fixture","source":"semif_authored144","tokens":182,"n_image_tokens":0,"state_tokens":20,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,5020,16067,264,1865,11619,18075,10782,4732,264,4906,321,50742,2640,279,17789,557,524,42140,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,18075,17789,557,42140,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[80,91],"option_spans":[[102,118],[124,140],[146,160]],"option_ids":["contradicted","insufficient","supported"],"logits":[-1.3059011697769165,-4.481929779052734,-4.6123247146606445],"probs":[0.9273013472557068,0.03871588036417961,0.033982813358306885],"argmax_index":0,"argmax_id":"contradicted","top2_margin":0.8885854668915272,"near_tie":false,"gold":"contradicted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"contradicted","confidence":0.9273,"probabilities":{"supported":0.034,"contradicted":0.9273,"insufficient":0.0387}}},"usage":{"input_tokens":182,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_3b0958cabc339005cf66","arm":"text","set":"fixture","source":"semif_authored144","tokens":178,"n_image_tokens":0,"state_tokens":16,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,32,5020,32726,449,18075,1973,21988,264,17789,466,3315,279,2605,369,42140,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,18075,17789,557,42140,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[76,87],"option_spans":[[98,114],[120,136],[142,156]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.5467119216918945,-0.796612024307251,-4.585146427154541],"probs":[0.02247816137969494,0.9558912515640259,0.02163061685860157],"argmax_index":1,"argmax_id":"insufficient","top2_margin":0.9334130901843309,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.9559,"probabilities":{"insufficient":0.9559,"supported":0.0216,"contradicted":0.0225}}},"usage":{"input_tokens":178,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_f2b4ec4930322fe45118","arm":"text","set":"fixture","source":"semif_authored144","tokens":193,"n_image_tokens":0,"state_tokens":19,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,11141,27751,14086,5933,264,3019,303,449,5138,4706,70753,321,50742,52903,3256,10910,10672,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,66215,10672,1132,440,279,11141,579,6416,7634,13,3114,10672,381,5898,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[79,94],"option_spans":[[105,127],[133,149],[155,171]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-3.2987911701202393,-3.97519850730896,-0.13175952434539795],"probs":[0.03961123153567314,0.02013995125889778,0.9402487874031067],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.9006375558674335,"near_tie":false,"gold":"prohibited","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.9402,"probabilities":{"insufficient":0.0396,"prohibited":0.9402,"permitted":0.0201}}},"usage":{"input_tokens":193,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_61ac20b07a5189231e15","arm":"text","set":"fixture","source":"semif_authored144","tokens":199,"n_image_tokens":0,"state_tokens":19,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,11141,27751,14086,5933,264,3019,303,449,5138,4706,70753,321,50742,52903,3256,10910,10672,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,11236,4706,37276,369,5252,948,279,11141,27751,421,958,13,3114,279,3019,4833,303,279,4706,70753,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[79,100],"option_spans":[[111,133],[139,155],[161,177]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.097441673278809,-0.9983890056610107,-4.611849784851074],"probs":[0.04206136614084244,0.9327921271324158,0.025146571919322014],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.8907307609915733,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.9328,"probabilities":{"prohibited":0.0251,"permitted":0.9328,"insufficient":0.0421}}},"usage":{"input_tokens":199,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_4b3d794c004566610181","arm":"text","set":"fixture","source":"semif_authored144","tokens":184,"n_image_tokens":0,"state_tokens":10,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,11141,50742,27751,10910,10672,314,279,3019,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,66215,10672,1132,440,279,11141,579,6416,7634,13,3114,10672,381,5898,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[70,85],"option_spans":[[96,118],[124,140],[146,162]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.035193920135498,-0.6501374244689941,-4.630793571472168],"probs":[0.03218446671962738,0.9500744342803955,0.01774110272526741],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.9178899675607681,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.9501,"probabilities":{"prohibited":0.0177,"insufficient":0.0322,"permitted":0.9501}}},"usage":{"input_tokens":184,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_b7c4e62d07986286fce0","arm":"text","set":"fixture","source":"semif_authored144","tokens":189,"n_image_tokens":0,"state_tokens":15,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,2014,11141,579,7634,6321,6513,11,694,279,5533,424,5954,513,38136,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,66215,10672,1132,440,279,11141,579,6416,7634,13,3114,10672,381,5898,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[75,90],"option_spans":[[101,123],[129,145],[151,167]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-0.44891834259033203,-2.2488443851470947,-2.5508201122283936],"probs":[0.7766780257225037,0.1283935159444809,0.09492851048707962],"argmax_index":0,"argmax_id":"insufficient","top2_margin":0.6482845097780228,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.7767,"probabilities":{"insufficient":0.7767,"permitted":0.1284,"prohibited":0.0949}}},"usage":{"input_tokens":189,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_9473603b44af63beab79","arm":"text","set":"fixture","source":"semif_authored144","tokens":190,"n_image_tokens":0,"state_tokens":13,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,8214,7888,1118,12,3651,4706,694,524,15822,37370,4706,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,760,1118,12,3651,23213,7225,1132,1118,12,3651,4706,13,3114,411,8214,5091,424,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[73,91],"option_spans":[[102,124],[130,146],[152,168]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.922661304473877,-1.2227650880813599,-5.1126861572265625],"probs":[0.023657411336898804,0.9567793607711792,0.019563227891921997],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.9331219494342804,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.9568,"probabilities":{"permitted":0.9568,"insufficient":0.0237,"prohibited":0.0196}}},"usage":{"input_tokens":190,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_d41072dc35d8cd14c056","arm":"text","set":"fixture","source":"semif_authored144","tokens":195,"n_image_tokens":0,"state_tokens":13,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,8214,7888,1118,12,3651,4706,694,524,15822,37370,4706,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,15771,279,10242,5728,11,279,23213,7225,2107,1118,12,3651,321,15822,37370,4706,13,3114,411,8214,5091,424,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[73,96],"option_spans":[[107,129],[135,151],[157,173]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.408201217651367,-4.517597675323486,-0.7099769115447998],"probs":[0.023656362667679787,0.021204974502325058,0.955138623714447],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.9314822610467672,"near_tie":false,"gold":"prohibited","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.9551,"probabilities":{"permitted":0.0212,"prohibited":0.9551,"insufficient":0.0237}}},"usage":{"input_tokens":195,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_33d6e5da58f0fee2490d","arm":"text","set":"fixture","source":"semif_authored144","tokens":190,"n_image_tokens":0,"state_tokens":13,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,8214,7888,15822,37370,4706,694,524,1118,12,3651,4706,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,760,1118,12,3651,23213,7225,1132,1118,12,3651,4706,13,3114,411,8214,5091,424,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[73,91],"option_spans":[[102,124],[130,146],[152,168]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.424664497375488,-4.581136226654053,-0.7304629683494568],"probs":[0.02377069741487503,0.020327646285295486,0.9559016823768616],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.9321309849619865,"near_tie":false,"gold":"prohibited","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.9559,"probabilities":{"prohibited":0.9559,"insufficient":0.0238,"permitted":0.0203}}},"usage":{"input_tokens":190,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_823bf736b92e9d8fa66d","arm":"text","set":"fixture","source":"semif_authored144","tokens":190,"n_image_tokens":0,"state_tokens":13,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,8214,9393,364,4706,11,694,9463,1928,369,524,16713,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,760,1118,12,3651,23213,7225,1132,1118,12,3651,4706,13,3114,411,8214,5091,424,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[73,91],"option_spans":[[102,124],[130,146],[152,168]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-1.3651903867721558,-4.367263317108154,-3.9302139282226562],"probs":[0.8876254558563232,0.04410076141357422,0.06827382743358612],"argmax_index":0,"argmax_id":"insufficient","top2_margin":0.8193516284227371,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.8876,"probabilities":{"prohibited":0.0683,"insufficient":0.8876,"permitted":0.0441}}},"usage":{"input_tokens":190,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_8a4c3de28be95c105ca1","arm":"text","set":"fixture","source":"semif_authored144","tokens":189,"n_image_tokens":0,"state_tokens":16,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,14268,90595,3154,264,5887,23213,694,50742,440,51334,14014,364,2973,5492,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,25001,5492,369,5252,1132,1238,2973,67208,14014,13,3114,424,3060,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[76,90],"option_spans":[[101,123],[129,145],[151,167]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-3.405310869216919,-2.3437893390655518,-1.0428643226623535],"probs":[0.06892910599708557,0.19925789535045624,0.7318130135536194],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.5325551182031631,"near_tie":false,"gold":"prohibited","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.7318,"probabilities":{"permitted":0.1993,"insufficient":0.0689,"prohibited":0.7318}}},"usage":{"input_tokens":189,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_ea7ec66a5e8eac886548","arm":"text","set":"fixture","source":"semif_authored144","tokens":192,"n_image_tokens":0,"state_tokens":16,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,14268,90595,3154,264,5887,23213,694,50742,440,51334,14014,364,2973,5492,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,32,5887,1189,381,1801,948,5887,5492,369,11454,13,3114,279,5887,381,1801,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[76,93],"option_spans":[[104,126],[132,148],[154,170]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.261549949645996,-1.263110637664795,-4.428025722503662],"probs":[0.045660313218832016,0.9156817197799683,0.03865800052881241],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.8700214065611362,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.9157,"probabilities":{"permitted":0.9157,"insufficient":0.0457,"prohibited":0.0387}}},"usage":{"input_tokens":192,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_25172cd6ddc49ab0ca70","arm":"text","set":"fixture","source":"semif_authored144","tokens":183,"n_image_tokens":0,"state_tokens":10,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,14268,20335,90595,2973,5492,314,279,23213,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,25001,5492,369,5252,1132,1238,2973,67208,14014,13,3114,424,3060,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[70,84],"option_spans":[[95,117],[123,139],[145,161]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.3352580070495605,-1.2126331329345703,-4.849424362182617],"probs":[0.04114566370844841,0.9342493414878845,0.02460505999624729],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.8931036777794361,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.9342,"probabilities":{"prohibited":0.0246,"insufficient":0.0411,"permitted":0.9342}}},"usage":{"input_tokens":183,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_fb73a85d386cf59b0c51","arm":"text","set":"fixture","source":"semif_authored144","tokens":185,"n_image_tokens":0,"state_tokens":12,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,14268,10579,264,5307,694,1141,5492,6676,369,33829,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,25001,5492,369,5252,1132,1238,2973,67208,14014,13,3114,424,3060,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[72,86],"option_spans":[[97,119],[125,141],[147,163]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-0.98735511302948,-3.1250948905944824,-0.8817703723907471],"probs":[0.44857507944107056,0.05289645493030548,0.49852854013442993],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.049953460693359375,"near_tie":false,"gold":"insufficient","gold_correct":false}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.4985,"probabilities":{"insufficient":0.4486,"permitted":0.0529,"prohibited":0.4985}}},"usage":{"input_tokens":185,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_a02fd582d2881cc462b2","arm":"text","set":"fixture","source":"semif_authored144","tokens":195,"n_image_tokens":0,"state_tokens":16,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,19568,579,31111,6833,14103,279,48285,13159,321,50742,61228,279,63956,49744,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,32,31111,6833,27751,4202,6681,310,279,5493,424,14103,13,3114,279,19568,3599,279,48285,13159,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[76,96],"option_spans":[[107,129],[135,151],[157,173]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.781173229217529,-0.918728232383728,-4.935282230377197],"probs":[0.020227059721946716,0.962434709072113,0.017338203266263008],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.9422076493501663,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.9624,"probabilities":{"prohibited":0.0173,"insufficient":0.0202,"permitted":0.9624}}},"usage":{"input_tokens":195,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_19f37d5a97034bd7d04f","arm":"text","set":"fixture","source":"semif_authored144","tokens":195,"n_image_tokens":0,"state_tokens":16,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,19568,579,31111,6833,14103,279,48285,13159,321,50742,61228,279,63956,49744,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,32,31111,6833,27751,4202,6681,310,279,5493,424,14103,13,3114,279,19568,3599,279,63956,49744,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[76,96],"option_spans":[[107,129],[135,151],[157,173]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.256224155426025,-4.236337184906006,-0.36452627182006836],"probs":[0.019602429121732712,0.01999616250395775,0.9604014158248901],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.9404052533209324,"near_tie":false,"gold":"prohibited","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.9604,"probabilities":{"insufficient":0.0196,"prohibited":0.9604,"permitted":0.02}}},"usage":{"input_tokens":195,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_fda0ca94652c6d739ecf","arm":"text","set":"fixture","source":"semif_authored144","tokens":194,"n_image_tokens":0,"state_tokens":15,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,31111,6833,50742,61228,279,48285,13159,321,14103,1132,279,63956,49744,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,32,31111,6833,27751,4202,6681,310,279,5493,424,14103,13,3114,279,19568,3599,279,48285,13159,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[75,95],"option_spans":[[106,128],[134,150],[156,172]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.50727653503418,-4.417235851287842,-0.6606711149215698],"probs":[0.020438184961676598,0.022363845258951187,0.9571979641914368],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.9348341189324856,"near_tie":false,"gold":"prohibited","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.9572,"probabilities":{"prohibited":0.9572,"permitted":0.0224,"insufficient":0.0204}}},"usage":{"input_tokens":194,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_d0e36c4cb10a29b3191c","arm":"text","set":"fixture","source":"semif_authored144","tokens":193,"n_image_tokens":0,"state_tokens":14,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,19568,682,264,31111,6833,694,1141,9469,5493,513,524,16713,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,32,31111,6833,27751,4202,6681,310,279,5493,424,14103,13,3114,279,19568,3599,279,48285,13159,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[74,94],"option_spans":[[105,127],[133,149],[155,171]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-1.761426568031311,-3.138834238052368,-0.7562479972839355],"probs":[0.25096452236175537,0.06330118328332901,0.6857342720031738],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.43476974964141846,"near_tie":false,"gold":"insufficient","gold_correct":false}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.6857,"probabilities":{"insufficient":0.251,"permitted":0.0633,"prohibited":0.6857}}},"usage":{"input_tokens":193,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_a0a39ace7e5337d038a7","arm":"text","set":"fixture","source":"semif_authored144","tokens":197,"n_image_tokens":0,"state_tokens":17,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,2605,6171,27751,9722,279,2940,803,694,50742,52903,3256,9722,279,12964,2469,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3936,449,2605,2002,1132,948,279,6171,3049,4603,9722,421,2002,13,3114,279,12964,2469,381,5324,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[77,98],"option_spans":[[109,131],[137,153],[159,175]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.658501148223877,-4.561951160430908,-0.4388434886932373],"probs":[0.014262980781495571,0.015708742663264275,0.9700283408164978],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.9543195981532335,"near_tie":false,"gold":"prohibited","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.97,"probabilities":{"prohibited":0.97,"permitted":0.0157,"insufficient":0.0143}}},"usage":{"input_tokens":197,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_eeae283841ce75dd25bf","arm":"text","set":"fixture","source":"semif_authored144","tokens":197,"n_image_tokens":0,"state_tokens":17,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,2605,6171,27751,9722,279,2940,803,694,50742,52903,3256,9722,279,12964,2469,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3936,449,2605,2002,1132,948,279,6171,3049,4603,9722,421,2002,13,3114,279,2940,803,381,5324,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[77,98],"option_spans":[[109,131],[137,153],[159,175]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-5.029676914215088,-0.8707584738731384,-5.120102405548096],"probs":[0.015170869417488575,0.9709699153900146,0.01385923195630312],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.9557990459725261,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.971,"probabilities":{"insufficient":0.0152,"permitted":0.971,"prohibited":0.0139}}},"usage":{"input_tokens":197,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_00bb4cc043f7d2b288d4","arm":"text","set":"fixture","source":"semif_authored144","tokens":189,"n_image_tokens":0,"state_tokens":9,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,6171,50742,27751,9722,279,12964,2469,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3936,449,2605,2002,1132,948,279,6171,3049,4603,9722,421,2002,13,3114,279,12964,2469,381,5324,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[69,90],"option_spans":[[101,123],[129,145],[151,167]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.704217433929443,-0.8725537061691284,-4.7644500732421875],"probs":[0.02079833298921585,0.9596190452575684,0.01958257518708706],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.9388207122683525,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.9596,"probabilities":{"insufficient":0.0208,"prohibited":0.0196,"permitted":0.9596}}},"usage":{"input_tokens":189,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_1a81ead69e93e5d0c86a","arm":"text","set":"fixture","source":"semif_authored144","tokens":196,"n_image_tokens":0,"state_tokens":16,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,6171,10897,449,2605,2560,11,694,279,4881,310,381,5324,513,38136,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3936,449,2605,2002,1132,948,279,6171,3049,4603,9722,421,2002,13,3114,279,12964,2469,381,5324,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[76,97],"option_spans":[[108,130],[136,152],[158,174]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-1.5110081434249878,-2.729757070541382,-1.787302017211914],"probs":[0.48680996894836426,0.14390091598033905,0.3692891597747803],"argmax_index":0,"argmax_id":"insufficient","top2_margin":0.11752080917358398,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.4868,"probabilities":{"permitted":0.1439,"prohibited":0.3693,"insufficient":0.4868}}},"usage":{"input_tokens":196,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_4444dad25e438f128673","arm":"text","set":"fixture","source":"semif_authored144","tokens":190,"n_image_tokens":0,"state_tokens":11,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,30241,6107,303,279,32469,694,4138,4732,424,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,32,18394,579,1440,7225,5214,303,279,32469,321,682,874,25526,18865,13,3114,424,381,10579,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[71,91],"option_spans":[[102,124],[130,146],[152,168]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.042767524719238,-1.0042476654052734,-4.87852668762207],"probs":[0.04482723027467728,0.9357381463050842,0.01943463273346424],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.890910916030407,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.9357,"probabilities":{"prohibited":0.0194,"permitted":0.9357,"insufficient":0.0448}}},"usage":{"input_tokens":190,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_3226202ba1ada2570e9f","arm":"text","set":"fixture","source":"semif_authored144","tokens":193,"n_image_tokens":0,"state_tokens":11,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,30241,6107,303,279,32469,694,4138,4732,424,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,15771,279,10242,5728,11,264,1440,7225,3133,303,279,32469,26,21015,7249,369,37043,13,3114,424,381,10579,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[71,94],"option_spans":[[105,127],[133,149],[155,171]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-2.9149467945098877,-3.4404146671295166,-0.647977352142334],"probs":[0.08895696699619293,0.05259836092591286,0.8584446907043457],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.7694877237081528,"near_tie":false,"gold":"prohibited","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.8584,"probabilities":{"prohibited":0.8584,"permitted":0.0526,"insufficient":0.089}}},"usage":{"input_tokens":193,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_3f783eec5c90812479bc","arm":"text","set":"fixture","source":"semif_authored144","tokens":190,"n_image_tokens":0,"state_tokens":11,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,30241,6107,4732,279,32469,321,4138,4613,424,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,32,18394,579,1440,7225,5214,303,279,32469,321,682,874,25526,18865,13,3114,424,381,10579,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[71,91],"option_spans":[[102,124],[130,146],[152,168]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-3.9627175331115723,-4.233498573303223,-0.7122749090194702],"probs":[0.03627849370241165,0.02767263725399971,0.9360488653182983],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.8997703716158867,"near_tie":false,"gold":"prohibited","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.936,"probabilities":{"permitted":0.0277,"prohibited":0.936,"insufficient":0.0363}}},"usage":{"input_tokens":190,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_ffee861474a9665a093a","arm":"text","set":"fixture","source":"semif_authored144","tokens":191,"n_image_tokens":0,"state_tokens":12,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,30241,16229,264,803,694,874,2046,466,944,3602,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,32,18394,579,1440,7225,5214,303,279,32469,321,682,874,25526,18865,13,3114,424,381,10579,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[72,92],"option_spans":[[103,125],[131,147],[153,169]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-2.038729667663574,-3.8414363861083984,-4.385117530822754],"probs":[0.7932941317558289,0.13077619671821594,0.07592956721782684],"argmax_index":0,"argmax_id":"insufficient","top2_margin":0.6625179350376129,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.7933,"probabilities":{"prohibited":0.0759,"permitted":0.1308,"insufficient":0.7933}}},"usage":{"input_tokens":191,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_8c992e17e4fb54b044f5","arm":"text","set":"fixture","source":"semif_authored144","tokens":178,"n_image_tokens":0,"state_tokens":28,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,66799,375,40988,421,279,64262,572,557,12238,13,47914,417,25,561,66799,375,5891,310,6158,264,64262,572,16089,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,12459,7888,9403,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[88,95],"option_spans":[[106,117],[123,134],[140,156]],"option_ids":["A","B","insufficient"],"logits":[-1.006616234779358,-5.04710578918457,-4.9243998527526855],"probs":[0.9638796448707581,0.016953540965914726,0.0191668588668108],"argmax_index":0,"argmax_id":"A","top2_margin":0.9447127860039473,"near_tie":false,"gold":"A","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.9639,"probabilities":{"B":0.017,"A":0.9639,"insufficient":0.0192}}},"usage":{"input_tokens":178,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_695dd61195ca025c4f4c","arm":"text","set":"fixture","source":"semif_authored144","tokens":179,"n_image_tokens":0,"state_tokens":28,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,66799,375,40988,421,279,64262,572,557,12238,13,47914,417,25,561,66799,375,5891,310,6158,264,64262,572,16089,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,9708,264,3725,9403,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[88,96],"option_spans":[[107,118],[124,135],[141,157]],"option_ids":["A","B","insufficient"],"logits":[-4.802351474761963,-0.21034550666809082,-4.414240837097168],"probs":[0.009884707629680634,0.9755433201789856,0.014571958221495152],"argmax_index":1,"argmax_id":"B","top2_margin":0.9609713619574904,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.9755,"probabilities":{"B":0.9755,"insufficient":0.0146,"A":0.0099}}},"usage":{"input_tokens":179,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_e6155431ab7b0f8170d4","arm":"text","set":"fixture","source":"semif_authored144","tokens":170,"n_image_tokens":0,"state_tokens":20,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,353,628,6158,264,64262,572,16089,13,47914,417,25,561,64262,572,557,12238,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,12459,7888,9403,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[80,87],"option_spans":[[98,109],[115,126],[132,148]],"option_ids":["A","B","insufficient"],"logits":[-4.942187786102295,-0.49033868312835693,-4.7267937660217285],"probs":[0.011360310018062592,0.974548876285553,0.014090766198933125],"argmax_index":1,"argmax_id":"B","top2_margin":0.9604581100866199,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.9745,"probabilities":{"insufficient":0.0141,"B":0.9745,"A":0.0114}}},"usage":{"input_tokens":170,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_b49ecda75086d8c68d2a","arm":"text","set":"fixture","source":"semif_authored144","tokens":170,"n_image_tokens":0,"state_tokens":20,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,66799,375,6486,449,2534,13,47914,417,25,561,64262,572,5435,324,7356,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,12459,7888,9403,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[80,87],"option_spans":[[98,109],[115,126],[132,148]],"option_ids":["A","B","insufficient"],"logits":[-0.032299041748046875,-2.8218135833740234,-1.3120677471160889],"probs":[0.7465178966522217,0.045874301344156265,0.20760783553123474],"argmax_index":0,"argmax_id":"A","top2_margin":0.5389100611209869,"near_tie":false,"gold":"insufficient","gold_correct":false}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.7465,"probabilities":{"B":0.0459,"A":0.7465,"insufficient":0.2076}}},"usage":{"input_tokens":170,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_587ced261c6092cd1325","arm":"text","set":"fixture","source":"semif_authored144","tokens":186,"n_image_tokens":0,"state_tokens":35,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,5813,16561,1204,449,48370,2241,1000,1353,694,20335,55049,3322,424,76964,3482,13,47914,417,25,561,5813,20420,82,279,7808,310,48370,511,279,9761,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,3049,4694,48370,2241,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[95,103],"option_spans":[[114,125],[131,142],[148,164]],"option_ids":["A","B","insufficient"],"logits":[-4.750429630279541,-0.1266489028930664,-4.695819854736328],"probs":[0.009621433913707733,0.9802170991897583,0.010161470621824265],"argmax_index":1,"argmax_id":"B","top2_margin":0.970055628567934,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.9802,"probabilities":{"insufficient":0.0102,"B":0.9802,"A":0.0096}}},"usage":{"input_tokens":186,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_533d4423d311de82b27d","arm":"text","set":"fixture","source":"semif_authored144","tokens":189,"n_image_tokens":0,"state_tokens":35,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,5813,16561,1204,449,48370,2241,1000,1353,694,20335,55049,3322,424,76964,3482,13,47914,417,25,561,5813,20420,82,279,7808,310,48370,511,279,9761,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,9859,1132,364,1928,1345,77735,22973,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[95,106],"option_spans":[[117,128],[134,145],[151,167]],"option_ids":["A","B","insufficient"],"logits":[-0.8166770935058594,-5.049540996551514,-5.107172966003418],"probs":[0.9725650548934937,0.01411267090588808,0.013322323560714722],"argmax_index":0,"argmax_id":"A","top2_margin":0.9584523839876056,"near_tie":false,"gold":"A","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.9726,"probabilities":{"insufficient":0.0133,"A":0.9726,"B":0.0141}}},"usage":{"input_tokens":189,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_2bf540d515a51755b25c","arm":"text","set":"fixture","source":"semif_authored144","tokens":178,"n_image_tokens":0,"state_tokens":27,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,5044,48370,511,279,9761,13,47914,417,25,6712,728,1204,424,1000,1353,11,694,635,524,48370,511,424,3482,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,3049,4694,48370,2241,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[87,95],"option_spans":[[106,117],[123,134],[140,156]],"option_ids":["A","B","insufficient"],"logits":[-0.5531618595123291,-4.939810752868652,-4.751161098480225],"probs":[0.9732663631439209,0.012109722942113876,0.014623923227190971],"argmax_index":0,"argmax_id":"A","top2_margin":0.9586424399167299,"near_tie":false,"gold":"A","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.9733,"probabilities":{"insufficient":0.0146,"B":0.0121,"A":0.9733}}},"usage":{"input_tokens":178,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_fdb9fc214aea2710763a","arm":"text","set":"fixture","source":"semif_authored144","tokens":168,"n_image_tokens":0,"state_tokens":17,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,5813,6486,264,803,13,47914,417,25,561,9761,369,14527,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,3049,4694,48370,2241,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[77,85],"option_spans":[[96,107],[113,124],[130,146]],"option_ids":["A","B","insufficient"],"logits":[-1.2011477947235107,-4.297614574432373,-4.3111252784729],"probs":[0.9175906181335449,0.041483037173748016,0.04092634096741676],"argmax_index":0,"argmax_id":"A","top2_margin":0.8761075809597969,"near_tie":false,"gold":"insufficient","gold_correct":false}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.9176,"probabilities":{"B":0.0415,"A":0.9176,"insufficient":0.0409}}},"usage":{"input_tokens":168,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_f6d68126b43f8c9ed183","arm":"text","set":"fixture","source":"semif_authored144","tokens":181,"n_image_tokens":0,"state_tokens":29,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,5020,7189,421,279,7888,46415,1228,5642,13,47914,417,25,561,5020,11140,46415,16347,7123,694,2640,874,1228,557,1542,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,12459,449,12784,1228,1067,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[89,98],"option_spans":[[109,120],[126,137],[143,159]],"option_ids":["A","B","insufficient"],"logits":[-0.8205393552780151,-4.906612396240234,-4.614716529846191],"probs":[0.9621800780296326,0.016169531270861626,0.021650414913892746],"argmax_index":0,"argmax_id":"A","top2_margin":0.9405296631157398,"near_tie":false,"gold":"A","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.9622,"probabilities":{"insufficient":0.0217,"A":0.9622,"B":0.0162}}},"usage":{"input_tokens":181,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_44bae0c8a1615a096c43","arm":"text","set":"fixture","source":"semif_authored144","tokens":184,"n_image_tokens":0,"state_tokens":29,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,5020,7189,421,279,7888,46415,1228,5642,13,47914,417,25,561,5020,11140,46415,16347,7123,694,2640,874,1228,557,1542,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,8222,1132,449,632,11411,2686,1228,10018,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[89,101],"option_spans":[[112,123],[129,140],[146,162]],"option_ids":["A","B","insufficient"],"logits":[-4.90635871887207,-0.17703676223754883,-4.621670246124268],"probs":[0.008654402568936348,0.979840874671936,0.011504712514579296],"argmax_index":1,"argmax_id":"B","top2_margin":0.9683361621573567,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.9798,"probabilities":{"B":0.9798,"A":0.0087,"insufficient":0.0115}}},"usage":{"input_tokens":184,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_2175379f4c69e6207626","arm":"text","set":"fixture","source":"semif_authored144","tokens":175,"n_image_tokens":0,"state_tokens":23,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,5514,513,46415,16347,7123,26,874,1228,557,1542,13,47914,417,25,561,7888,46415,1228,5642,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,12459,449,12784,1228,1067,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[83,92],"option_spans":[[103,114],[120,131],[137,153]],"option_ids":["A","B","insufficient"],"logits":[-4.489068031311035,-0.5509898662567139,-3.7548296451568604],"probs":[0.01838107779622078,0.9433146715164185,0.038304220885038376],"argmax_index":1,"argmax_id":"B","top2_margin":0.9050104506313801,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.9433,"probabilities":{"insufficient":0.0383,"B":0.9433,"A":0.0184}}},"usage":{"input_tokens":175,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_def9fe59d262357a673a","arm":"text","set":"fixture","source":"semif_authored144","tokens":170,"n_image_tokens":0,"state_tokens":18,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,14231,369,2438,13,47914,417,25,561,35881,369,303,264,25635,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,12459,449,12784,1228,1067,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[78,87],"option_spans":[[98,109],[115,126],[132,148]],"option_ids":["A","B","insufficient"],"logits":[-0.774901270866394,-4.127593040466309,-3.413599967956543],"probs":[0.9037961363792419,0.03162387013435364,0.0645800307393074],"argmax_index":0,"argmax_id":"A","top2_margin":0.8392161056399345,"near_tie":false,"gold":"insufficient","gold_correct":false}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.9038,"probabilities":{"insufficient":0.0646,"A":0.9038,"B":0.0316}}},"usage":{"input_tokens":170,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_712f6ffb763eef7c9a12","arm":"text","set":"fixture","source":"semif_authored144","tokens":181,"n_image_tokens":0,"state_tokens":30,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,6238,16687,264,13340,364,264,2216,15949,1132,310,17061,2272,424,13,47914,417,25,561,6238,809,21735,30869,264,2216,15949,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,12250,264,2216,15949,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[90,98],"option_spans":[[109,120],[126,137],[143,159]],"option_ids":["A","B","insufficient"],"logits":[-4.853603363037109,0.15235722064971924,-4.645056247711182],"probs":[0.0065992530435323715,0.9852712154388428,0.008129533380270004],"argmax_index":1,"argmax_id":"B","top2_margin":0.9771416820585728,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.9853,"probabilities":{"B":0.9853,"A":0.0066,"insufficient":0.0081}}},"usage":{"input_tokens":181,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_e1d610bd14e3d16c09b9","arm":"text","set":"fixture","source":"semif_authored144","tokens":181,"n_image_tokens":0,"state_tokens":30,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,6238,16687,264,13340,364,264,2216,15949,1132,310,17061,2272,424,13,47914,417,25,561,6238,809,21735,30869,264,2216,15949,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,29236,264,2216,15949,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[90,98],"option_spans":[[109,120],[126,137],[143,159]],"option_ids":["A","B","insufficient"],"logits":[-0.6676591634750366,-4.901455879211426,-4.858930587768555],"probs":[0.9712280035018921,0.014080130495131016,0.014691806398332119],"argmax_index":0,"argmax_id":"A","top2_margin":0.95653619710356,"near_tie":false,"gold":"A","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.9712,"probabilities":{"A":0.9712,"insufficient":0.0147,"B":0.0141}}},"usage":{"input_tokens":181,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_22ec57822db50ec121cf","arm":"text","set":"fixture","source":"semif_authored144","tokens":171,"n_image_tokens":0,"state_tokens":20,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,353,18397,30869,264,2216,15949,13,47914,417,25,353,7602,30869,264,2216,15949,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,12250,264,2216,15949,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[80,88],"option_spans":[[99,110],[116,127],[133,149]],"option_ids":["A","B","insufficient"],"logits":[-0.4391871690750122,-4.877831935882568,-4.969343185424805],"probs":[0.9779080748558044,0.01155098620802164,0.010540864430367947],"argmax_index":0,"argmax_id":"A","top2_margin":0.9663570886477828,"near_tie":false,"gold":"A","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.9779,"probabilities":{"insufficient":0.0105,"A":0.9779,"B":0.0116}}},"usage":{"input_tokens":171,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_3e75f2b84e8e08e3d3f2","arm":"text","set":"fixture","source":"semif_authored144","tokens":170,"n_image_tokens":0,"state_tokens":19,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,6238,4874,279,14005,13,47914,417,25,561,3779,682,264,6105,4108,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,12250,264,2216,15949,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[79,87],"option_spans":[[98,109],[115,126],[132,148]],"option_ids":["A","B","insufficient"],"logits":[-1.7104068994522095,-3.0123088359832764,-0.45890820026397705],"probs":[0.2097495049238205,0.0570547953248024,0.7331957221031189],"argmax_index":2,"argmax_id":"insufficient","top2_margin":0.5234462171792984,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.7332,"probabilities":{"B":0.0571,"A":0.2097,"insufficient":0.7332}}},"usage":{"input_tokens":170,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_ebd74aa05324a1556288","arm":"text","set":"fixture","source":"semif_authored144","tokens":186,"n_image_tokens":0,"state_tokens":33,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,6171,16561,310,196718,278,279,6149,3136,1345,9976,1141,3899,13,47914,417,25,561,6171,16561,310,3921,279,3136,321,4786,264,4434,13329,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,9859,310,12368,279,6149,3136,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[93,103],"option_spans":[[114,125],[131,142],[148,164]],"option_ids":["A","B","insufficient"],"logits":[-0.8870011568069458,-5.0312676429748535,-4.969150543212891],"probs":[0.9683107733726501,0.015352624468505383,0.016336528584361076],"argmax_index":0,"argmax_id":"A","top2_margin":0.9519742447882891,"near_tie":false,"gold":"A","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.9683,"probabilities":{"insufficient":0.0163,"A":0.9683,"B":0.0154}}},"usage":{"input_tokens":186,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_d7e84e4245a29817da1f","arm":"text","set":"fixture","source":"semif_authored144","tokens":184,"n_image_tokens":0,"state_tokens":33,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,6171,16561,310,196718,278,279,6149,3136,1345,9976,1141,3899,13,47914,417,25,561,6171,16561,310,3921,279,3136,321,4786,264,4434,13329,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,9859,364,4434,13329,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[93,101],"option_spans":[[112,123],[129,140],[146,162]],"option_ids":["A","B","insufficient"],"logits":[-4.978426456451416,-0.3416473865509033,-4.871684551239014],"probs":[0.009494511410593987,0.9799413681030273,0.010564041323959827],"argmax_index":1,"argmax_id":"B","top2_margin":0.9693773267790675,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.9799,"probabilities":{"insufficient":0.0106,"A":0.0095,"B":0.9799}}},"usage":{"input_tokens":184,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_f41ae66b3c09630c7a42","arm":"text","set":"fixture","source":"semif_authored144","tokens":178,"n_image_tokens":0,"state_tokens":25,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,10466,279,3136,321,4433,264,13329,13,47914,417,25,1737,68,278,279,6149,3136,321,2426,1141,3899,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,9859,310,12368,279,6149,3136,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[85,95],"option_spans":[[106,117],[123,134],[140,156]],"option_ids":["A","B","insufficient"],"logits":[-4.684166431427002,-0.4458194971084595,-4.570445537567139],"probs":[0.014002921991050243,0.9703075885772705,0.01568942703306675],"argmax_index":1,"argmax_id":"B","top2_margin":0.9546181615442038,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.9703,"probabilities":{"insufficient":0.0157,"A":0.014,"B":0.9703}}},"usage":{"input_tokens":178,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_82c2cd97c160ad27951b","arm":"text","set":"fixture","source":"semif_authored144","tokens":171,"n_image_tokens":0,"state_tokens":18,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,3136,11947,264,63080,13,47914,417,25,561,6171,6486,264,70362,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,9859,310,12368,279,6149,3136,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[78,88],"option_spans":[[99,110],[116,127],[133,149]],"option_ids":["A","B","insufficient"],"logits":[-3.4349727630615234,-0.8052483797073364,-3.2694766521453857],"probs":[0.062305573374032974,0.8641750812530518,0.0735192522406578],"argmax_index":1,"argmax_id":"B","top2_margin":0.790655829012394,"near_tie":false,"gold":"insufficient","gold_correct":false}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.8642,"probabilities":{"insufficient":0.0735,"B":0.8642,"A":0.0623}}},"usage":{"input_tokens":171,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_7ada97b6f99f4788e5c7","arm":"text","set":"fixture","source":"semif_authored144","tokens":184,"n_image_tokens":0,"state_tokens":33,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,23662,40988,21867,314,264,12816,3638,694,20335,2640,874,9786,5307,682,978,1801,13,47914,417,25,561,23662,40988,421,279,12816,557,20672,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,12459,449,20672,12816,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[93,101],"option_spans":[[112,123],[129,140],[146,162]],"option_ids":["A","B","insufficient"],"logits":[-5.0054030418396,-0.340732216835022,-4.865033149719238],"probs":[0.009235200472176075,0.9801378846168518,0.010626939125359058],"argmax_index":1,"argmax_id":"B","top2_margin":0.9695109454914927,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.9801,"probabilities":{"B":0.9801,"A":0.0092,"insufficient":0.0106}}},"usage":{"input_tokens":184,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_4d7a003342db5b0a17e3","arm":"text","set":"fixture","source":"semif_authored144","tokens":187,"n_image_tokens":0,"state_tokens":33,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,23662,40988,21867,314,264,12816,3638,694,20335,2640,874,9786,5307,682,978,1801,13,47914,417,25,561,23662,40988,421,279,12816,557,20672,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,12459,21867,440,279,9786,1990,90236,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[93,104],"option_spans":[[115,126],[132,143],[149,165]],"option_ids":["A","B","insufficient"],"logits":[-0.8047996759414673,-4.9891252517700195,-4.941465854644775],"probs":[0.9697359800338745,0.014771477319300175,0.015492524020373821],"argmax_index":0,"argmax_id":"A","top2_margin":0.9542434560135007,"near_tie":false,"gold":"A","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.9697,"probabilities":{"insufficient":0.0155,"A":0.9697,"B":0.0148}}},"usage":{"input_tokens":187,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_19d3d8d0bf773e88be7b","arm":"text","set":"fixture","source":"semif_authored144","tokens":172,"n_image_tokens":0,"state_tokens":21,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,12816,557,20672,13,47914,417,25,561,3638,557,3816,26,279,9786,8198,90236,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,12459,449,20672,12816,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[81,89],"option_spans":[[100,111],[117,128],[134,150]],"option_ids":["A","B","insufficient"],"logits":[-0.7087392807006836,-5.018698692321777,-5.0593671798706055],"probs":[0.9743428826332092,0.013089414685964584,0.012567766942083836],"argmax_index":0,"argmax_id":"A","top2_margin":0.9612534679472446,"near_tie":false,"gold":"A","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.9743,"probabilities":{"A":0.9743,"B":0.0131,"insufficient":0.0126}}},"usage":{"input_tokens":172,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_d6de731cca52359b1c87","arm":"text","set":"fixture","source":"semif_authored144","tokens":170,"n_image_tokens":0,"state_tokens":19,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,23662,6486,1141,803,13,47914,417,25,561,3638,682,264,5604,1324,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,279,8871,12459,449,20672,12816,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[79,87],"option_spans":[[98,109],[115,126],[132,148]],"option_ids":["A","B","insufficient"],"logits":[-1.1553090810775757,-3.2681052684783936,-2.9951555728912354],"probs":[0.7814079523086548,0.094471774995327,0.12412025034427643],"argmax_index":0,"argmax_id":"A","top2_margin":0.6572877019643784,"near_tie":false,"gold":"insufficient","gold_correct":false}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.7814,"probabilities":{"insufficient":0.1241,"A":0.7814,"B":0.0945}}},"usage":{"input_tokens":170,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_b45c63fb92729af27f05","arm":"text","set":"fixture","source":"semif_authored144","tokens":197,"n_image_tokens":0,"state_tokens":32,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,1860,6796,279,6171,18069,33939,1024,35935,13,1848,7348,1292,20335,59584,421,22973,13,561,3150,2640,82265,8848,6575,7634,11,321,874,2843,7634,557,2574,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,7634,310,8307,279,35935,369,4862,2610,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[92,106],"option_spans":[[117,133],[139,155],[161,175]],"option_ids":["contradicted","insufficient","supported"],"logits":[-2.192722797393799,-4.284230709075928,-4.534605026245117],"probs":[0.8199091553688049,0.10125941038131714,0.07883138954639435],"argmax_index":0,"argmax_id":"contradicted","top2_margin":0.7186497449874878,"near_tie":false,"gold":"contradicted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"contradicted","confidence":0.8199,"probabilities":{"contradicted":0.8199,"supported":0.0788,"insufficient":0.1013}}},"usage":{"input_tokens":197,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_46b7029b9a704138b77a","arm":"text","set":"fixture","source":"semif_authored144","tokens":195,"n_image_tokens":0,"state_tokens":32,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,1860,6796,279,6171,18069,33939,1024,35935,13,1848,7348,1292,20335,59584,421,22973,13,561,3150,2640,82265,8848,6575,7634,11,321,874,2843,7634,557,2574,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,6171,6348,7634,383,6796,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[92,104],"option_spans":[[115,131],[137,153],[159,173]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.796773910522461,-4.772089004516602,-1.122923493385315],"probs":[0.0241380725055933,0.024741332978010178,0.9511206150054932],"argmax_index":2,"argmax_id":"supported","top2_margin":0.926379282027483,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.9511,"probabilities":{"insufficient":0.0247,"contradicted":0.0241,"supported":0.9511}}},"usage":{"input_tokens":195,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_caf773b23934fa7362a1","arm":"text","set":"fixture","source":"semif_authored144","tokens":190,"n_image_tokens":0,"state_tokens":25,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,6171,18069,33939,279,35935,383,6796,13,561,4434,7634,3150,40988,421,1292,682,524,47913,424,321,424,8198,2610,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,7634,310,8307,279,35935,369,4862,2610,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[85,99],"option_spans":[[110,126],[132,148],[154,168]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.913613319396973,-4.862112998962402,-1.2163126468658447],"probs":[0.0235898457467556,0.024836556985974312,0.9515735507011414],"argmax_index":2,"argmax_id":"supported","top2_margin":0.926736993715167,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.9516,"probabilities":{"supported":0.9516,"contradicted":0.0236,"insufficient":0.0248}}},"usage":{"input_tokens":190,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_f46f392ef9e9e9df564b","arm":"text","set":"fixture","source":"semif_authored144","tokens":188,"n_image_tokens":0,"state_tokens":23,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,6171,18069,33939,383,6796,13,561,16713,3150,7613,1159,660,2843,7634,4203,321,1503,524,5517,1428,30794,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,7634,310,8307,279,35935,369,4862,2610,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[83,97],"option_spans":[[108,124],[130,146],[152,166]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.310145378112793,-1.1344752311706543,-4.655224323272705],"probs":[0.03898479789495468,0.9334074854850769,0.027607645839452744],"argmax_index":1,"argmax_id":"insufficient","top2_margin":0.8944226875901222,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.9334,"probabilities":{"supported":0.0276,"insufficient":0.9334,"contradicted":0.039}}},"usage":{"input_tokens":188,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_005dcf6d2c0f21d64796","arm":"text","set":"fixture","source":"semif_authored144","tokens":209,"n_image_tokens":0,"state_tokens":26,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,64443,1118,5252,264,10258,269,310,3599,279,71119,13,12851,279,3814,1292,20335,59584,421,7634,13,2233,2843,5307,6513,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,9947,279,64443,579,5362,7634,5307,26,264,28765,60210,1943,279,6575,12816,13,3114,279,10258,269,3599,279,71119,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[86,110],"option_spans":[[121,143],[149,165],[171,187]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.098462104797363,-4.285274028778076,-1.07466721534729],"probs":[0.044645246118307114,0.0370376855134964,0.9183170795440674],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.8736718334257603,"near_tie":false,"gold":"prohibited","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.9183,"probabilities":{"permitted":0.037,"prohibited":0.9183,"insufficient":0.0446}}},"usage":{"input_tokens":209,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_edb708bd0fe5fd67d6ee","arm":"text","set":"fixture","source":"semif_authored144","tokens":210,"n_image_tokens":0,"state_tokens":26,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,64443,1118,5252,264,10258,269,310,3599,279,71119,13,12851,279,3814,1292,20335,59584,421,7634,13,2233,2843,5307,6513,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,2381,411,42140,11176,11,279,1118,12076,5307,369,10649,321,2843,82265,513,11693,13,3114,279,10258,269,3599,279,71119,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[86,111],"option_spans":[[122,144],[150,166],[172,188]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.3408122062683105,-1.4055778980255127,-4.748959541320801],"probs":[0.04880240559577942,0.9187498688697815,0.03244778513908386],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.8699474632740021,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.9187,"probabilities":{"permitted":0.9187,"prohibited":0.0324,"insufficient":0.0488}}},"usage":{"input_tokens":210,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_97a35499bd8717f3e786","arm":"text","set":"fixture","source":"semif_authored144","tokens":203,"n_image_tokens":0,"state_tokens":20,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,64443,1118,15578,71119,4202,11,1179,20335,11340,424,1518,279,3814,13,2233,2843,5307,6513,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,9947,279,64443,579,5362,7634,5307,26,264,28765,60210,1943,279,6575,12816,13,3114,279,10258,269,3599,279,71119,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[80,104],"option_spans":[[115,137],[143,159],[165,181]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.889636516571045,-1.3269375562667847,-4.995432376861572],"probs":[0.026912227272987366,0.9488773345947266,0.024210462346673012],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.9219651073217392,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.9489,"probabilities":{"prohibited":0.0242,"insufficient":0.0269,"permitted":0.9489}}},"usage":{"input_tokens":203,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_31eb4533d2c89959ec46","arm":"text","set":"fixture","source":"semif_authored144","tokens":202,"n_image_tokens":0,"state_tokens":19,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,64443,6348,449,3988,7634,5307,11,694,13468,1141,2144,6130,866,2843,10856,513,2420,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,9947,279,64443,579,5362,7634,5307,26,264,28765,60210,1943,279,6575,12816,13,3114,279,10258,269,3599,279,71119,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[79,103],"option_spans":[[114,136],[142,158],[164,180]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-1.1801484823226929,-3.918215036392212,-3.8496456146240234],"probs":[0.8818478584289551,0.057051416486501694,0.06110065430402756],"argmax_index":0,"argmax_id":"insufficient","top2_margin":0.8207472041249275,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.8818,"probabilities":{"insufficient":0.8818,"prohibited":0.0611,"permitted":0.0571}}},"usage":{"input_tokens":202,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_0aa24a6701ac9c56c4a0","arm":"text","set":"fixture","source":"semif_authored144","tokens":194,"n_image_tokens":0,"state_tokens":34,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,18040,18069,16350,11,1179,20335,59584,279,22973,1518,279,1484,13,47914,417,25,561,18040,18069,16350,321,279,4434,3150,40988,424,8198,303,5176,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,8871,440,4862,2610,16350,7634,26,264,28765,60210,1943,449,6575,12816,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[94,111],"option_spans":[[122,133],[139,150],[156,172]],"option_ids":["A","B","insufficient"],"logits":[-4.4633355140686035,-0.5476541519165039,-4.436190128326416],"probs":[0.019153136759996414,0.9611666798591614,0.019680175930261612],"argmax_index":1,"argmax_id":"B","top2_margin":0.9414865039288998,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.9612,"probabilities":{"insufficient":0.0197,"B":0.9612,"A":0.0192}}},"usage":{"input_tokens":194,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_796d5c0da6eaadae6a3f","arm":"text","set":"fixture","source":"semif_authored144","tokens":188,"n_image_tokens":0,"state_tokens":34,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,561,18040,18069,16350,11,1179,20335,59584,279,22973,1518,279,1484,13,47914,417,25,561,18040,18069,16350,321,279,4434,3150,40988,424,8198,303,5176,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,8871,64708,449,22973,421,557,2843,47913,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[94,105],"option_spans":[[116,127],[133,144],[150,166]],"option_ids":["A","B","insufficient"],"logits":[-0.5251048803329468,-5.043308734893799,-5.097842216491699],"probs":[0.979203462600708,0.010681740008294582,0.010114826261997223],"argmax_index":0,"argmax_id":"A","top2_margin":0.9685217225924134,"near_tie":false,"gold":"A","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.9792,"probabilities":{"B":0.0107,"A":0.9792,"insufficient":0.0101}}},"usage":{"input_tokens":188,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_599b3092b6fd18312d23","arm":"text","set":"fixture","source":"semif_authored144","tokens":187,"n_image_tokens":0,"state_tokens":27,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,43320,557,18069,321,279,4434,3150,2640,7634,8198,303,5176,13,47914,417,25,43320,557,18069,11,1179,20335,47913,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,8871,440,4862,2610,16350,7634,26,264,28765,60210,1943,449,6575,12816,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[87,104],"option_spans":[[115,126],[132,143],[149,165]],"option_ids":["A","B","insufficient"],"logits":[-0.7919070720672607,-4.778350353240967,-4.764895439147949],"probs":[0.9639644026756287,0.017896605655550957,0.018139028921723366],"argmax_index":0,"argmax_id":"A","top2_margin":0.9458253737539053,"near_tie":false,"gold":"A","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.964,"probabilities":{"B":0.0179,"insufficient":0.0181,"A":0.964}}},"usage":{"input_tokens":187,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_960a4bcd27363e6db4b9","arm":"text","set":"fixture","source":"semif_authored144","tokens":189,"n_image_tokens":0,"state_tokens":29,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,25,357,7634,1304,369,9444,694,1141,3665,321,2843,4203,513,7167,13,47914,417,25,561,1484,803,369,16713,1973,7634,3712,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,8871,440,4862,2610,16350,7634,26,264,28765,60210,1943,449,6575,12816,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[89,106],"option_spans":[[117,128],[134,145],[151,167]],"option_ids":["A","B","insufficient"],"logits":[-3.2023634910583496,-2.5894699096679688,-0.6769276857376099],"probs":[0.06518009305000305,0.12030711770057678,0.8145127296447754],"argmax_index":2,"argmax_id":"insufficient","top2_margin":0.6942056119441986,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.8145,"probabilities":{"B":0.1203,"A":0.0652,"insufficient":0.8145}}},"usage":{"input_tokens":189,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_c6a411e7f2bbdf377173","arm":"text","set":"fixture","source":"semif_authored144","tokens":207,"n_image_tokens":0,"state_tokens":43,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,7300,6348,436,2141,10872,310,27237,29238,8928,321,20335,52903,986,4473,43752,13,436,2141,1179,3125,5861,310,27237,1070,13,561,5728,5134,449,16693,35594,43752,23353,874,10872,26,5861,682,874,975,12816,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,5861,682,10872,310,27237,29238,8928,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[103,116],"option_spans":[[127,143],[149,165],[171,185]],"option_ids":["contradicted","insufficient","supported"],"logits":[-1.834521770477295,-4.559390544891357,-4.704526901245117],"probs":[0.8910642266273499,0.0584135428071022,0.050522126257419586],"argmax_index":0,"argmax_id":"contradicted","top2_margin":0.8326506838202477,"near_tie":false,"gold":"contradicted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"contradicted","confidence":0.8911,"probabilities":{"contradicted":0.8911,"insufficient":0.0584,"supported":0.0505}}},"usage":{"input_tokens":207,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_e332aa927d0c2ed044aa","arm":"text","set":"fixture","source":"semif_authored144","tokens":208,"n_image_tokens":0,"state_tokens":43,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,7300,6348,436,2141,10872,310,27237,29238,8928,321,20335,52903,986,4473,43752,13,436,2141,1179,3125,5861,310,27237,1070,13,561,5728,5134,449,16693,35594,43752,23353,874,10872,26,5861,682,874,975,12816,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,436,2141,682,10872,310,27237,29238,8928,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[103,117],"option_spans":[[128,144],[150,166],[172,186]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.952633857727051,-4.8588666915893555,-1.1899545192718506],"probs":[0.022142531350255013,0.024319229647517204,0.9535382986068726],"argmax_index":2,"argmax_id":"supported","top2_margin":0.9292190689593554,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.9535,"probabilities":{"contradicted":0.0221,"insufficient":0.0243,"supported":0.9535}}},"usage":{"input_tokens":208,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_a0128e551d095f3449f8","arm":"text","set":"fixture","source":"semif_authored144","tokens":183,"n_image_tokens":0,"state_tokens":19,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,7300,6486,5861,2050,10872,310,27237,29238,8928,26,279,3150,40988,411,12816,369,2610,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,5861,682,10872,310,27237,29238,8928,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[79,92],"option_spans":[[103,119],[125,141],[147,161]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.9151482582092285,-4.786859035491943,-1.1417920589447021],"probs":[0.021899674087762833,0.024897338822484016,0.953203022480011],"argmax_index":2,"argmax_id":"supported","top2_margin":0.928305683657527,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.9532,"probabilities":{"supported":0.9532,"insufficient":0.0249,"contradicted":0.0219}}},"usage":{"input_tokens":183,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_46d6a0731bd288c370f0","arm":"text","set":"fixture","source":"semif_authored144","tokens":182,"n_image_tokens":0,"state_tokens":18,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,24797,369,6725,303,42270,883,8928,11,694,874,10872,23353,466,43752,3665,513,16713,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,5861,682,10872,310,27237,29238,8928,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[78,91],"option_spans":[[102,118],[124,140],[146,160]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.704029083251953,-1.3284270763397217,-4.715813636779785],"probs":[0.03202031925320625,0.9363344311714172,0.031645189970731735],"argmax_index":1,"argmax_id":"insufficient","top2_margin":0.904314111918211,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.9363,"probabilities":{"insufficient":0.9363,"contradicted":0.032,"supported":0.0316}}},"usage":{"input_tokens":182,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_21ca7a8b970e36b8e20f","arm":"text","set":"fixture","source":"semif_authored144","tokens":214,"n_image_tokens":0,"state_tokens":28,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,16359,3049,4603,279,6228,6436,310,27237,6698,13,561,6228,6436,15734,279,3274,310,449,17313,11,852,682,874,975,14014,10872,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,760,6228,6436,1189,27237,6698,694,1189,524,12993,421,10872,13,8020,18069,2889,2909,1189,27237,13,3114,279,17313,27237,264,1937,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[88,115],"option_spans":[[126,148],[154,170],[176,192]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-3.8915634155273438,-3.633610725402832,-1.526853322982788],"probs":[0.07730824500322342,0.10005832463502884,0.8226334452629089],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.7225751206278801,"near_tie":false,"gold":"prohibited","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.8226,"probabilities":{"prohibited":0.8226,"insufficient":0.0773,"permitted":0.1001}}},"usage":{"input_tokens":214,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_4aa83af95bb69e90bc2d","arm":"text","set":"fixture","source":"semif_authored144","tokens":212,"n_image_tokens":0,"state_tokens":28,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,16359,3049,4603,279,6228,6436,310,27237,6698,13,561,6228,6436,15734,279,3274,310,449,17313,11,852,682,874,975,14014,10872,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,760,6228,6436,1189,12993,1937,14014,310,449,17313,11,321,3604,748,23353,279,17313,14014,10872,13,3114,411,17313,27237,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[88,113],"option_spans":[[124,146],[152,168],[174,190]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-3.130096912384033,-1.4295003414154053,-2.8530025482177734],"probs":[0.12826260924339294,0.7025217413902283,0.16921569406986237],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.5333060473203659,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.7025,"probabilities":{"permitted":0.7025,"prohibited":0.1692,"insufficient":0.1283}}},"usage":{"input_tokens":212,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_cdfa9eed51fafcf09b72","arm":"text","set":"fixture","source":"semif_authored144","tokens":197,"n_image_tokens":0,"state_tokens":11,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,16359,5774,3049,4603,279,17313,310,27237,6698,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,760,6228,6436,1189,27237,6698,694,1189,524,12993,421,10872,13,8020,18069,2889,2909,1189,27237,13,3114,279,17313,27237,264,1937,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[71,98],"option_spans":[[109,131],[137,153],[159,175]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-3.378614902496338,-1.0449756383895874,-3.7232279777526855],"probs":[0.08316764235496521,0.8579084873199463,0.058923810720443726],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.7747408449649811,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.8579,"probabilities":{"insufficient":0.0832,"prohibited":0.0589,"permitted":0.8579}}},"usage":{"input_tokens":197,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_d51b05bfb9d6d7f09331","arm":"text","set":"fixture","source":"semif_authored144","tokens":203,"n_image_tokens":0,"state_tokens":17,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,2014,17313,4138,383,279,5492,11,694,874,14014,23353,466,43752,3712,513,3766,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,760,6228,6436,1189,27237,6698,694,1189,524,12993,421,10872,13,8020,18069,2889,2909,1189,27237,13,3114,279,17313,27237,264,1937,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[77,104],"option_spans":[[115,137],[143,159],[165,181]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-1.474212408065796,-3.4666476249694824,-1.1162221431732178],"probs":[0.389587938785553,0.05312535911798477,0.5572867393493652],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.16769880056381226,"near_tie":false,"gold":"insufficient","gold_correct":false}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.5573,"probabilities":{"prohibited":0.5573,"insufficient":0.3896,"permitted":0.0531}}},"usage":{"input_tokens":203,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_fdd902cc6e65fad7cb4f","arm":"text","set":"fixture","source":"semif_authored144","tokens":201,"n_image_tokens":0,"state_tokens":44,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,5728,27751,2050,43752,539,279,7300,694,52903,3256,866,4473,43752,13,47914,357,25,353,3816,1937,14014,10872,5774,494,279,7300,13,47914,417,25,353,3816,424,1132,494,357,11,321,599,874,975,12816,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,8871,9668,2610,1937,14014,10872,1172,279,10661,43752,5728,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[104,118],"option_spans":[[129,140],[146,157],[163,179]],"option_ids":["A","B","insufficient"],"logits":[-0.9184119701385498,-3.609280586242676,-4.350942611694336],"probs":[0.9089858531951904,0.06164924055337906,0.02936486527323723],"argmax_index":0,"argmax_id":"A","top2_margin":0.8473366126418114,"near_tie":false,"gold":"A","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.909,"probabilities":{"B":0.0616,"A":0.909,"insufficient":0.0294}}},"usage":{"input_tokens":201,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_42384af9ae56beb37ff5","arm":"text","set":"fixture","source":"semif_authored144","tokens":201,"n_image_tokens":0,"state_tokens":44,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,5728,27751,2050,43752,539,279,7300,694,52903,3256,866,4473,43752,13,47914,357,25,353,3816,1937,14014,10872,5774,494,279,7300,13,47914,417,25,353,3816,424,1132,494,357,11,321,599,874,975,12816,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,8871,6483,11324,10872,3905,1132,1472,264,25091,2018,43752,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[104,118],"option_spans":[[129,140],[146,157],[163,179]],"option_ids":["A","B","insufficient"],"logits":[-4.78025484085083,-0.5035197734832764,-4.251823902130127],"probs":[0.013386656530201435,0.9639059901237488,0.022707372903823853],"argmax_index":1,"argmax_id":"B","top2_margin":0.9411986172199249,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.9639,"probabilities":{"insufficient":0.0227,"A":0.0134,"B":0.9639}}},"usage":{"input_tokens":201,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_cdf7f428857e1b841cb3","arm":"text","set":"fixture","source":"semif_authored144","tokens":194,"n_image_tokens":0,"state_tokens":37,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,5728,27751,2050,43752,539,279,7300,694,52903,3256,4473,43752,13,47914,357,25,2921,1132,11324,10872,3571,494,417,13,47914,417,25,561,7300,5774,11340,728,1937,14014,10872,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,8871,9668,2610,1937,14014,10872,1172,279,10661,43752,5728,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[97,111],"option_spans":[[122,133],[139,150],[156,172]],"option_ids":["A","B","insufficient"],"logits":[-4.6944804191589355,-0.4335209131240845,-4.66948127746582],"probs":[0.013716802932322025,0.9722191095352173,0.014064033515751362],"argmax_index":1,"argmax_id":"B","top2_margin":0.9581550760194659,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.9722,"probabilities":{"A":0.0137,"insufficient":0.0141,"B":0.9722}}},"usage":{"input_tokens":194,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_774f7f9245f83c796d44","arm":"text","set":"fixture","source":"semif_authored144","tokens":180,"n_image_tokens":0,"state_tokens":23,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,7300,579,43752,5728,369,16713,11,694,47914,357,321,47914,417,3300,1132,836,4874,321,874,10872,50360,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,8871,9668,2610,1937,14014,10872,1172,279,10661,43752,5728,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[83,97],"option_spans":[[108,119],[125,136],[142,158]],"option_ids":["A","B","insufficient"],"logits":[-5.154650688171387,-4.914322853088379,-1.037321925163269],"probs":[0.015706803649663925,0.01997380703687668,0.9643194079399109],"argmax_index":2,"argmax_id":"insufficient","top2_margin":0.9443456009030342,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.9643,"probabilities":{"B":0.02,"A":0.0157,"insufficient":0.9643}}},"usage":{"input_tokens":180,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_92ec2ffa3dd46ada1335","arm":"text","set":"fixture","source":"semif_authored144","tokens":204,"n_image_tokens":0,"state_tokens":38,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,460,4687,25503,55639,47788,8552,8660,22465,11,694,20335,61228,2919,5383,1442,948,279,27056,369,55639,13,1061,8660,27056,369,55639,1521,2919,10321,424,13,2233,975,5073,369,2945,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,411,27056,77154,364,460,1172,279,10661,4687,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[98,113],"option_spans":[[124,140],[146,162],[168,182]],"option_ids":["contradicted","insufficient","supported"],"logits":[-1.7426729202270508,-4.421910285949707,-4.545462131500244],"probs":[0.8855386972427368,0.060761649161577225,0.05369967222213745],"argmax_index":0,"argmax_id":"contradicted","top2_margin":0.8247770480811596,"near_tie":false,"gold":"contradicted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"contradicted","confidence":0.8855,"probabilities":{"contradicted":0.8855,"insufficient":0.0608,"supported":0.0537}}},"usage":{"input_tokens":204,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_6149a17bc154f9c5b4a0","arm":"text","set":"fixture","source":"semif_authored144","tokens":200,"n_image_tokens":0,"state_tokens":38,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,460,4687,25503,55639,47788,8552,8660,22465,11,694,20335,61228,2919,5383,1442,948,279,27056,369,55639,13,1061,8660,27056,369,55639,1521,2919,10321,424,13,2233,975,5073,369,2945,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,27056,682,264,14109,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[98,109],"option_spans":[[120,136],[142,158],[164,178]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.029994487762451,-4.148406028747559,-1.3690533638000488],"probs":[0.06173570826649666,0.05484170839190483,0.8834225535392761],"argmax_index":2,"argmax_id":"supported","top2_margin":0.8216868452727795,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.8834,"probabilities":{"supported":0.8834,"insufficient":0.0548,"contradicted":0.0617}}},"usage":{"input_tokens":200,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_be866e1dbec5d42d9423","arm":"text","set":"fixture","source":"semif_authored144","tokens":201,"n_image_tokens":0,"state_tokens":35,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,460,4687,25503,55639,47788,8552,8660,22465,11,694,61228,2919,5383,1442,948,55639,13,1061,8660,27056,682,264,14488,18258,14109,11,440,874,2919,5383,466,975,39846,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,411,27056,77154,364,460,1172,279,10661,4687,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[95,110],"option_spans":[[121,137],[143,159],[165,179]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.740293025970459,-4.804949760437012,-1.4272316694259644],"probs":[0.034006115049123764,0.03187696635723114,0.9341168999671936],"argmax_index":2,"argmax_id":"supported","top2_margin":0.9001107849180698,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.9341,"probabilities":{"contradicted":0.034,"insufficient":0.0319,"supported":0.9341}}},"usage":{"input_tokens":201,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_97d2ed2591fc7638806c","arm":"text","set":"fixture","source":"semif_authored144","tokens":199,"n_image_tokens":0,"state_tokens":33,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,460,4687,25503,55639,47788,8552,8660,22465,694,61228,2919,5383,1442,948,55639,13,561,27056,369,55639,11,694,279,5073,321,9084,314,2919,5383,513,38136,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,411,27056,77154,364,460,1172,279,10661,4687,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[93,108],"option_spans":[[119,135],[141,157],[163,177]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.236184120178223,-1.7604689598083496,-2.744403839111328],"probs":[0.05768606439232826,0.6858992576599121,0.25641465187072754],"argmax_index":1,"argmax_id":"insufficient","top2_margin":0.42948460578918457,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.6859,"probabilities":{"insufficient":0.6859,"supported":0.2564,"contradicted":0.0577}}},"usage":{"input_tokens":199,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_4533dd91267b4d815690","arm":"text","set":"fixture","source":"semif_authored144","tokens":205,"n_image_tokens":0,"state_tokens":19,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,8660,5984,24007,682,264,14488,21531,321,1048,5435,3568,66564,13,11396,12688,513,10457,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,90943,7602,8660,5984,13,20195,424,364,264,14488,21531,11,3525,421,3568,66564,2590,3520,424,92502,13,3114,411,460,381,11330,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[79,106],"option_spans":[[117,139],[145,161],[167,183]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.196014404296875,-4.478527069091797,-0.8603951930999756],"probs":[0.033501219004392624,0.025256134569644928,0.9412426948547363],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.9077414758503437,"near_tie":false,"gold":"prohibited","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.9412,"probabilities":{"permitted":0.0253,"insufficient":0.0335,"prohibited":0.9412}}},"usage":{"input_tokens":205,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_d880e65fef5e2993fdd0","arm":"text","set":"fixture","source":"semif_authored144","tokens":202,"n_image_tokens":0,"state_tokens":19,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,8660,5984,24007,682,264,14488,21531,321,1048,5435,3568,66564,13,11396,12688,513,10457,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,16156,61649,5984,1442,413,8660,466,9238,440,66564,26,13468,2876,34282,279,21531,4531,13,3114,411,460,381,11330,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[79,103],"option_spans":[[114,136],[142,158],[164,180]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-1.3330128192901611,-1.4348037242889404,0.16625308990478516],"probs":[0.1567000448703766,0.14153437316417694,0.7017655372619629],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.5450654923915863,"near_tie":false,"gold":"permitted","gold_correct":false}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.7018,"probabilities":{"permitted":0.1415,"insufficient":0.1567,"prohibited":0.7018}}},"usage":{"input_tokens":202,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_90494c76078ffc3c717a","arm":"text","set":"fixture","source":"semif_authored144","tokens":202,"n_image_tokens":0,"state_tokens":16,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,8660,24007,682,264,14488,21531,321,279,24865,40988,874,66564,557,3568,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,90943,7602,8660,5984,13,20195,424,364,264,14488,21531,11,3525,421,3568,66564,2590,3520,424,92502,13,3114,411,460,381,11330,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[76,103],"option_spans":[[114,136],[142,158],[164,180]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.5208516120910645,-1.3150945901870728,-4.856809616088867],"probs":[0.037894826382398605,0.9350236058235168,0.027081655338406563],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.8971287794411182,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.935,"probabilities":{"permitted":0.935,"insufficient":0.0379,"prohibited":0.0271}}},"usage":{"input_tokens":202,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_9ac5cc82c4126c91174a","arm":"text","set":"fixture","source":"semif_authored144","tokens":201,"n_image_tokens":0,"state_tokens":15,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,8660,24007,369,61649,11,694,3315,66564,557,3568,369,524,3750,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,90943,7602,8660,5984,13,20195,424,364,264,14488,21531,11,3525,421,3568,66564,2590,3520,424,92502,13,3114,411,460,381,11330,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[75,102],"option_spans":[[113,135],[141,157],[163,179]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-1.2856658697128296,-4.593633651733398,-4.7447896003723145],"probs":[0.9362877011299133,0.03425920009613037,0.029453102499246597],"argmax_index":0,"argmax_id":"insufficient","top2_margin":0.902028501033783,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.9363,"probabilities":{"permitted":0.0343,"insufficient":0.9363,"prohibited":0.0295}}},"usage":{"input_tokens":201,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_386e10b1c1ce906473a4","arm":"text","set":"fixture","source":"semif_authored144","tokens":205,"n_image_tokens":0,"state_tokens":51,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,16156,3200,35614,364,25530,680,1098,11,3525,421,7167,31548,2590,801,1689,1386,13,47914,357,25,2921,19878,557,3200,1521,314,264,25530,85093,321,5469,279,2483,11618,13,47914,417,25,29682,557,3200,1521,314,264,25530,85093,694,35538,279,11618,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,19878,34919,1172,279,10661,4531,321,39846,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[111,122],"option_spans":[[133,144],[150,161],[167,183]],"option_ids":["A","B","insufficient"],"logits":[-0.7328402996063232,-4.812448501586914,-4.984321117401123],"probs":[0.9697842001914978,0.016403023153543472,0.013812756165862083],"argmax_index":0,"argmax_id":"A","top2_margin":0.9533811770379543,"near_tie":false,"gold":"A","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.9698,"probabilities":{"B":0.0164,"A":0.9698,"insufficient":0.0138}}},"usage":{"input_tokens":205,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_4da6aedc1139224eeff2","arm":"text","set":"fixture","source":"semif_authored144","tokens":208,"n_image_tokens":0,"state_tokens":51,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,16156,3200,35614,364,25530,680,1098,11,3525,421,7167,31548,2590,801,1689,1386,13,47914,357,25,2921,19878,557,3200,1521,314,264,25530,85093,321,5469,279,2483,11618,13,47914,417,25,29682,557,3200,1521,314,264,25530,85093,694,35538,279,11618,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,19878,90354,11352,539,279,11618,39846,8552,279,85093,4531,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[111,125],"option_spans":[[136,147],[153,164],[170,186]],"option_ids":["A","B","insufficient"],"logits":[-4.900668144226074,-0.3466099500656128,-4.968170166015625],"probs":[0.010314389131963253,0.9800445437431335,0.009641125798225403],"argmax_index":1,"argmax_id":"B","top2_margin":0.9697301546111703,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.98,"probabilities":{"insufficient":0.0096,"B":0.98,"A":0.0103}}},"usage":{"input_tokens":208,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_a22dab8ba073ff6cdb8d","arm":"text","set":"fixture","source":"semif_authored144","tokens":201,"n_image_tokens":0,"state_tokens":47,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,16156,3200,35614,364,25530,680,1098,11,3525,421,7167,31548,2590,801,1689,1386,13,47914,357,25,2921,3200,85093,13505,19878,35538,279,11618,13,47914,417,25,2921,3200,85093,13505,19878,682,279,11618,13,11396,680,1098,513,25530,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,19878,34919,1172,279,10661,4531,321,39846,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[107,118],"option_spans":[[129,140],[146,157],[163,179]],"option_ids":["A","B","insufficient"],"logits":[-4.822176456451416,-0.6027024984359741,-4.899712085723877],"probs":[0.014301427640020847,0.9724641442298889,0.013234456069767475],"argmax_index":1,"argmax_id":"B","top2_margin":0.9581627165898681,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.9725,"probabilities":{"insufficient":0.0132,"B":0.9725,"A":0.0143}}},"usage":{"input_tokens":201,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_42c54fa2ad45743e24d8","arm":"text","set":"fixture","source":"semif_authored144","tokens":188,"n_image_tokens":0,"state_tokens":34,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,16156,3200,35614,364,25530,680,1098,11,3525,421,7167,31548,2590,801,1689,1386,13,47914,357,321,47914,417,1830,3200,35614,694,3300,13468,85093,9417,6130,11618,1928,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,19878,34919,1172,279,10661,4531,321,39846,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[94,105],"option_spans":[[116,127],[133,144],[150,166]],"option_ids":["A","B","insufficient"],"logits":[-5.026548862457275,-4.920945167541504,-0.8071836233139038],"probs":[0.01426499243825674,0.015853846445679665,0.9698811769485474],"argmax_index":2,"argmax_id":"insufficient","top2_margin":0.9540273305028677,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.9699,"probabilities":{"A":0.0143,"insufficient":0.9699,"B":0.0159}}},"usage":{"input_tokens":188,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_f3d13cf184f916677def","arm":"text","set":"fixture","source":"semif_authored144","tokens":208,"n_image_tokens":0,"state_tokens":40,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,17471,1433,2640,279,17189,369,303,5638,13,561,29128,14991,2640,424,682,2047,5638,13,11396,6473,513,1428,13,561,16713,11176,2640,279,29128,14991,11241,3602,10856,14910,1439,1330,7916,11712,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,1172,279,16713,11176,11,279,17189,369,4862,303,5638,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[100,117],"option_spans":[[128,144],[150,166],[172,186]],"option_ids":["contradicted","insufficient","supported"],"logits":[-1.7683124542236328,-4.3024492263793945,-4.513944625854492],"probs":[0.8744791150093079,0.06937257200479507,0.056148286908864975],"argmax_index":0,"argmax_id":"contradicted","top2_margin":0.8051065430045128,"near_tie":false,"gold":"contradicted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"contradicted","confidence":0.8745,"probabilities":{"contradicted":0.8745,"insufficient":0.0694,"supported":0.0561}}},"usage":{"input_tokens":208,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_1105577da4c8dad4609d","arm":"text","set":"fixture","source":"semif_authored144","tokens":205,"n_image_tokens":0,"state_tokens":40,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,17471,1433,2640,279,17189,369,303,5638,13,561,29128,14991,2640,424,682,2047,5638,13,11396,6473,513,1428,13,561,16713,11176,2640,279,29128,14991,11241,3602,10856,14910,1439,1330,7916,11712,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,17471,1433,6473,279,17189,303,5638,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[100,114],"option_spans":[[125,141],[147,163],[169,183]],"option_ids":["contradicted","insufficient","supported"],"logits":[0.353561669588089,-1.526219367980957,-1.0014375448226929],"probs":[0.7089326977729797,0.10819981247186661,0.18286742269992828],"argmax_index":0,"argmax_id":"contradicted","top2_margin":0.5260652750730515,"near_tie":false,"gold":"supported","gold_correct":false}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"contradicted","confidence":0.7089,"probabilities":{"contradicted":0.7089,"insufficient":0.1082,"supported":0.1829}}},"usage":{"input_tokens":205,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_0ebd299739302d8b057b","arm":"text","set":"fixture","source":"semif_authored144","tokens":201,"n_image_tokens":0,"state_tokens":33,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,17471,1433,2640,279,17189,682,2047,5638,26,279,29128,14991,2640,424,369,303,5638,13,11396,6473,513,1428,11,321,279,14991,11241,24979,1172,279,11176,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,1172,279,16713,11176,11,279,17189,369,4862,303,5638,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[93,110],"option_spans":[[121,137],[143,159],[165,179]],"option_ids":["contradicted","insufficient","supported"],"logits":[-0.24593567848205566,0.10532855987548828,-1.331939697265625],"probs":[0.3625256419181824,0.5150991678237915,0.12237513810396194],"argmax_index":1,"argmax_id":"insufficient","top2_margin":0.15257352590560913,"near_tie":false,"gold":"supported","gold_correct":false}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.5151,"probabilities":{"contradicted":0.3625,"insufficient":0.5151,"supported":0.1224}}},"usage":{"input_tokens":201,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_eafc22c8c40df3932a8e","arm":"text","set":"fixture","source":"semif_authored144","tokens":189,"n_image_tokens":0,"state_tokens":21,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,11176,6486,279,14991,10304,11,694,279,1428,14991,321,17471,44788,10379,364,279,17189,513,38136,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,1172,279,16713,11176,11,279,17189,369,4862,303,5638,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[81,98],"option_spans":[[109,125],[131,147],[153,167]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.13878870010376,-1.5169490575790405,-3.8778045177459717],"probs":[0.06226951628923416,0.8568917512893677,0.08083871752023697],"argmax_index":1,"argmax_id":"insufficient","top2_margin":0.7760530337691307,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.8569,"probabilities":{"supported":0.0808,"contradicted":0.0623,"insufficient":0.8569}}},"usage":{"input_tokens":189,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_0aa1447680e4f25862bb","arm":"text","set":"fixture","source":"semif_authored144","tokens":208,"n_image_tokens":0,"state_tokens":23,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,17471,42787,6473,279,6875,16051,430,6945,13,561,8239,45366,6473,424,430,52269,13,11396,6473,513,1428,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,15604,6875,6681,948,279,16051,369,12076,6945,26,303,24979,11,279,8239,45366,34282,279,17471,42787,13,3114,424,381,5698,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[83,109],"option_spans":[[120,142],[148,164],[170,186]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-3.5207297801971436,-3.441898822784424,-1.4016138315200806],"probs":[0.09610029309988022,0.10398255288600922,0.7999171614646912],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.695934608578682,"near_tie":false,"gold":"prohibited","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.7999,"probabilities":{"insufficient":0.0961,"prohibited":0.7999,"permitted":0.104}}},"usage":{"input_tokens":208,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_980e76145c925747220c","arm":"text","set":"fixture","source":"semif_authored144","tokens":209,"n_image_tokens":0,"state_tokens":23,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,17471,42787,6473,279,6875,16051,430,6945,13,561,8239,45366,6473,424,430,52269,13,11396,6473,513,1428,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,15604,6875,6681,948,279,16051,369,12076,6945,26,1172,411,10242,11176,11,279,17471,42787,34282,279,45366,13,3114,424,381,5698,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[83,110],"option_spans":[[121,143],[149,165],[171,187]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-3.040069580078125,-1.4542186260223389,-3.757823944091797],"probs":[0.15695405006408691,0.7664765119552612,0.07656949758529663],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.6095224618911743,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.7665,"probabilities":{"insufficient":0.157,"prohibited":0.0766,"permitted":0.7665}}},"usage":{"input_tokens":209,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_4aee37a6a155747766aa","arm":"text","set":"fixture","source":"semif_authored144","tokens":206,"n_image_tokens":0,"state_tokens":21,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,17471,42787,6473,279,16051,52269,11,1345,279,8239,45366,6473,424,6945,13,11396,6473,513,1428,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,15604,6875,6681,948,279,16051,369,12076,6945,26,303,24979,11,279,8239,45366,34282,279,17471,42787,13,3114,424,381,5698,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[81,107],"option_spans":[[118,140],[146,162],[168,184]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-3.691023349761963,-1.137373685836792,-4.428318500518799],"probs":[0.06977228820323944,0.896848201751709,0.03337949514389038],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.8270759135484695,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.8968,"probabilities":{"permitted":0.8968,"prohibited":0.0334,"insufficient":0.0698}}},"usage":{"input_tokens":206,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_6db17cc22bd656d78558","arm":"text","set":"fixture","source":"semif_authored144","tokens":199,"n_image_tokens":0,"state_tokens":14,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,3150,32726,279,17471,42787,321,45366,694,16229,13468,16051,2552,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,15604,6875,6681,948,279,16051,369,12076,6945,26,303,24979,11,279,8239,45366,34282,279,17471,42787,13,3114,424,381,5698,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[74,100],"option_spans":[[111,133],[139,155],[161,177]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-0.9805396795272827,-4.814464569091797,-4.593575477600098],"probs":[0.9536575078964233,0.020622441545128822,0.02572000026702881],"argmax_index":0,"argmax_id":"insufficient","top2_margin":0.9279375076293945,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.9537,"probabilities":{"prohibited":0.0257,"permitted":0.0206,"insufficient":0.9537}}},"usage":{"input_tokens":199,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_ef3e0f8e657bf59f8a29","arm":"text","set":"fixture","source":"semif_authored144","tokens":200,"n_image_tokens":0,"state_tokens":46,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,2381,411,3274,11,279,1428,8239,50802,11241,59610,948,424,24979,440,6040,13,47914,357,25,12485,2640,5354,11,694,279,8239,50802,2640,524,5354,13,47914,417,25,12485,2640,524,5354,11,694,279,8239,50802,2640,5354,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,8871,5354,3956,310,279,25074,8239,50802,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[106,117],"option_spans":[[128,139],[145,156],[162,178]],"option_ids":["A","B","insufficient"],"logits":[-0.7696751952171326,-2.443060874938965,-2.40246844291687],"probs":[0.7230689525604248,0.13565555214881897,0.14127543568611145],"argmax_index":0,"argmax_id":"A","top2_margin":0.5817935168743134,"near_tie":false,"gold":"B","gold_correct":false}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.7231,"probabilities":{"A":0.7231,"insufficient":0.1413,"B":0.1357}}},"usage":{"input_tokens":200,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_f756e1cb04e11766341e","arm":"text","set":"fixture","source":"semif_authored144","tokens":206,"n_image_tokens":0,"state_tokens":46,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,2381,411,3274,11,279,1428,8239,50802,11241,59610,948,424,24979,440,6040,13,47914,357,25,12485,2640,5354,11,694,279,8239,50802,2640,524,5354,13,47914,417,25,12485,2640,524,5354,11,694,279,8239,50802,2640,5354,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,15771,279,10242,11176,11,6040,11241,24979,13,8167,279,8871,5354,3956,310,6040,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[106,123],"option_spans":[[134,145],[151,162],[168,184]],"option_ids":["A","B","insufficient"],"logits":[0.3937191367149353,-0.9481428861618042,-1.5279929637908936],"probs":[0.7103711366653442,0.18566159904003143,0.10396719723939896],"argmax_index":0,"argmax_id":"A","top2_margin":0.5247095376253128,"near_tie":false,"gold":"A","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.7104,"probabilities":{"insufficient":0.104,"A":0.7104,"B":0.1857}}},"usage":{"input_tokens":206,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_d167e51bb1f499099645","arm":"text","set":"fixture","source":"semif_authored144","tokens":186,"n_image_tokens":0,"state_tokens":32,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,8239,50802,11241,24979,13,47914,357,25,561,50802,2640,5354,321,6040,2640,524,5354,13,47914,417,25,561,50802,2640,524,5354,321,6040,2640,5354,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,8871,5354,3956,310,279,25074,8239,50802,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[92,103],"option_spans":[[114,125],[131,142],[148,164]],"option_ids":["A","B","insufficient"],"logits":[-0.8636826276779175,-3.1702027320861816,-1.5368220806121826],"probs":[0.6212291717529297,0.061878953129053116,0.3168918490409851],"argmax_index":0,"argmax_id":"A","top2_margin":0.3043373227119446,"near_tie":false,"gold":"A","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.6212,"probabilities":{"A":0.6212,"B":0.0619,"insufficient":0.3169}}},"usage":{"input_tokens":186,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_674ca4e9346216255466","arm":"text","set":"fixture","source":"semif_authored144","tokens":176,"n_image_tokens":0,"state_tokens":22,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,8239,50802,11241,24979,11,694,47914,357,321,47914,417,3300,1132,2313,4874,1973,50802,466,6040,2552,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,8871,5354,3956,310,279,25074,8239,50802,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[82,93],"option_spans":[[104,115],[121,132],[138,154]],"option_ids":["A","B","insufficient"],"logits":[-5.1939473152160645,-4.9289398193359375,-0.8596611022949219],"probs":[0.012726865708827972,0.016588713973760605,0.9706844091415405],"argmax_index":2,"argmax_id":"insufficient","top2_margin":0.9540956951677799,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.9707,"probabilities":{"insufficient":0.9707,"B":0.0166,"A":0.0127}}},"usage":{"input_tokens":176,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_1df851a202994a07f123","arm":"text","set":"fixture","source":"semif_authored144","tokens":194,"n_image_tokens":0,"state_tokens":31,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,7003,1375,1908,7225,2107,279,2450,3425,321,279,1156,8214,13,561,9403,3150,40988,279,2450,3425,11136,321,20335,2640,279,8214,557,524,12238,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,7003,1375,1908,369,4434,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[91,103],"option_spans":[[114,130],[136,152],[158,172]],"option_ids":["contradicted","insufficient","supported"],"logits":[-1.4828146696090698,-4.265439510345459,-4.647427558898926],"probs":[0.9057098031044006,0.056041598320007324,0.03824856877326965],"argmax_index":0,"argmax_id":"contradicted","top2_margin":0.8496682047843933,"near_tie":false,"gold":"contradicted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"contradicted","confidence":0.9057,"probabilities":{"contradicted":0.9057,"insufficient":0.056,"supported":0.0382}}},"usage":{"input_tokens":194,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_b5e6788d95b8395d2888","arm":"text","set":"fixture","source":"semif_authored144","tokens":193,"n_image_tokens":0,"state_tokens":31,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,7003,1375,1908,7225,2107,279,2450,3425,321,279,1156,8214,13,561,9403,3150,40988,279,2450,3425,11136,321,20335,2640,279,8214,557,524,12238,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,2450,3425,998,12238,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[91,102],"option_spans":[[113,129],[135,151],[157,171]],"option_ids":["contradicted","insufficient","supported"],"logits":[-5.115021705627441,-5.089676380157471,-0.9650599360466003],"probs":[0.015277144499123096,0.01566929742693901,0.9690535068511963],"argmax_index":2,"argmax_id":"supported","top2_margin":0.9533842094242573,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.9691,"probabilities":{"insufficient":0.0157,"contradicted":0.0153,"supported":0.9691}}},"usage":{"input_tokens":193,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_32f36cd0fbbfb3604586","arm":"text","set":"fixture","source":"semif_authored144","tokens":183,"n_image_tokens":0,"state_tokens":20,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,7003,1375,1908,7225,279,2450,3425,321,1156,8214,13,561,9403,3150,40988,421,2107,11136,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,7003,1375,1908,369,4434,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[80,92],"option_spans":[[103,119],[125,141],[147,161]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.865823268890381,-4.615105152130127,-1.2933645248413086],"probs":[0.026392901316285133,0.0339135006070137,0.9396935701370239],"argmax_index":2,"argmax_id":"supported","top2_margin":0.9057800695300102,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.9397,"probabilities":{"supported":0.9397,"insufficient":0.0339,"contradicted":0.0264}}},"usage":{"input_tokens":183,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_843185706adff0c60f8b","arm":"text","set":"fixture","source":"semif_authored144","tokens":184,"n_image_tokens":0,"state_tokens":21,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,8855,7225,2450,3425,321,264,1156,8214,11,694,279,9403,3150,6486,874,1928,883,2892,1455,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,7003,1375,1908,369,4434,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[81,93],"option_spans":[[104,120],[126,142],[148,162]],"option_ids":["contradicted","insufficient","supported"],"logits":[-2.950343370437622,-1.2090286016464233,-4.406998157501221],"probs":[0.14413680136203766,0.8222772479057312,0.033585961908102036],"argmax_index":1,"argmax_id":"insufficient","top2_margin":0.6781404465436935,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.8223,"probabilities":{"contradicted":0.1441,"supported":0.0336,"insufficient":0.8223}}},"usage":{"input_tokens":184,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_b531dd603bf080dab820","arm":"text","set":"fixture","source":"semif_authored144","tokens":199,"n_image_tokens":0,"state_tokens":15,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,28994,16713,279,23756,11048,694,20335,1467,524,7892,279,34180,640,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,27654,571,1534,1375,1908,6681,948,2107,279,23756,11048,321,34180,640,599,978,16713,13,3114,1534,1375,1908,381,11454,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[75,100],"option_spans":[[111,133],[139,155],[161,177]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.511086940765381,-4.668023109436035,-0.8683821558952332],"probs":[0.024968935176730156,0.021342413499951363,0.9536886215209961],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.9287196863442659,"near_tie":false,"gold":"prohibited","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.9537,"probabilities":{"prohibited":0.9537,"permitted":0.0213,"insufficient":0.025}}},"usage":{"input_tokens":199,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_be2dd67bc733c1213bef","arm":"text","set":"fixture","source":"semif_authored144","tokens":202,"n_image_tokens":0,"state_tokens":15,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,28994,16713,279,23756,11048,694,20335,1467,524,7892,279,34180,640,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,27654,571,279,14053,40793,948,279,11048,682,978,16713,26,279,34180,640,369,524,2483,364,411,40793,13,3114,411,40793,381,11454,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[75,103],"option_spans":[[114,136],[142,158],[164,180]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-3.9849371910095215,-1.3248473405838013,-4.729798793792725],"probs":[0.06340198963880539,0.9064947366714478,0.03010326623916626],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.8430927470326424,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.9065,"probabilities":{"prohibited":0.0301,"permitted":0.9065,"insufficient":0.0634}}},"usage":{"input_tokens":202,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_b8e2750307df9761e716","arm":"text","set":"fixture","source":"semif_authored144","tokens":196,"n_image_tokens":0,"state_tokens":12,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,28994,16713,2107,279,23756,11048,321,279,34180,640,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,27654,571,1534,1375,1908,6681,948,2107,279,23756,11048,321,34180,640,599,978,16713,13,3114,1534,1375,1908,381,11454,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[72,97],"option_spans":[[108,130],[136,152],[158,174]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.9243388175964355,-1.3660871982574463,-5.017913341522217],"probs":[0.02701793797314167,0.9483775496482849,0.02460443042218685],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.9213596116751432,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.9484,"probabilities":{"permitted":0.9484,"prohibited":0.0246,"insufficient":0.027}}},"usage":{"input_tokens":196,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_5d31b4bf2d0fc193f53b","arm":"text","set":"fixture","source":"semif_authored144","tokens":201,"n_image_tokens":0,"state_tokens":17,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,28994,6473,3133,383,279,1375,1908,694,16229,874,9403,3150,364,2892,1455,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,27654,571,1534,1375,1908,6681,948,2107,279,23756,11048,321,34180,640,599,978,16713,13,3114,1534,1375,1908,381,11454,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[77,102],"option_spans":[[113,135],[141,157],[163,179]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-3.932516098022461,-4.779226303100586,-0.9209278225898743],"probs":[0.045980241149663925,0.019717402756214142,0.9343023896217346],"argmax_index":2,"argmax_id":"prohibited","top2_margin":0.8883221484720707,"near_tie":false,"gold":"insufficient","gold_correct":false}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"prohibited","confidence":0.9343,"probabilities":{"permitted":0.0197,"insufficient":0.046,"prohibited":0.9343}}},"usage":{"input_tokens":201,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_a9230edfc6ec9e524af0","arm":"text","set":"fixture","source":"semif_authored144","tokens":185,"n_image_tokens":0,"state_tokens":36,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,32,4434,10142,6129,7225,2107,264,24024,321,264,37843,1830,13,47914,357,25,11396,9017,513,12037,13,47914,417,25,8020,279,24024,369,12037,26,279,37843,1830,369,26334,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,4434,10142,6129,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[96,102],"option_spans":[[113,124],[130,141],[147,163]],"option_ids":["A","B","insufficient"],"logits":[-0.5274320840835571,-4.713111400604248,-4.5898919105529785],"probs":[0.9685994386672974,0.014734205789864063,0.016666341572999954],"argmax_index":0,"argmax_id":"A","top2_margin":0.9519330970942974,"near_tie":false,"gold":"A","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.9686,"probabilities":{"B":0.0147,"insufficient":0.0167,"A":0.9686}}},"usage":{"input_tokens":185,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_c100e84585149a04be49","arm":"text","set":"fixture","source":"semif_authored144","tokens":191,"n_image_tokens":0,"state_tokens":36,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,32,4434,10142,6129,7225,2107,264,24024,321,264,37843,1830,13,47914,357,25,11396,9017,513,12037,13,47914,417,25,8020,279,24024,369,12037,26,279,37843,1830,369,26334,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,6129,8222,279,24024,694,7167,279,37843,1830,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[96,108],"option_spans":[[119,130],[136,147],[153,169]],"option_ids":["A","B","insufficient"],"logits":[-4.781346797943115,-0.39190852642059326,-4.727744102478027],"probs":[0.012099185958504677,0.9751353859901428,0.0127654317766428],"argmax_index":1,"argmax_id":"B","top2_margin":0.9623699542135,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.9751,"probabilities":{"B":0.9751,"A":0.0121,"insufficient":0.0128}}},"usage":{"input_tokens":191,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_579beed184ffc1104aa9","arm":"text","set":"fixture","source":"semif_authored144","tokens":182,"n_image_tokens":0,"state_tokens":33,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,32,4434,6129,7225,2107,9017,13,47914,357,25,561,24024,369,12037,694,279,37843,1830,369,26334,13,47914,417,25,11396,279,24024,321,37843,1830,513,12037,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,4434,10142,6129,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[93,99],"option_spans":[[110,121],[127,138],[144,160]],"option_ids":["A","B","insufficient"],"logits":[-4.678561687469482,-0.03699076175689697,-4.696479320526123],"probs":[0.009461687877774239,0.981244683265686,0.009293666109442711],"argmax_index":1,"argmax_id":"B","top2_margin":0.9717829953879118,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.9812,"probabilities":{"B":0.9812,"A":0.0095,"insufficient":0.0093}}},"usage":{"input_tokens":182,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_c4e9bd463b6fde5be743","arm":"text","set":"fixture","source":"semif_authored144","tokens":170,"n_image_tokens":0,"state_tokens":21,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,32,4434,6129,7225,2107,9017,13,47914,357,321,47914,417,2873,1132,6129,14865,321,874,19375,1928,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,3270,279,4434,10142,6129,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[81,87],"option_spans":[[98,109],[115,126],[132,148]],"option_ids":["A","B","insufficient"],"logits":[-5.058294773101807,-4.7816033363342285,-0.9253431558609009],"probs":[0.015460613183677197,0.020388828590512276,0.9641505479812622],"argmax_index":2,"argmax_id":"insufficient","top2_margin":0.9437617193907499,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.9642,"probabilities":{"B":0.0204,"insufficient":0.9642,"A":0.0155}}},"usage":{"input_tokens":170,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_b863998d4be2721a5dfd","arm":"text","set":"fixture","source":"semif_authored144","tokens":205,"n_image_tokens":0,"state_tokens":38,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,28109,5728,48233,29201,310,279,303,34712,24279,948,424,628,2649,279,944,26,958,279,4732,24279,1132,948,279,303,34712,24279,4021,13,11396,33681,7440,781,628,2649,411,12368,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,28109,5728,48233,411,12368,310,279,4732,24279,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[98,114],"option_spans":[[125,141],[147,163],[169,183]],"option_ids":["contradicted","insufficient","supported"],"logits":[-2.1538548469543457,-4.10241174697876,-4.499169826507568],"probs":[0.8075610399246216,0.11506092548370361,0.07737808674573898],"argmax_index":0,"argmax_id":"contradicted","top2_margin":0.692500114440918,"near_tie":false,"gold":"contradicted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"contradicted","confidence":0.8076,"probabilities":{"supported":0.0774,"insufficient":0.1151,"contradicted":0.8076}}},"usage":{"input_tokens":205,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_37cebc3792e98049439f","arm":"text","set":"fixture","source":"semif_authored144","tokens":202,"n_image_tokens":0,"state_tokens":38,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,28109,5728,48233,29201,310,279,303,34712,24279,948,424,628,2649,279,944,26,958,279,4732,24279,1132,948,279,303,34712,24279,4021,13,11396,33681,7440,781,628,2649,411,12368,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,4732,24279,628,2649,411,12368,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[98,111],"option_spans":[[122,138],[144,160],[166,180]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.804859638214111,-4.672128677368164,-0.8327572345733643],"probs":[0.01810348406434059,0.02067314274609089,0.9612233638763428],"argmax_index":2,"argmax_id":"supported","top2_margin":0.9405502211302519,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.9612,"probabilities":{"insufficient":0.0207,"supported":0.9612,"contradicted":0.0181}}},"usage":{"input_tokens":202,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_64d63f86e328aff1481c","arm":"text","set":"fixture","source":"semif_authored144","tokens":210,"n_image_tokens":0,"state_tokens":43,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,28109,5728,48233,29201,310,279,303,34712,24279,948,424,628,2649,279,944,26,958,279,4732,24279,1132,948,279,303,34712,24279,4021,13,561,303,34712,24279,4021,2649,411,12368,26,279,4732,24279,628,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,28109,5728,48233,411,12368,310,279,4732,24279,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[103,119],"option_spans":[[130,146],[152,168],[174,188]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.864868640899658,-5.007477283477783,-1.0895216464996338],"probs":[0.02198781631886959,0.01906548999249935,0.958946704864502],"argmax_index":2,"argmax_id":"supported","top2_margin":0.9369588885456324,"near_tie":false,"gold":"supported","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"supported","confidence":0.9589,"probabilities":{"contradicted":0.022,"insufficient":0.0191,"supported":0.9589}}},"usage":{"input_tokens":210,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_6d64a8459f64eac7f4cc","arm":"text","set":"fixture","source":"semif_authored144","tokens":207,"n_image_tokens":0,"state_tokens":40,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,28109,5728,48233,29201,310,279,303,34712,24279,948,424,628,2649,279,944,26,958,279,4732,24279,1132,948,279,303,34712,24279,4021,13,23472,24279,579,5548,310,2649,411,12368,369,16713,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,5439,425,279,3591,25,279,28109,5728,48233,411,12368,310,279,4732,24279,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5721,61416,279,13600,2129,1979,816,3147,7965,327,12404,8934,198,32929,220,17,25,220,4754,4532,3147,760,5721,1503,524,5517,2892,2129,1979,816,3147,1283,25831,8934,198,32929,220,18,25,220,4754,4532,3147,760,5721,61416,279,3591,2129,1979,816,3147,17675,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[100,116],"option_spans":[[127,143],[149,165],[171,185]],"option_ids":["contradicted","insufficient","supported"],"logits":[-4.738045692443848,-0.8032106161117554,-4.758543968200684],"probs":[0.018820548430085182,0.9627407789230347,0.018438685685396194],"argmax_index":1,"argmax_id":"insufficient","top2_margin":0.9439202304929495,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.9627,"probabilities":{"insufficient":0.9627,"contradicted":0.0188,"supported":0.0184}}},"usage":{"input_tokens":207,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_2708193212d8a4a523c7","arm":"text","set":"fixture","source":"semif_authored144","tokens":207,"n_image_tokens":0,"state_tokens":16,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,6092,13592,321,2812,2107,7440,781,513,2420,321,14037,364,279,6236,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,6911,279,2812,1132,430,264,31686,948,279,6092,13592,369,33829,466,6758,82529,26,5752,2236,279,6092,13592,13,3114,279,2812,381,31877,1172,411,4687,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[76,108],"option_spans":[[119,141],[147,163],[169,185]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.003655433654785,-2.321430206298828,-3.8108673095703125],"probs":[0.1317499727010727,0.7084866166114807,0.1597634106874466],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.5487232059240341,"near_tie":false,"gold":"prohibited","gold_correct":false}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.7085,"probabilities":{"prohibited":0.1598,"permitted":0.7085,"insufficient":0.1317}}},"usage":{"input_tokens":207,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_5a37d1dfcad20810a76a","arm":"text","set":"fixture","source":"semif_authored144","tokens":196,"n_image_tokens":0,"state_tokens":16,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,6092,13592,321,2812,2107,7440,781,513,2420,321,14037,364,279,6236,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,15771,279,10242,4687,11,2892,14037,2420,20777,1189,381,31877,1973,10304,13,3114,279,2812,381,31877,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[76,97],"option_spans":[[108,130],[136,152],[158,174]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.150921821594238,-1.3420687913894653,-4.823034763336182],"probs":[0.05524401366710663,0.9165467619895935,0.02820916660130024],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.8613027483224869,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.9165,"probabilities":{"prohibited":0.0282,"permitted":0.9165,"insufficient":0.0552}}},"usage":{"input_tokens":196,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_acacd8b46acd25cb954b","arm":"text","set":"fixture","source":"semif_authored144","tokens":206,"n_image_tokens":0,"state_tokens":15,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,6092,13592,369,10457,33829,11,1345,279,2812,369,2420,321,14037,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,6911,279,2812,1132,430,264,31686,948,279,6092,13592,369,33829,466,6758,82529,26,5752,2236,279,6092,13592,13,3114,279,2812,381,31877,1172,411,4687,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[75,107],"option_spans":[[118,140],[146,162],[168,184]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-4.519132137298584,-1.500509262084961,-4.5823750495910645],"probs":[0.04463925212621689,0.9134571552276611,0.041903555393218994],"argmax_index":1,"argmax_id":"permitted","top2_margin":0.8688179031014442,"near_tie":false,"gold":"permitted","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"permitted","confidence":0.9135,"probabilities":{"insufficient":0.0446,"permitted":0.9135,"prohibited":0.0419}}},"usage":{"input_tokens":206,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_4089c199b8b796ec3ed8","arm":"text","set":"fixture","source":"semif_authored144","tokens":205,"n_image_tokens":0,"state_tokens":14,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,760,1330,34982,513,6725,11,694,13468,17509,6130,77846,369,3766,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,6911,279,2812,1132,430,264,31686,948,279,6092,13592,369,33829,466,6758,82529,26,5752,2236,279,6092,13592,13,3114,279,2812,381,31877,1172,411,4687,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,16713,1928,1503,524,23950,3315,279,5728,27751,279,1852,2129,1979,816,3147,1283,25831,8934,198,32929,220,17,25,220,4754,4532,3147,760,10661,5728,27751,279,1852,2129,1979,816,3147,692,5310,8934,198,32929,220,18,25,220,4754,4532,3147,760,10661,5728,56467,279,1852,2129,1979,816,3147,751,89324,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[74,106],"option_spans":[[117,139],[145,161],[167,183]],"option_ids":["insufficient","permitted","prohibited"],"logits":[-1.7440648078918457,-3.8232436180114746,-3.979477882385254],"probs":[0.8117009401321411,0.10148927569389343,0.08680974692106247],"argmax_index":0,"argmax_id":"insufficient","top2_margin":0.7102116644382477,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.8117,"probabilities":{"insufficient":0.8117,"prohibited":0.0868,"permitted":0.1015}}},"usage":{"input_tokens":205,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_02f2b95a5c5a4777a45c","arm":"text","set":"fixture","source":"semif_authored144","tokens":191,"n_image_tokens":0,"state_tokens":29,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,369,279,14459,43971,321,369,10457,2858,310,3579,411,3992,13,47914,417,369,279,31686,43971,321,369,1048,2858,310,3579,424,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,357,14910,357,369,2858,26,4992,417,1132,948,357,369,11550,321,417,369,2858,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[89,108],"option_spans":[[119,130],[136,147],[153,169]],"option_ids":["A","B","insufficient"],"logits":[-0.9022272825241089,-5.06242036819458,-5.164846420288086],"probs":[0.97116619348526,0.015154607594013214,0.013679229654371738],"argmax_index":0,"argmax_id":"A","top2_margin":0.9560115858912468,"near_tie":false,"gold":"A","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"A","confidence":0.9712,"probabilities":{"B":0.0152,"A":0.9712,"insufficient":0.0137}}},"usage":{"input_tokens":191,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_ebea4681e3dcc7d92ca2","arm":"text","set":"fixture","source":"semif_authored144","tokens":196,"n_image_tokens":0,"state_tokens":29,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,369,279,14459,43971,321,369,10457,2858,310,3579,411,3992,13,47914,417,369,279,31686,43971,321,369,1048,2858,310,3579,424,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,15771,279,10242,10304,11,4992,417,14910,417,369,2858,26,4992,357,1132,948,417,369,11550,321,357,369,2858,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[89,113],"option_spans":[[124,135],[141,152],[158,174]],"option_ids":["A","B","insufficient"],"logits":[-3.446012020111084,-1.1591873168945312,-4.190361976623535],"probs":[0.08834957331418991,0.869680643081665,0.041969843208789825],"argmax_index":1,"argmax_id":"B","top2_margin":0.7813310697674751,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.8697,"probabilities":{"insufficient":0.042,"B":0.8697,"A":0.0883}}},"usage":{"input_tokens":196,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_b8c5b9b285cbdddc9f8d","arm":"text","set":"fixture","source":"semif_authored144","tokens":187,"n_image_tokens":0,"state_tokens":25,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,369,279,14459,43971,694,369,10457,11550,310,3579,411,3992,13,47914,417,369,279,31686,321,369,10457,2858,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,357,14910,357,369,2858,26,4992,417,1132,948,357,369,11550,321,417,369,2858,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[85,104],"option_spans":[[115,126],[132,143],[149,165]],"option_ids":["A","B","insufficient"],"logits":[-4.947561740875244,-0.6627171039581299,-4.88720703125],"probs":[0.013395226560533047,0.9723762273788452,0.014228580519557],"argmax_index":1,"argmax_id":"B","top2_margin":0.9581476468592882,"near_tie":false,"gold":"B","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"B","confidence":0.9724,"probabilities":{"A":0.0134,"insufficient":0.0142,"B":0.9724}}},"usage":{"input_tokens":187,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"semif_7ed75d854d99b3a81916","arm":"text","set":"fixture","source":"semif_authored144","tokens":185,"n_image_tokens":0,"state_tokens":23,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,61744,357,369,14459,321,47914,417,369,279,31686,11,694,13468,8871,579,5548,310,3579,279,3992,369,16713,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5307,198,9222,25,5576,198,669,55530,25,220,23298,357,14910,357,369,2858,26,4992,417,1132,948,357,369,11550,321,417,369,2858,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,61744,357,2129,1979,816,3147,32,8934,198,32929,220,17,25,220,4754,4532,3147,61744,417,2129,1979,816,3147,33,8934,198,32929,220,18,25,220,4754,4532,3147,60099,8871,16229,279,10897,5721,2129,1979,816,3147,1283,25831,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"decision","type":"choice","question_span":[83,102],"option_spans":[[113,124],[130,141],[147,163]],"option_ids":["A","B","insufficient"],"logits":[-3.238260269165039,-4.661498546600342,-1.1393043994903564],"probs":[0.1063990443944931,0.025634992867708206,0.8679659366607666],"argmax_index":2,"argmax_id":"insufficient","top2_margin":0.7615668922662735,"near_tie":false,"gold":"insufficient","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"decision":{"type":"choice","choice":"insufficient","confidence":0.868,"probabilities":{"insufficient":0.868,"B":0.0256,"A":0.1064}}},"usage":{"input_tokens":185,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_t01","arm":"text","set":"heldout","source":"heldout_text","tokens":708,"n_image_tokens":0,"state_tokens":139,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,12675,245402,11,353,1000,1040,310,2236,9483,5106,494,6796,220,24,6492,310,6394,220,16,18,6492,11,4097,3133,2756,11,310,9138,821,12553,579,12635,21444,13,353,1990,599,220,16,17,2756,314,5106,364,411,1007,11,748,220,22,1000,6922,25950,13,41201,65,432,16980,19883,682,7003,310,1831,888,279,16556,18254,6526,1345,353,1044,3023,11,321,353,668,1375,888,279,1724,7384,9861,310,1024,383,6394,220,21,6492,13,353,628,381,8379,539,4392,303,264,1865,12483,11,694,353,1000,4598,524,1716,6397,13,16018,488,1042,728,1366,539,279,809,314,1727,1936,11,748,421,353,628,2236,821,24118,30,8727,9061,11,345,3986,258,7379,263,74,1090,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5106,27430,198,9222,25,5576,198,669,55530,25,220,4199,1599,3133,2756,314,5106,513,10897,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,17,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,18,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,19,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,17,198,887,25,3311,10884,3600,198,9222,25,307,10644,198,669,55530,25,220,52681,682,7003,310,1353,1238,279,80701,579,20645,2261,279,5106,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,2756,9293,19239,198,9222,25,5576,198,669,55530,25,220,4199,1599,5106,2756,668,279,80701,599,2047,25950,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16,17,2756,2129,1979,816,3147,16,17,8934,198,32929,220,17,25,220,4754,4532,3147,16,22,2756,2129,1979,816,3147,16,22,8934,198,32929,220,18,25,220,4754,4532,3147,20,2756,2129,1979,816,3147,20,8934,198,32929,220,19,25,220,4754,4532,3147,22,2756,2129,1979,816,3147,22,8934,198,4537,38945,198,198,27806,220,19,198,887,25,2781,198,9222,25,5576,198,669,55530,25,220,9930,369,279,5106,10897,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1206,9138,264,2902,1484,2129,1979,816,3147,18411,6536,8934,198,32929,220,17,25,220,4754,4532,3147,2381,6255,6179,466,16621,2129,1979,816,3147,65817,8934,198,32929,220,18,25,220,4754,4532,3147,1206,3166,3626,2129,1979,816,3147,62916,8934,198,32929,220,19,25,220,4754,4532,3147,1206,2364,449,6788,466,9138,264,3199,2129,1979,816,3147,53116,8934,198,4537,38945,198,198,27806,220,20,198,887,25,13813,3456,92308,198,9222,25,5284,198,669,55530,25,220,4199,3999,1000,424,381,364,264,6436,310,27237,411,1622,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,18140,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,53390,264,10125,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,85154,13048,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"leave_days","type":"choice","question_span":[200,209],"option_spans":[[220,230],[236,246],[252,262],[268,278]],"option_ids":["3","4","5","6"],"logits":[-4.94727897644043,-4.76135778427124,-0.5646947622299194,-4.6150665283203125],"probs":[0.011955575086176395,0.014398426748812199,0.9569793343544006,0.016666658222675323],"argmax_index":2,"argmax_id":"5","top2_margin":0.9403126761317253,"near_tie":false,"gold":"5","gold_correct":true},{"question_id":"cover_arranged","type":"noul","question_span":[302,316],"option_spans":[[327,345],[351,369]],"option_ids":["true","false"],"logits":[-1.7976644039154053,-4.453482151031494],"probs":[0.9343686699867249,0.06563134491443634],"argmax_index":0,"argmax_id":"true","top2_margin":0.8687373250722885,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"days_left_after","type":"choice","question_span":[392,403],"option_spans":[[414,427],[433,446],[452,463],[469,480]],"option_ids":["12","17","5","7"],"logits":[-4.064053058624268,-3.944965362548828,-3.5786592960357666,-1.424957275390625],"probs":[0.056332241743803024,0.06345650553703308,0.09152945131063461,0.78868168592453],"argmax_index":3,"argmax_id":"7","top2_margin":0.6971522346138954,"near_tie":false,"gold":"7","gold_correct":true},{"question_id":"reason","type":"choice","question_span":[501,507],"option_spans":[[518,533],[539,553],[559,571],[577,594]],"option_ids":["family_event","medical","moving","study"],"logits":[0.44367626309394836,-4.931265830993652,-5.073748588562012,-4.93848991394043],"probs":[0.9869279265403748,0.0045706466771662235,0.003963676281273365,0.0045377472415566444],"argmax_index":0,"argmax_id":"family_event","top2_margin":0.9823572798632085,"near_tie":false,"gold":"family_event","gold_correct":true},{"question_id":"ease_of_approval","type":"score","question_span":[617,630],"option_spans":[[641,651],[657,669],[675,686]],"option_ids":["0","1","2"],"logits":[-3.9428749084472656,-3.407135248184204,-1.5355912446975708],"probs":[0.07239833474159241,0.12370789051055908,0.8038937449455261],"argmax_index":2,"argmax_id":"2","top2_margin":0.680185854434967,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"leave_days":{"type":"choice","choice":"5","confidence":0.957,"probabilities":{"3":0.012,"4":0.0144,"5":0.957,"6":0.0167}},"cover_arranged":{"type":"noul","noul":0.9344},"days_left_after":{"type":"choice","choice":"7","confidence":0.7887,"probabilities":{"5":0.0915,"7":0.7887,"12":0.0563,"17":0.0635}},"reason":{"type":"choice","choice":"family_event","confidence":0.9869,"probabilities":{"family_event":0.9869,"medical":0.0046,"study":0.0045,"moving":0.004}},"ease_of_approval":{"type":"score","score":1.7315,"confidence":0.8039,"legend":{"0":"hard","1":"needs a conversation","2":"straightforward"},"probabilities":{"0":0.0724,"1":0.1237,"2":0.8039}}},"usage":{"input_tokens":708,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_t02","arm":"text","set":"heldout","source":"heldout_text","tokens":746,"n_image_tokens":0,"state_tokens":179,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,54,41766,3591,457,12,20,23,17,16,18,11,3816,220,17,23,5930,220,17,15,17,21,13,11926,25,7305,323,4564,633,549,1336,13,5466,25,345,7671,463,5701,12,17,16,15,21630,1657,27286,30835,11,10471,491,383,220,17,7165,220,17,15,17,20,440,264,1330,4514,13562,579,8215,13,75921,14109,25,279,11269,874,4860,9687,1141,6545,26,1238,264,4220,3603,383,279,26655,279,30835,8213,364,2599,1056,2943,4276,11,2272,279,34227,4276,10661,303,279,11048,13,2838,682,8341,279,3889,21867,321,16687,5953,1324,5701,17,16,15,12,22,22,42,12,15,19,16,24,13,2838,32726,421,279,30835,10729,1441,264,10650,314,30129,1483,7094,11,694,2640,424,8341,3133,13870,364,3818,1238,421,13,2838,16561,364,264,13329,11269,310,381,8194,310,1024,2046,2534,11,1521,1292,874,4860,682,264,3618,310,3502,279,4220,30835,1142,303,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,303,1611,41766,198,9222,25,307,10644,198,669,55530,25,220,1860,279,2321,279,3591,557,3816,11,279,1918,369,1990,4613,1141,8215,4031,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,55639,10182,198,9222,25,5576,198,669,55530,25,220,22365,919,1503,279,5813,1910,369,55639,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,11269,2129,1979,816,3147,71831,8934,198,32929,220,17,25,220,4754,4532,3147,760,15166,6299,2129,1979,816,3147,34867,8934,198,32929,220,18,25,220,4754,4532,3147,760,21938,26655,2129,1979,816,3147,74419,8934,198,32929,220,19,25,220,4754,4532,3147,760,15333,3915,2129,1979,816,3147,5147,8934,198,32929,220,20,25,220,4754,4532,3147,760,8604,2129,1979,816,3147,56042,8934,198,4537,38945,198,198,27806,220,18,198,887,25,3102,39546,198,9222,25,307,10644,198,669,55530,25,220,12525,279,1918,599,978,19510,539,279,5813,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,5813,32726,449,11095,1680,430,264,4255,11,264,37514,466,264,13722,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,2665,11095,466,10841,11256,369,9444,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,37843,198,9222,25,5576,198,669,55530,25,220,3710,1503,279,5813,2466,364,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,32,4434,491,1918,2129,1979,816,3147,902,14402,8934,198,32929,220,17,25,220,4754,4532,3147,32,13329,919,8194,310,1024,2129,1979,816,3147,4332,3584,6133,8934,198,32929,220,18,25,220,4754,4532,3147,20079,3117,1142,2129,1979,816,3147,66165,8934,198,32929,220,19,25,220,4754,4532,3147,32,12368,1238,11293,279,1918,303,2129,1979,816,3147,73825,8934,198,4537,38945,198,198,27806,220,20,198,887,25,3591,69451,198,9222,25,5284,198,669,55530,25,220,4199,3619,369,279,3591,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,12768,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,1299,7373,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,4371,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"in_warranty","type":"noul","question_span":[242,259],"option_spans":[[270,288],[294,312]],"option_ids":["true","false"],"logits":[-2.6216204166412354,-2.2450199127197266],"probs":[0.4069470763206482,0.5930529236793518],"argmax_index":1,"argmax_id":"false","top2_margin":0.1861058473587036,"near_tie":false,"gold":"true","gold_correct":false},{"question_id":"faulty_part","type":"choice","question_span":[334,343],"option_spans":[[354,365],[371,383],[389,401],[407,419],[425,436]],"option_ids":["battery","brush","dock","filter","motor"],"logits":[-0.24475616216659546,-4.8423051834106445,-4.540416240692139,-4.891828536987305,-4.601766109466553],"probs":[0.9559220671653748,0.00963235180824995,0.013026898726820946,0.009166945703327656,0.012251721695065498],"argmax_index":0,"argmax_id":"battery","top2_margin":0.9428951684385538,"near_tie":false,"gold":"battery","gold_correct":true},{"question_id":"possible_damage","type":"noul","question_span":[459,469],"option_spans":[[480,504],[510,526]],"option_ids":["true","false"],"logits":[-0.7511169910430908,-3.8359639644622803],"probs":[0.9562633633613586,0.04373664781451225],"argmax_index":0,"argmax_id":"true","top2_margin":0.9125267155468464,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"remedy","type":"choice","question_span":[547,554],"option_spans":[[565,579],[585,602],[608,620],[626,642]],"option_ids":["new_unit","part_by_post","refund","repair"],"logits":[-4.495284557342529,0.4948127865791321,-4.764180660247803,-4.154328346252441],"probs":[0.006661281455308199,0.9788803458213806,0.00509070185944438,0.009367705322802067],"argmax_index":1,"argmax_id":"part_by_post","top2_margin":0.9695126404985785,"near_tie":false,"gold":"part_by_post","gold_correct":true},{"question_id":"claim_strength","type":"score","question_span":[664,670],"option_spans":[[681,691],[697,708],[714,724]],"option_ids":["0","1","2"],"logits":[-1.7716851234436035,-0.6251367330551147,-2.044718027114868],"probs":[0.20373326539993286,0.6412119269371033,0.15505482256412506],"argmax_index":1,"argmax_id":"1","top2_margin":0.4374786615371704,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"in_warranty":{"type":"noul","noul":0.4069},"faulty_part":{"type":"choice","choice":"battery","confidence":0.9559,"probabilities":{"battery":0.9559,"motor":0.0123,"brush":0.0096,"dock":0.013,"filter":0.0092}},"possible_damage":{"type":"noul","noul":0.9563},"remedy":{"type":"choice","choice":"part_by_post","confidence":0.9789,"probabilities":{"refund":0.0051,"part_by_post":0.9789,"repair":0.0094,"new_unit":0.0067}},"claim_strength":{"type":"score","score":0.9513,"confidence":0.6412,"legend":{"0":"weak","1":"uncertain","2":"strong"},"probabilities":{"0":0.2037,"1":0.6412,"2":0.1551}}},"usage":{"input_tokens":746,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_t03","arm":"text","set":"heldout","source":"heldout_text","tokens":842,"n_image_tokens":0,"state_tokens":164,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,29664,4315,85,26395,6558,1997,11,353,5312,678,7447,364,279,1330,64609,9766,383,279,4688,6220,314,231830,21845,432,4526,318,15812,6759,12,17,17,15,22,8,321,1000,1040,310,29880,264,19489,13,353,944,3603,27700,430,264,27187,11,748,45022,55287,1238,220,16,15,25,15,15,7535,728,1786,13,353,1000,381,6992,303,440,821,8005,321,1004,27878,19177,72539,13,561,7447,2640,2526,24307,1132,11,748,4338,3184,728,3315,264,18717,220,18,15,20332,5388,1000,381,11330,13,1165,635,524,1765,264,1740,11,748,279,6639,314,264,12551,3433,369,874,3377,364,586,13,16018,488,1048,7440,3315,279,7765,314,220,16,11,19,20,15,264,2182,5469,22842,11,321,1204,3349,279,16051,369,30,1165,3769,310,3166,303,383,220,16,6443,13,16348,23187,11,604,41242,92463,73187,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,16561,56029,73363,198,9222,25,307,10644,198,669,55530,25,220,760,6700,16561,3315,449,9572,1000,381,5252,303,279,9766,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,26694,198,9222,25,5576,198,669,55530,25,220,2634,9459,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,588,13433,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,19186,26694,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,26952,26694,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,64,13626,2129,1979,816,3147,57024,8934,198,4537,38945,198,198,27806,220,18,198,887,25,3749,605,32342,198,9222,25,307,10644,198,669,55530,25,220,760,6700,3749,264,12551,3433,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,19489,2914,198,9222,25,5576,198,669,55530,25,220,4350,1000,279,6700,1040,310,1593,279,9766,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,2665,21257,369,2574,2129,1979,816,3147,3639,8934,198,32929,220,17,25,220,4754,4532,3147,1860,264,45022,11129,2129,1979,816,3147,75317,66034,286,8934,198,32929,220,18,25,220,4754,4532,3147,1860,264,45022,6353,2129,1979,816,3147,75317,697,76833,8934,198,32929,220,19,25,220,4754,4532,3147,1597,279,8729,2129,1979,816,3147,9958,400,8934,198,4537,38945,198,198,27806,220,20,198,887,25,975,27973,198,9222,25,5576,198,669,55530,25,220,67150,494,24307,11,1092,745,1503,279,6700,2466,883,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,9947,314,264,13159,2129,1979,816,3147,70,8082,8934,198,32929,220,17,25,220,4754,4532,3147,760,7361,3511,2129,1979,816,3147,82783,8934,198,32929,220,18,25,220,4754,4532,3147,4199,1248,279,5600,6364,38346,2129,1979,816,3147,1567,4956,8934,198,32929,220,19,25,220,4754,4532,3147,3710,279,7765,14103,321,279,1331,314,279,16051,2129,1979,816,3147,7725,82834,8934,198,4537,38945,198,198,27806,220,21,198,887,25,2353,198,9222,25,5284,198,669,55530,25,220,4199,1575,1503,279,9766,2714,310,7535,279,6700,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,5200,269,391,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,4332,391,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,8884,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,1164,1575,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"asks_about_pet","type":"noul","question_span":[227,240],"option_spans":[[251,269],[275,293]],"option_ids":["true","false"],"logits":[-0.5420043468475342,-3.6343932151794434],"probs":[0.956577718257904,0.04342230036854744],"argmax_index":0,"argmax_id":"true","top2_margin":0.9131554178893566,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"bedrooms","type":"choice","question_span":[314,316],"option_spans":[[327,338],[344,355],[361,372],[378,389]],"option_ids":["1","2","3","studio"],"logits":[-3.7620577812194824,0.9751730561256409,-4.064438819885254,-4.117854595184326],"probs":[0.008579462766647339,0.97906893491745,0.006340706720948219,0.006010899785906076],"argmax_index":1,"argmax_id":"2","top2_margin":0.9704894721508026,"near_tie":false,"gold":"2","gold_correct":true},{"question_id":"needs_parking","type":"noul","question_span":[413,420],"option_spans":[[431,449],[455,473]],"option_ids":["true","false"],"logits":[-2.529954195022583,1.923886775970459],"probs":[0.011500004678964615,0.9885000586509705],"argmax_index":1,"argmax_id":"false","top2_margin":0.9770000539720058,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"viewing_time","type":"choice","question_span":[495,505],"option_spans":[[516,529],[535,550],[556,571],[577,590]],"option_ids":["any","weekday_evening","weekday_morning","weekend"],"logits":[-3.4680445194244385,-3.6905529499053955,-0.016958922147750854,-3.656270980834961],"probs":[0.029270969331264496,0.023431649431586266,0.9230484366416931,0.024248862639069557],"argmax_index":2,"argmax_id":"weekday_morning","top2_margin":0.8937774673104286,"near_tie":false,"gold":"weekday_morning","gold_correct":true},{"question_id":"other_question","type":"choice","question_span":[612,624],"option_spans":[[635,649],[655,667],[673,689],[695,715]],"option_ids":["garden","internet","lease_length","rent_deposit"],"logits":[-4.645993709564209,-5.056751728057861,-4.621515274047852,1.678515911102295],"probs":[0.0017832558369264007,0.0011825613910332322,0.0018274456961080432,0.9952066540718079],"argmax_index":3,"argmax_id":"rent_deposit","top2_margin":0.9933792083756998,"near_tie":false,"gold":"rent_deposit","gold_correct":true},{"question_id":"match","type":"score","question_span":[736,747],"option_spans":[[758,770],[776,787],[793,803],[809,820]],"option_ids":["0","1","2","3"],"logits":[-1.9131457805633545,-0.6368253231048584,-0.13967476785182953,-2.103362560272217],"probs":[0.0884840190410614,0.31707626581192017,0.5212829113006592,0.07315680384635925],"argmax_index":2,"argmax_id":"2","top2_margin":0.204206645488739,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"asks_about_pet":{"type":"noul","noul":0.9566},"bedrooms":{"type":"choice","choice":"2","confidence":0.9791,"probabilities":{"1":0.0086,"2":0.9791,"3":0.0063,"studio":0.006}},"needs_parking":{"type":"noul","noul":0.0115},"viewing_time":{"type":"choice","choice":"weekday_morning","confidence":0.923,"probabilities":{"weekday_morning":0.923,"weekday_evening":0.0234,"weekend":0.0242,"any":0.0293}},"other_question":{"type":"choice","choice":"rent_deposit","confidence":0.9952,"probabilities":{"rent_deposit":0.9952,"garden":0.0018,"lease_length":0.0018,"internet":0.0012}},"match":{"type":"score","score":1.5791,"confidence":0.5213,"legend":{"0":"poorly","1":"partly","2":"well","3":"very well"},"probabilities":{"0":0.0885,"1":0.3171,"2":0.5213,"3":0.0732}}},"usage":{"input_tokens":842,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_t04","arm":"text","set":"heldout","source":"heldout_text","tokens":658,"n_image_tokens":0,"state_tokens":149,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,34177,383,279,2734,6957,57908,11754,14168,11,7483,13,561,44854,61230,998,279,1786,919,314,279,1834,25,279,16657,41278,321,279,45952,7397,41278,2107,5898,680,539,3988,12967,13,38850,557,264,9156,13,561,16000,2002,8660,449,6258,3200,321,279,6951,314,9054,38965,1142,310,279,1822,5459,11,748,567,7157,34227,4276,1066,3583,303,13,15324,557,220,23,791,6474,11,864,6273,6415,11,694,21938,2748,888,4097,279,2400,6474,3240,1467,524,13,561,4155,702,42345,3727,383,854,11,7643,279,20670,303,279,16171,15173,8341,13964,680,321,567,13235,4139,314,279,2963,383,65354,286,23133,13,561,60092,19013,4101,660,1834,13,1165,1000,2445,1142,1727,1007,413,279,12551,369,10300,680,13,15982,278,474,44048,455,7480,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,8276,58111,198,9222,25,5576,198,669,55530,25,220,3710,557,279,8276,3377,364,411,19568,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,4131,314,279,3487,2129,1979,816,3147,13515,8934,198,32929,220,17,25,220,4754,4532,3147,91061,321,3583,303,2129,1979,816,3147,27795,286,8934,198,32929,220,18,25,220,4754,4532,3147,760,4202,3240,2129,1979,816,3147,6352,8934,198,32929,220,19,25,220,4754,4532,3147,760,60092,2129,1979,816,3147,239350,2491,8934,198,32929,220,20,25,220,4754,4532,3147,760,8831,2129,1979,816,3147,14764,8934,198,4537,38945,198,198,27806,220,17,198,887,25,1000,12153,198,9222,25,307,10644,198,669,55530,25,220,27083,279,19568,2445,1495,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,19568,1000,9138,1495,11,10451,383,264,2876,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,19568,1000,524,9138,1495,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,3487,9293,198,9222,25,307,10644,198,669,55530,25,220,11179,41278,1990,995,3487,506,279,809,314,279,1834,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,15754,198,9222,25,5284,198,669,55530,25,220,4199,78386,369,279,10732,7830,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,3662,448,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,1164,8751,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,40167,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,54789,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,69,25167,470,2129,1979,816,3147,19,8934,198,32929,220,21,25,220,4754,4532,3147,1164,78386,2129,1979,816,3147,20,8934,198,32929,220,22,25,220,4754,4532,3147,5882,22460,2129,1979,816,3147,21,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"biggest_problem","type":"choice","question_span":[210,219],"option_spans":[[230,244],[250,264],[270,282],[288,300],[306,317]],"option_ids":["food","parking","price","toilets","weather"],"logits":[-4.388947010040283,-0.04731941223144531,-4.1138482093811035,-5.024718761444092,-4.971222877502441],"probs":[0.012463031336665154,0.9575657248497009,0.016409585252404213,0.006599507760256529,0.006962168030440807],"argmax_index":1,"argmax_id":"parking","top2_margin":0.9411561395972967,"near_tie":false,"gold":"parking","gold_correct":true},{"question_id":"would_return","type":"noul","question_span":[340,346],"option_spans":[[357,376],[382,397]],"option_ids":["true","false"],"logits":[-0.58756422996521,-4.131371021270752],"probs":[0.9719088077545166,0.02809116803109646],"argmax_index":0,"argmax_id":"true","top2_margin":0.9438176397234201,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"food_left","type":"noul","question_span":[420,432],"option_spans":[[443,461],[467,485]],"option_ids":["true","false"],"logits":[-3.7742786407470703,0.7698715925216675],"probs":[0.010517409071326256,0.9894826412200928],"argmax_index":1,"argmax_id":"false","top2_margin":0.9789652321487665,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"tone","type":"score","question_span":[506,513],"option_spans":[[524,535],[541,552],[558,568],[574,584],[590,602],[608,619],[625,636]],"option_ids":["0","1","2","3","4","5","6"],"logits":[-4.502910614013672,-4.717281818389893,-4.61263370513916,-0.9044182300567627,-2.9777884483337402,-4.569973468780518,-4.624217987060547],"probs":[0.021899646148085594,0.01767408289015293,0.019623884931206703,0.8002809882164001,0.1006443202495575,0.02047915756702423,0.01939786784350872],"argmax_index":3,"argmax_id":"3","top2_margin":0.6996366679668427,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"biggest_problem":{"type":"choice","choice":"parking","confidence":0.9576,"probabilities":{"parking":0.9576,"food":0.0125,"price":0.0164,"toilets":0.0066,"weather":0.007}},"would_return":{"type":"noul","noul":0.9719},"food_left":{"type":"noul","noul":0.0105},"tone":{"type":"score","score":3.0791,"confidence":0.8003,"legend":{"0":"hostile","1":"very critical","2":"critical","3":"mixed","4":"favourable","5":"very favourable","6":"glowing"},"probabilities":{"0":0.0219,"1":0.0177,"2":0.0196,"3":0.8003,"4":0.1006,"5":0.0205,"6":0.0194}}},"usage":{"input_tokens":658,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_t05","arm":"text","set":"heldout","source":"heldout_text","tokens":727,"n_image_tokens":0,"state_tokens":131,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,45209,1830,11,72734,86,12713,17529,220,18,13,23,13,17,383,264,20071,440,279,5362,1785,2560,13,38540,25,1724,264,35935,421,682,777,1056,220,17,15,15,6602,11,3289,310,279,20454,1593,11,1179,2404,279,20071,310,17865,13,30003,25,279,20454,25710,1724,383,279,1788,2081,13,32254,25,279,878,90162,364,883,5600,6283,321,1179,32147,11,321,279,1483,1330,4276,314,18904,513,7795,13,1049,8311,1396,854,440,3349,75842,321,2496,440,2526,5981,13,5889,220,18,13,22,13,24,1467,524,34512,1040,411,13,5374,18864,25,5128,279,4034,12331,1518,8306,279,20454,13,75921,539,37344,90020,192297,5229,10951,5988,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,21531,32061,198,9222,25,5576,198,669,55530,25,220,3710,2993,314,21531,369,411,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,21624,23057,76952,3147,1719,878,17397,28378,466,32147,2129,77459,648,3344,3147,276,632,37537,1412,56658,1979,816,3147,4920,956,8934,198,32929,220,17,25,220,4754,4532,21624,23057,76952,3147,32228,369,14336,303,279,4808,1925,2129,77459,648,3344,3147,64,6576,16036,56658,1979,816,3147,5320,8934,198,32929,220,18,25,220,4754,4532,21624,23057,76952,3147,1719,878,369,6103,694,13205,3133,2129,77459,648,3344,3147,472,529,5217,8427,56658,1979,816,3147,3122,754,425,8934,198,32929,220,19,25,220,4754,4532,21624,23057,76952,3147,18778,635,524,5372,975,7370,2129,77459,648,3344,3147,64,3421,466,3790,14109,56658,1979,816,3147,12624,8934,198,4537,38945,198,198,27806,220,17,198,887,25,29551,198,9222,25,307,10644,198,669,55530,25,220,760,3377,1467,524,2905,303,449,6575,2243,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,31130,11167,198,9222,25,307,10644,198,669,55530,25,220,8114,314,279,1156,579,944,369,5384,948,279,3377,8311,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,2526,26331,11766,198,9222,25,307,10644,198,669,55530,25,220,760,3377,1048,13273,440,2526,75842,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,20,198,887,25,10304,198,9222,25,5284,198,669,55530,25,220,3710,10304,1220,279,4884,615,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,47,19,68328,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,47,18,4472,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,47,17,1496,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,47,16,32556,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"defect_kind","type":"choice","question_span":[192,199],"option_spans":[[210,238],[244,271],[277,307],[313,341]],"option_ids":["crash","display","slowness","sync"],"logits":[-0.5201346278190613,-3.9802451133728027,-3.6339614391326904,-4.800596714019775],"probs":[0.9176895022392273,0.028839578852057457,0.04077349975705147,0.01269739493727684],"argmax_index":0,"argmax_id":"crash","top2_margin":0.8769160024821758,"near_tie":false,"gold":"crash","gold_correct":true},{"question_id":"regression","type":"noul","question_span":[363,373],"option_spans":[[384,402],[408,426]],"option_ids":["true","false"],"logits":[-0.5481787919998169,-3.3880844116210938],"probs":[0.9447945952415466,0.05520545691251755],"argmax_index":0,"argmax_id":"true","top2_margin":0.8895891383290291,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"loses_work","type":"noul","question_span":[449,462],"option_spans":[[473,491],[497,515]],"option_ids":["true","false"],"logits":[-0.5018942952156067,-3.587813138961792],"probs":[0.9563081860542297,0.04369184002280235],"argmax_index":0,"argmax_id":"true","top2_margin":0.9126163460314274,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"small_notebooks","type":"noul","question_span":[539,547],"option_spans":[[558,576],[582,600]],"option_ids":["true","false"],"logits":[-3.1231071949005127,1.680635690689087],"probs":[0.008132325485348701,0.99186772108078],"argmax_index":1,"argmax_id":"false","top2_margin":0.9837353955954313,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"priority","type":"score","question_span":[621,628],"option_spans":[[639,651],[657,669],[675,687],[693,705]],"option_ids":["0","1","2","3"],"logits":[-2.057436943054199,-1.2389657497406006,0.2243494987487793,-1.1540836095809937],"probs":[0.06439540535211563,0.145986407995224,0.6306989789009094,0.15891915559768677],"argmax_index":2,"argmax_id":"2","top2_margin":0.47177982330322266,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"defect_kind":{"type":"choice","choice":"crash","confidence":0.9177,"probabilities":{"crash":0.9177,"display":0.0288,"sync":0.0127,"slowness":0.0408}},"regression":{"type":"noul","noul":0.9448},"loses_work":{"type":"noul","noul":0.9563},"small_notebooks":{"type":"noul","noul":0.0081},"priority":{"type":"score","score":1.8841,"confidence":0.6307,"legend":{"0":"P4 backlog","1":"P3 normal","2":"P2 high","3":"P1 urgent"},"probabilities":{"0":0.0644,"1":0.146,"2":0.6307,"3":0.1589}}},"usage":{"input_tokens":727,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_t06","arm":"text","set":"heldout","source":"heldout_text","tokens":721,"n_image_tokens":0,"state_tokens":146,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4286,220,21,21,12,16,24,15,19,11,3106,440,60298,81,402,88,181973,1943,364,1727,10902,9403,13,561,14419,2081,2640,359,9302,41636,11,2047,303,264,5902,1925,6,506,220,16,19,25,20,17,13278,11,694,1017,557,4161,506,821,5817,11,303,279,43167,466,440,279,34667,383,2892,3008,13,353,557,3133,494,2046,660,12967,321,18022,40707,279,27491,13,561,28350,9687,264,13329,4392,4034,5621,220,23,24,11,321,353,1144,424,364,264,12368,353,599,31877,383,7548,13,561,50545,4263,314,1004,4599,682,524,3816,3977,2892,13,5044,18730,11,321,2892,1423,279,28350,466,3502,264,491,799,1518,7548,13,353,635,524,1318,821,3117,1142,13,5963,258,436,8903,540,15377,220457,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,3377,198,9222,25,5576,198,669,55530,25,220,3710,3804,4808,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,28350,11136,19510,2129,1979,816,3147,14959,3173,8934,198,32929,220,17,25,220,4754,4532,3147,760,28350,369,3200,2129,1979,816,3147,4913,8934,198,32929,220,18,25,220,4754,4532,3147,35889,2640,12238,694,4161,11136,2129,1979,816,3147,29617,19239,27915,8934,198,32929,220,19,25,220,4754,4532,3147,760,28350,557,3106,310,2361,2534,383,1622,2129,1979,816,3147,7861,290,8934,198,32929,220,20,25,220,4754,4532,3147,760,4808,1455,11136,2129,1979,816,3147,33713,5457,8934,198,4537,38945,198,198,27806,220,17,198,887,25,557,20990,198,9222,25,307,10644,198,669,55530,25,220,760,5813,557,506,2046,948,279,9403,557,12076,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,14830,1611,7327,198,9222,25,5576,198,669,55530,25,220,3710,14830,1503,279,5813,1318,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,1906,24863,2129,1979,816,3147,17961,8934,198,32929,220,17,25,220,4754,4532,3147,11279,19252,2129,1979,816,3147,5511,40249,8934,198,32929,220,18,25,220,4754,4532,3147,32,20325,2129,1979,816,3147,66165,8934,198,32929,220,19,25,220,4754,4532,3147,760,28350,1669,466,3106,1495,2129,1979,816,3147,408,400,8934,198,4537,38945,198,198,27806,220,19,198,887,25,854,63118,198,9222,25,307,10644,198,669,55530,25,220,1619,63118,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,20,198,887,25,19070,198,9222,25,5284,198,669,55530,25,220,4199,21760,1503,279,5813,4950,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,5248,76,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,435,27612,290,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,446,723,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,69,26634,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"problem","type":"choice","question_span":[206,210],"option_spans":[[221,235],[241,254],[260,277],[283,302],[308,322]],"option_ids":["damaged","late","missing_after_scan","redirected","wrong_item"],"logits":[-4.725399971008301,-4.296517848968506,-0.3177794814109802,-4.473583221435547,-4.559125900268555],"probs":[0.011484123766422272,0.017634332180023193,0.9425472021102905,0.014772721566259861,0.01356156449764967],"argmax_index":2,"argmax_id":"missing_after_scan","top2_margin":0.9249128699302673,"near_tie":false,"gold":"missing_after_scan","gold_correct":true},{"question_id":"was_home","type":"noul","question_span":[345,356],"option_spans":[[367,385],[391,409]],"option_ids":["true","false"],"logits":[-2.45329213142395,-4.433873176574707],"probs":[0.878743052482605,0.12125690281391144],"argmax_index":0,"argmax_id":"true","top2_margin":0.7574861496686935,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"outcome_wanted","type":"choice","question_span":[432,439],"option_spans":[[450,462],[468,480],[486,497],[503,519]],"option_ids":["cancel","compensation","refund","resend"],"logits":[-4.909331321716309,-4.869506359100342,-4.735398769378662,-0.9086815714836121],"probs":[0.017281828448176384,0.017983967438340187,0.020564954727888107,0.944169282913208],"argmax_index":3,"argmax_id":"resend","top2_margin":0.9236043281853199,"near_tie":false,"gold":"resend","gold_correct":true},{"question_id":"time_sensitive","type":"noul","question_span":[542,544],"option_spans":[[555,573],[579,597]],"option_ids":["true","false"],"logits":[-2.1030523777008057,-4.511847496032715],"probs":[0.9174954891204834,0.08250447362661362],"argmax_index":0,"argmax_id":"true","top2_margin":0.8349910154938698,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"mood","type":"score","question_span":[618,625],"option_spans":[[636,647],[653,665],[671,682],[688,699]],"option_ids":["0","1","2","3"],"logits":[-3.1446821689605713,-2.0378990173339844,-1.00859797000885,-2.112217426300049],"probs":[0.06536456197500229,0.19770249724388123,0.5533907413482666,0.18354223668575287],"argmax_index":2,"argmax_id":"2","top2_margin":0.3556882441043854,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"problem":{"type":"choice","choice":"missing_after_scan","confidence":0.9425,"probabilities":{"late":0.0176,"damaged":0.0115,"missing_after_scan":0.9425,"wrong_item":0.0136,"redirected":0.0148}},"was_home":{"type":"noul","noul":0.8787},"outcome_wanted":{"type":"choice","choice":"resend","confidence":0.9442,"probabilities":{"refund":0.0206,"resend":0.9442,"compensation":0.018,"cancel":0.0173}},"time_sensitive":{"type":"noul","noul":0.9175},"mood":{"type":"score","score":1.8551,"confidence":0.5534,"legend":{"0":"calm","1":"concerned","2":"upset","3":"furious"},"probabilities":{"0":0.0654,"1":0.1977,"2":0.5534,"3":0.1835}}},"usage":{"input_tokens":721,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_t07","arm":"text","set":"heldout","source":"heldout_text","tokens":1086,"n_image_tokens":0,"state_tokens":159,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,40,1044,26079,685,821,36919,579,64191,26261,10793,364,264,2812,21844,6080,13,561,3889,3520,799,33540,314,220,16,17,30308,25,220,17,20,15,338,81267,11,220,16,17,20,338,13696,11,220,16,15,15,338,13477,12781,11,220,19,53626,314,20117,42752,321,264,47589,314,11679,11,39334,506,220,16,23,15,35087,34,364,220,17,20,4276,13,353,1144,220,18,21,30308,11,864,3254,2250,88562,13,2921,9488,27477,1132,708,685,310,220,20,15,15,338,321,353,599,5384,821,28043,18861,2305,11,694,353,1366,421,799,59437,314,42752,47778,883,220,17,15,338,13,11910,279,26912,854,2222,413,353,21844,279,88562,799,1238,2361,303,279,1788,22660,30,1541,628,353,8032,4139,314,279,13696,440,6797,37471,5415,1973,279,15436,15218,10431,30,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,7911,198,9222,25,5576,198,669,55530,25,220,1311,1092,7911,1902,279,31591,381,53070,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16,13,20,2129,1979,816,3147,16,13,20,8934,198,32929,220,17,25,220,4754,4532,3147,17,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,18,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,19,2129,1979,816,3147,19,8934,198,32929,220,20,25,220,4754,4532,3147,21,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,17,198,887,25,81267,10467,198,9222,25,5576,198,669,55530,25,220,4199,1691,81267,369,4221,303,2702,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16,11,15,15,15,338,2129,1979,816,3147,16,15,15,15,8934,198,32929,220,17,25,220,4754,4532,3147,16,11,17,20,15,338,2129,1979,816,3147,16,17,20,15,8934,198,32929,220,18,25,220,4754,4532,3147,20,15,15,338,2129,1979,816,3147,20,15,15,8934,198,32929,220,19,25,220,4754,4532,3147,22,20,15,338,2129,1979,816,3147,22,20,15,8934,198,4537,38945,198,198,27806,220,18,198,887,25,42752,10467,198,9222,25,5576,198,669,55530,25,220,4199,1599,32216,314,20117,42752,513,4221,303,2702,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,17,19,15,338,2129,1979,816,3147,17,19,15,8934,198,32929,220,17,25,220,4754,4532,3147,18,17,15,338,2129,1979,816,3147,18,17,15,8934,198,32929,220,18,25,220,4754,4532,3147,21,15,338,2129,1979,816,3147,21,15,8934,198,32929,220,19,25,220,4754,4532,3147,23,15,338,2129,1979,816,3147,23,15,8934,198,4537,38945,198,198,27806,220,19,198,887,25,81267,11331,44390,1055,286,198,9222,25,307,10644,198,669,55530,25,220,2324,314,279,81267,628,381,45054,303,799,708,383,279,4155,579,27477,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,20,198,887,25,16561,39637,198,9222,25,307,10644,198,669,55530,25,220,760,4155,16561,883,24101,799,23980,440,2361,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,21,198,887,25,22660,51729,198,9222,25,5576,198,669,55530,25,220,1597,1092,9039,513,279,88562,39334,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16,21,15,35087,34,2129,1979,816,3147,16,21,15,8934,198,32929,220,17,25,220,4754,4532,3147,16,22,15,35087,34,2129,1979,816,3147,16,22,15,8934,198,32929,220,18,25,220,4754,4532,3147,16,23,15,35087,34,2129,1979,816,3147,16,23,15,8934,198,32929,220,19,25,220,4754,4532,3147,17,15,15,35087,34,2129,1979,816,3147,17,15,15,8934,198,4537,38945,198,198,27806,220,22,198,887,25,4602,89084,290,198,9222,25,5576,198,669,55530,25,220,4199,1599,4602,1503,279,1876,2466,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,4537,38945,198,198,27806,220,23,198,887,25,16337,198,9222,25,5284,198,669,55530,25,220,4199,32343,369,411,26079,2531,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,371,25806,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,43518,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,371,18391,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"factor","type":"choice","question_span":[219,228],"option_spans":[[239,253],[259,269],[275,285],[291,301],[307,317]],"option_ids":["1.5","2","3","4","6"],"logits":[-2.447477340698242,-2.089803457260132,0.2732090950012207,-3.8073301315307617,-4.122585296630859],"probs":[0.05535648763179779,0.07915972918272018,0.8409063220024109,0.01420992985367775,0.010367603041231632],"argmax_index":2,"argmax_id":"3","top2_margin":0.7617465928196907,"near_tie":false,"gold":"3","gold_correct":true},{"question_id":"oats_total","type":"choice","question_span":[339,347],"option_spans":[[358,376],[382,400],[406,421],[427,442]],"option_ids":["1000","1250","500","750"],"logits":[-1.553785800933838,-2.819949150085449,-3.9256703853607178,1.1058349609375],"probs":[0.06383215636014938,0.017994998022913933,0.005955840460956097,0.9122170209884644],"argmax_index":3,"argmax_id":"750","top2_margin":0.848384864628315,"near_tie":false,"gold":"750","gold_correct":true},{"question_id":"syrup_total","type":"choice","question_span":[464,475],"option_spans":[[486,501],[507,522],[528,541],[547,560]],"option_ids":["240","320","60","80"],"logits":[0.9443778395652771,-2.394657611846924,-4.232697010040283,-3.6887431144714355],"probs":[0.9516197443008423,0.03375504910945892,0.005371410865336657,0.009253894910216331],"argmax_index":0,"argmax_id":"240","top2_margin":0.9178646951913834,"near_tie":false,"gold":"240","gold_correct":true},{"question_id":"oats_one_weighing","type":"noul","question_span":[586,602],"option_spans":[[613,631],[637,655]],"option_ids":["true","false"],"logits":[-0.9096747040748596,0.16398903727531433],"probs":[0.2547069489955902,0.7452929615974426],"argmax_index":1,"argmax_id":"false","top2_margin":0.4905860126018524,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"asks_swap","type":"noul","question_span":[678,688],"option_spans":[[699,717],[723,741]],"option_ids":["true","false"],"logits":[1.195202112197876,-2.503868341445923],"probs":[0.9758510589599609,0.024148916825652122],"argmax_index":0,"argmax_id":"true","top2_margin":0.9517021421343088,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"oven_temperature","type":"choice","question_span":[763,771],"option_spans":[[782,798],[804,820],[826,842],[848,864]],"option_ids":["160","170","180","200"],"logits":[-3.9338855743408203,-3.9635651111602783,0.8915537595748901,-3.9012715816497803],"probs":[0.007834219373762608,0.007605121936649084,0.976466715335846,0.0080939382314682],"argmax_index":2,"argmax_id":"180","top2_margin":0.9683727771043777,"near_tie":false,"gold":"180","gold_correct":true},{"question_id":"questions_asked","type":"choice","question_span":[887,895],"option_spans":[[906,916],[922,932],[938,948],[954,964]],"option_ids":["1","2","3","4"],"logits":[-3.865295171737671,-0.1368631273508072,-0.1657632738351822,-0.9524722099304199],"probs":[0.009856987744569778,0.410186767578125,0.3985019624233246,0.18145422637462616],"argmax_index":1,"argmax_id":"2","top2_margin":0.011684805154800415,"near_tie":true,"gold":"2","gold_correct":true},{"question_id":"difficulty","type":"score","question_span":[985,992],"option_spans":[[1003,1014],[1020,1030],[1036,1047],[1053,1064]],"option_ids":["0","1","2","3"],"logits":[-2.3407249450683594,-1.5610216856002808,0.649451494216919,-1.3033589124679565],"probs":[0.03862227872014046,0.08422844111919403,0.7681658864021301,0.10898337513208389],"argmax_index":2,"argmax_id":"2","top2_margin":0.6591825112700462,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"factor":{"type":"choice","choice":"3","confidence":0.8409,"probabilities":{"1.5":0.0554,"2":0.0792,"3":0.8409,"4":0.0142,"6":0.0104}},"oats_total":{"type":"choice","choice":"750","confidence":0.9122,"probabilities":{"500":0.006,"750":0.9122,"1000":0.0638,"1250":0.018}},"syrup_total":{"type":"choice","choice":"240","confidence":0.9516,"probabilities":{"60":0.0054,"80":0.0093,"240":0.9516,"320":0.0338}},"oats_one_weighing":{"type":"noul","noul":0.2547},"asks_swap":{"type":"noul","noul":0.9759},"oven_temperature":{"type":"choice","choice":"180","confidence":0.9765,"probabilities":{"160":0.0078,"170":0.0076,"180":0.9765,"200":0.0081}},"questions_asked":{"type":"choice","choice":"2","confidence":0.4102,"probabilities":{"1":0.0099,"2":0.4102,"3":0.3985,"4":0.1815}},"difficulty":{"type":"score","score":1.9475,"confidence":0.7682,"legend":{"0":"trivial","1":"easy","2":"moderate","3":"tricky"},"probabilities":{"0":0.0386,"1":0.0842,"2":0.7682,"3":0.109}}},"usage":{"input_tokens":1086,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_t08","arm":"text","set":"heldout","source":"heldout_text","tokens":864,"n_image_tokens":0,"state_tokens":175,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,13429,13322,12,19,19,22,16,17,15,11,326,6957,2573,327,17558,17227,11,3074,8151,494,220,18,310,220,16,22,5963,13,2921,20275,10650,2046,557,24863,1521,314,264,12787,11,321,353,995,310,4565,1330,4799,20803,303,264,9220,1518,279,31402,1362,311,2121,728,13,353,6945,220,17,18,21,364,279,9220,321,220,19,16,364,20329,321,353,599,1396,21867,13,561,31402,15578,310,2216,11,5315,264,12787,369,4732,1141,2444,13,2921,4687,58196,14103,359,46894,7383,314,777,1056,220,16,17,3979,6,685,310,220,18,15,15,11,440,449,13230,314,220,20,15,791,3591,11,321,61228,359,15958,82,421,2361,2724,682,7003,310,20325,4282,2921,84821,1048,11136,264,1834,3200,11,694,4161,303,424,557,5384,13,3437,628,353,3591,11,321,635,353,1144,264,5158,4952,494,279,31402,30,18982,7725,50585,35366,77,18556,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,4687,15743,198,9222,25,5576,198,669,55530,25,220,22365,3644,314,279,4687,17759,279,1822,3591,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,45677,466,21635,58508,2129,1979,816,3147,20598,14511,8934,198,32929,220,17,25,220,4754,4532,3147,34,1093,6095,279,8151,1518,9101,2129,1979,816,3147,66,47567,8934,198,32929,220,18,25,220,4754,4532,3147,38916,7383,2129,1979,816,3147,21572,8934,198,32929,220,19,25,220,4754,4532,3147,23500,310,975,1208,466,836,3234,2129,1979,816,3147,719,2803,8934,198,32929,220,20,25,220,4754,4532,3147,76266,6255,6829,2129,1979,816,3147,65817,8934,198,4537,38945,198,198,27806,220,17,198,887,25,6829,10467,198,9222,25,5576,198,669,55530,25,220,3710,635,279,9220,321,14704,6829,884,685,310,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,17,18,21,2129,1979,816,3147,17,18,21,8934,198,32929,220,17,25,220,4754,4532,3147,17,22,22,2129,1979,816,3147,17,22,22,8934,198,32929,220,18,25,220,4754,4532,3147,17,23,21,2129,1979,816,3147,17,23,21,8934,198,32929,220,19,25,220,4754,4532,3147,18,15,15,2129,1979,816,3147,18,15,15,8934,198,4537,38945,198,198,27806,220,18,198,887,25,44359,198,9222,25,5576,198,669,55530,25,220,2592,279,3591,369,11330,303,2400,11,1204,1691,668,279,70070,2216,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,17,17,22,2129,1979,816,3147,17,17,22,8934,198,32929,220,17,25,220,4754,4532,3147,17,20,15,2129,1979,816,3147,17,20,15,8934,198,32929,220,18,25,220,4754,4532,3147,17,22,22,2129,1979,816,3147,17,22,22,8934,198,32929,220,19,25,220,4754,4532,3147,18,15,15,2129,1979,816,3147,18,15,15,8934,198,4537,38945,198,198,27806,220,19,198,887,25,31402,61072,198,9222,25,307,10644,198,669,55530,25,220,760,31402,682,7003,310,20325,279,6829,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,20,198,887,25,58508,85608,198,9222,25,307,10644,198,669,55530,25,220,8114,314,279,82255,579,58508,557,5384,364,1603,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,21,198,887,25,30246,198,9222,25,5284,198,669,55530,25,220,549,10432,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1299,1493,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,60777,2708,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,1164,2708,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"policy_section","type":"choice","question_span":[236,246],"option_spans":[[257,271],[277,294],[300,311],[317,334],[340,352]],"option_ids":["baggage","cancellation","delay","liability","medical"],"logits":[-3.399714708328247,-3.5896201133728027,0.6478246450424194,-4.192816734313965,-4.449000358581543],"probs":[0.016698379069566727,0.013810181058943272,0.9560889601707458,0.007554999087005854,0.005847567692399025],"argmax_index":2,"argmax_id":"delay","top2_margin":0.9393905811011791,"near_tie":false,"gold":"delay","gold_correct":true},{"question_id":"costs_total","type":"choice","question_span":[374,385],"option_spans":[[396,410],[416,430],[436,450],[456,470]],"option_ids":["236","277","286","300"],"logits":[-3.273216724395752,1.8200938701629639,-2.3739781379699707,-3.945863723754883],"probs":[0.005991741083562374,0.9762243628501892,0.014726088382303715,0.0030579189769923687],"argmax_index":1,"argmax_id":"277","top2_margin":0.9614982744678855,"near_tie":false,"gold":"277","gold_correct":true},{"question_id":"payout","type":"choice","question_span":[491,506],"option_spans":[[517,531],[537,551],[557,571],[577,591]],"option_ids":["227","250","277","300"],"logits":[-0.3858455419540405,-1.8905377388000488,-2.38273549079895,-2.8081820011138916],"probs":[0.6912968754768372,0.15352711081504822,0.09384827315807343,0.061327774077653885],"argmax_index":0,"argmax_id":"227","top2_margin":0.5377697646617889,"near_tie":false,"gold":"227","gold_correct":true},{"question_id":"airline_paid","type":"noul","question_span":[614,623],"option_spans":[[634,652],[658,676]],"option_ids":["true","false"],"logits":[-0.9346691370010376,2.030959129333496],"probs":[0.049003057181835175,0.9509969353675842],"argmax_index":1,"argmax_id":"false","top2_margin":0.901993878185749,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"baggage_lost","type":"noul","question_span":[699,710],"option_spans":[[721,739],[745,763]],"option_ids":["true","false"],"logits":[-2.703249216079712,2.024325370788574],"probs":[0.008770307525992393,0.9912296533584595],"argmax_index":1,"argmax_id":"false","top2_margin":0.9824593458324671,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"clarity","type":"score","question_span":[784,786],"option_spans":[[797,808],[814,825],[831,842]],"option_ids":["0","1","2"],"logits":[-0.6285924315452576,0.5291945338249207,-0.7551361322402954],"probs":[0.19747166335582733,0.6285289525985718,0.1739993393421173],"argmax_index":1,"argmax_id":"1","top2_margin":0.43105728924274445,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"policy_section":{"type":"choice","choice":"delay","confidence":0.9561,"probabilities":{"medical":0.0058,"delay":0.9561,"baggage":0.0167,"cancellation":0.0138,"liability":0.0076}},"costs_total":{"type":"choice","choice":"277","confidence":0.9762,"probabilities":{"236":0.006,"277":0.9762,"286":0.0147,"300":0.0031}},"payout":{"type":"choice","choice":"227","confidence":0.6913,"probabilities":{"227":0.6913,"250":0.1535,"277":0.0938,"300":0.0613}},"airline_paid":{"type":"noul","noul":0.049},"baggage_lost":{"type":"noul","noul":0.0088},"clarity":{"type":"score","score":0.9765,"confidence":0.6285,"legend":{"0":"unclear","1":"mostly clear","2":"very clear"},"probabilities":{"0":0.1975,"1":0.6285,"2":0.174}}},"usage":{"input_tokens":864,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_t09","arm":"text","set":"heldout","source":"heldout_text","tokens":867,"n_image_tokens":0,"state_tokens":197,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,29209,64291,11,710,4240,86,787,20271,14066,13,29577,1589,627,629,17386,31800,68314,383,87071,321,6318,4717,13382,494,220,16,21,25,15,15,310,220,16,22,25,18,15,303,2956,220,17,13,10731,29120,496,156095,20664,31800,28829,383,79831,13382,494,220,16,22,25,15,15,310,220,16,23,25,15,15,303,2956,220,20,11,321,6486,279,1788,17225,2767,383,79833,13382,506,220,16,23,25,15,15,13,8618,57948,1189,2236,506,1379,2250,15243,264,1936,13,85611,335,315,360,318,9210,220,16,15,8,682,8693,4706,1396,7348,494,220,16,21,25,18,15,310,220,16,23,25,18,15,321,6587,1330,68314,15243,321,799,28829,3669,1754,1936,13,733,4064,318,9210,220,16,17,8,369,1845,1396,1834,1238,220,16,20,25,18,15,694,682,874,20012,506,2046,13,561,11911,369,7629,383,7670,220,17,16,6324,364,5395,4706,26,279,17820,314,421,1834,3166,310,6394,220,17,18,6324,506,279,1788,2942,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,3498,337,335,315,360,62,49914,198,9222,25,5576,198,669,55530,25,220,22365,28829,3669,628,85611,335,315,360,9138,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,60099,2129,1979,816,3147,785,2405,8934,198,32929,220,17,25,220,4754,4532,3147,36539,2767,2129,1979,816,3147,337,6955,8934,198,32929,220,18,25,220,4754,4532,3147,38867,303,2956,220,20,2129,1979,816,3147,83,6827,8934,198,4537,38945,198,198,27806,220,17,198,887,25,6537,1805,879,15905,198,9222,25,307,10644,198,669,55530,25,220,623,4064,682,264,20012,506,2046,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,2785,14335,198,9222,25,307,10644,198,669,55530,25,220,3318,337,335,315,360,579,34990,14751,15243,4565,2785,279,16556,21185,3798,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,7481,62,26603,198,9222,25,5576,198,669,55530,25,220,3710,8311,310,279,68314,17225,314,7670,220,17,16,6324,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,2064,369,24863,2129,1979,816,3147,91993,8934,198,32929,220,17,25,220,4754,4532,3147,2064,10480,310,6394,220,17,18,6324,11,1788,854,2129,1979,816,3147,188335,62,17,18,8934,198,32929,220,18,25,220,4754,4532,3147,2064,369,5467,2767,421,1834,2129,1979,816,3147,24831,8934,198,32929,220,19,25,220,4754,4532,3147,2064,10480,310,7548,220,17,17,6324,2129,1979,816,3147,199231,62,17,17,8934,198,4537,38945,198,198,27806,220,20,198,887,25,68314,4956,198,9222,25,5576,198,669,55530,25,220,4199,1248,369,799,68314,3669,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16,17,15,4276,2129,1979,816,3147,16,17,15,8934,198,32929,220,17,25,220,4754,4532,3147,21,15,4276,2129,1979,816,3147,21,15,8934,198,32929,220,18,25,220,4754,4532,3147,22,20,4276,2129,1979,816,3147,22,20,8934,198,32929,220,19,25,220,4754,4532,3147,24,15,4276,2129,1979,816,3147,24,15,8934,198,4537,38945,198,198,27806,220,21,198,887,25,8842,50199,366,198,9222,25,5284,198,669,55530,25,220,4199,2503,369,411,64291,310,3019,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,22230,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,24321,470,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,5511,13330,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"bartholomew_chemistry","type":"choice","question_span":[263,273],"option_spans":[[284,295],[301,313],[319,334]],"option_ids":["neither","thursday","tuesday"],"logits":[-1.7182679176330566,-2.3459908962249756,-2.5321903228759766],"probs":[0.5058367252349854,0.27001863718032837,0.22414463758468628],"argmax_index":0,"argmax_id":"neither","top2_margin":0.23581808805465698,"near_tie":false,"gold":"thursday","gold_correct":false},{"question_id":"ingrid_laptop","type":"noul","question_span":[359,367],"option_spans":[[378,396],[402,420]],"option_ids":["true","false"],"logits":[-3.3143234252929688,1.4790667295455933],"probs":[0.008216258138418198,0.9917837381362915],"argmax_index":1,"argmax_id":"false","top2_margin":0.9835674799978733,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"within_limit","type":"noul","question_span":[443,459],"option_spans":[[470,488],[494,512]],"option_ids":["true","false"],"logits":[-0.21831604838371277,-1.5075669288635254],"probs":[0.7840203642845154,0.21597963571548462],"argmax_index":0,"argmax_id":"true","top2_margin":0.5680407285690308,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"moved_lesson","type":"choice","question_span":[535,548],"option_spans":[[559,571],[577,600],[606,621],[627,647]],"option_ids":["cancelled","fri_23","online","thu_22"],"logits":[-4.2007975578308105,1.717532753944397,-4.870396614074707,-4.342280387878418],"probs":[0.002672579139471054,0.9936392903327942,0.0013681302079930902,0.0023199860006570816],"argmax_index":1,"argmax_id":"fri_23","top2_margin":0.9909667111933231,"near_tie":false,"gold":"fri_23","gold_correct":true},{"question_id":"maths_length","type":"choice","question_span":[669,676],"option_spans":[[687,702],[708,721],[727,740],[746,759]],"option_ids":["120","60","75","90"],"logits":[-1.9204975366592407,-3.7826688289642334,-1.7469218969345093,1.0338929891586304],"probs":[0.04643623158335686,0.007213171571493149,0.05523824691772461,0.8911123275756836],"argmax_index":3,"argmax_id":"90","top2_margin":0.835874080657959,"near_tie":false,"gold":"90","gold_correct":true},{"question_id":"planning_effort","type":"score","question_span":[782,790],"option_spans":[[801,811],[817,828],[834,845]],"option_ids":["0","1","2"],"logits":[-2.227717161178589,-0.47533100843429565,-0.5494868159294128],"probs":[0.08247818052768707,0.4757630228996277,0.44175881147384644],"argmax_index":1,"argmax_id":"1","top2_margin":0.03400421142578125,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"bartholomew_chemistry":{"type":"choice","choice":"neither","confidence":0.5058,"probabilities":{"tuesday":0.2241,"thursday":0.27,"neither":0.5058}},"ingrid_laptop":{"type":"noul","noul":0.0082},"within_limit":{"type":"noul","noul":0.784},"moved_lesson":{"type":"choice","choice":"fri_23","confidence":0.9936,"probabilities":{"cancelled":0.0027,"fri_23":0.9936,"thu_22":0.0023,"online":0.0014}},"maths_length":{"type":"choice","choice":"90","confidence":0.8911,"probabilities":{"60":0.0072,"75":0.0552,"90":0.8911,"120":0.0464}},"planning_effort":{"type":"score","score":1.3593,"confidence":0.4758,"legend":{"0":"simple","1":"manageable","2":"complicated"},"probabilities":{"0":0.0825,"1":0.4758,"2":0.4418}}},"usage":{"input_tokens":867,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_t10","arm":"text","set":"heldout","source":"heldout_text","tokens":800,"n_image_tokens":0,"state_tokens":134,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,674,6976,310,279,5930,4125,314,279,783,5855,31788,57939,13,198,52401,7287,25,81130,279,4400,883,279,2235,2463,13,5044,1133,777,2136,3712,13,198,33,1820,5923,25,13065,2166,1248,411,2182,13,353,6348,685,1238,2081,4590,13,198,56,4958,4229,667,25,10791,364,2583,279,491,9261,4281,2756,13,8771,314,279,88250,10926,998,4808,11,3384,13,198,42353,89,379,25,4254,2136,3712,11,4338,0,561,23487,998,16200,2166,13,198,47700,39311,25,16018,488,2842,16506,424,310,728,383,5392,30,353,1000,4598,1301,424,383,821,4034,13,198,52401,7287,25,353,7268,440,12694,39311,11,279,5392,10672,615,66544,303,821,6321,1944,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,1379,1611,7327,198,9222,25,5576,198,669,55530,25,220,22365,8282,635,12361,2466,310,1436,777,314,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,84806,3543,2129,1979,816,3147,66,6480,8934,198,32929,220,17,25,220,4754,4532,3147,7090,3712,2129,1979,816,3147,2359,19197,8934,198,32929,220,18,25,220,4754,4532,3147,47,8621,624,2129,1979,816,3147,79,8621,624,8934,198,32929,220,19,25,220,4754,4532,3147,67488,2129,1979,816,3147,48181,8934,198,32929,220,20,25,220,4754,4532,3147,56750,2129,1979,816,3147,58849,8934,198,4537,38945,198,198,27806,220,17,198,887,25,16036,14384,290,198,9222,25,307,10644,198,669,55530,25,220,32,6396,3387,680,449,1412,303,279,4125,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,660,52339,198,9222,25,307,10644,198,669,55530,25,220,11179,9559,369,6572,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,7143,6241,382,198,9222,25,5576,198,669,55530,25,220,4199,1599,2086,12361,1318,264,7143,2880,4370,314,5392,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,6487,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,4537,38945,198,198,27806,220,20,198,887,25,9559,3079,198,9222,25,5284,198,669,55530,25,220,4199,1599,29636,513,9711,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,588,466,1330,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,26952,466,2943,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,50897,466,4590,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,6183,1056,4590,2129,1979,816,3147,18,8934,198,4537,38945,198,198,27806,220,21,198,887,25,22396,198,9222,25,5284,198,669,55530,25,220,4199,1575,557,279,4125,3816,7830,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,13458,391,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,4058,9238,15191,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,8884,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,1164,1575,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"most_wanted","type":"choice","question_span":[196,206],"option_spans":[[217,229],[235,247],[253,267],[273,283],[289,299]],"option_ids":["council","local_history","puzzles","recipes","sport"],"logits":[-4.49011754989624,0.774727463722229,-3.933271646499634,-5.372121334075928,-5.381842613220215],"probs":[0.005076519679278135,0.9818816781044006,0.008859334513545036,0.0021014385856688023,0.0020811089780181646],"argmax_index":1,"argmax_id":"local_history","top2_margin":0.9730223435908556,"near_tie":false,"gold":"local_history","gold_correct":true},{"question_id":"mistake_reported","type":"noul","question_span":[323,333],"option_spans":[[344,362],[368,386]],"option_ids":["true","false"],"logits":[-1.305759310722351,-0.9163533449172974],"probs":[0.40386030077934265,0.596139669418335],"argmax_index":1,"argmax_id":"false","top2_margin":0.1922793686389923,"near_tie":false,"gold":"true","gold_correct":false},{"question_id":"all_positive","type":"noul","question_span":[409,414],"option_spans":[[425,443],[449,467]],"option_ids":["true","false"],"logits":[-1.8824115991592407,1.896518588066101],"probs":[0.02233678661286831,0.9776631593704224],"argmax_index":1,"argmax_id":"false","top2_margin":0.955326372757554,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"digital_readers","type":"choice","question_span":[490,502],"option_spans":[[513,523],[529,539],[545,555],[561,571]],"option_ids":["0","1","2","3"],"logits":[-3.5014920234680176,-2.364795446395874,0.26134103536605835,-3.094998836517334],"probs":[0.020538825541734695,0.06400835514068604,0.8846129179000854,0.03083992935717106],"argmax_index":2,"argmax_id":"2","top2_margin":0.8206045627593994,"near_tie":false,"gold":"2","gold_correct":true},{"question_id":"reply_count","type":"score","question_span":[593,599],"option_spans":[[610,622],[628,640],[646,658],[664,676]],"option_ids":["0","1","2","3"],"logits":[-3.262695074081421,-0.9365512132644653,-0.18822035193443298,-3.309251070022583],"probs":[0.0295583326369524,0.30262959003448486,0.6395983695983887,0.028213756158947945],"argmax_index":2,"argmax_id":"2","top2_margin":0.3369687795639038,"near_tie":false,"gold":"2","gold_correct":true},{"question_id":"reception","type":"score","question_span":[697,705],"option_spans":[[716,727],[733,745],[751,761],[767,778]],"option_ids":["0","1","2","3"],"logits":[-2.602142333984375,0.7380218505859375,-0.12349213659763336,-2.3334150314331055],"probs":[0.023553121834993362,0.6647576689720154,0.28087469935417175,0.030814511701464653],"argmax_index":1,"argmax_id":"1","top2_margin":0.38388296961784363,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"most_wanted":{"type":"choice","choice":"local_history","confidence":0.9819,"probabilities":{"local_history":0.9819,"recipes":0.0021,"sport":0.0021,"puzzles":0.0089,"council":0.0051}},"mistake_reported":{"type":"noul","noul":0.4039},"all_positive":{"type":"noul","noul":0.0223},"digital_readers":{"type":"choice","choice":"2","confidence":0.8846,"probabilities":{"0":0.0205,"1":0.064,"2":0.8846,"3":0.0308}},"reply_count":{"type":"score","score":1.6665,"confidence":0.6396,"legend":{"0":"one or two","1":"three or four","2":"five or six","3":"more than six"},"probabilities":{"0":0.0296,"1":0.3026,"2":0.6396,"3":0.0282}},"reception":{"type":"score","score":1.319,"confidence":0.6648,"legend":{"0":"badly","1":"with mixed feelings","2":"well","3":"very well"},"probabilities":{"0":0.0236,"1":0.6648,"2":0.2809,"3":0.0308}}},"usage":{"input_tokens":800,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_j01","arm":"text","set":"heldout","source":"heldout_json","tokens":769,"n_image_tokens":0,"state_tokens":206,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,14695,3548,21624,32791,3584,763,2827,1288,19641,40003,763,18,22,13,20,1288,13382,64331,8689,7033,763,18,15,1288,993,3147,17,15,17,21,12,15,24,12,17,23,2129,258,3147,15,23,25,20,23,2129,403,3147,16,22,25,18,15,35303,8689,7033,763,18,15,1288,993,3147,17,15,17,21,12,15,24,12,17,24,2129,258,3147,15,24,25,15,20,2129,403,3147,16,22,25,18,20,35303,993,3147,17,15,17,21,12,15,24,12,18,15,2129,2738,3147,82,839,35303,8689,7033,763,19,20,1288,993,3147,17,15,17,21,12,16,15,12,15,16,2129,258,3147,15,23,25,19,20,2129,403,3147,16,24,25,16,20,35303,8689,7033,763,15,1288,993,3147,17,15,17,21,12,16,15,12,15,17,2129,258,3147,15,24,25,15,15,2129,9679,3147,35778,10902,5106,2129,403,3147,16,18,25,15,15,8934,27573,12575,3147,43,643,833,2906,189541,2129,9958,4747,286,3147,17,15,17,21,12,15,24,12,17,23,29958,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,13633,27430,198,9222,25,5576,198,669,55530,25,220,4199,1599,13633,2756,513,12076,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,6487,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,4537,38945,198,198,27806,220,17,198,887,25,21354,16271,198,9222,25,5576,198,669,55530,25,220,1860,864,2321,557,279,21354,6206,6237,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,34072,2129,1979,816,3147,17,15,17,21,12,15,24,12,17,23,8934,198,32929,220,17,25,220,4754,4532,3147,38867,2129,1979,816,3147,17,15,17,21,12,15,24,12,17,24,8934,198,32929,220,18,25,220,4754,4532,3147,36539,2129,1979,816,3147,17,15,17,21,12,16,15,12,15,16,8934,198,32929,220,19,25,220,4754,4532,3147,33475,2129,1979,816,3147,17,15,17,21,12,16,15,12,15,17,8934,198,4537,38945,198,198,27806,220,18,198,887,25,11454,198,9222,25,307,10644,198,669,55530,25,220,52681,682,11454,279,2942,3548,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,3200,4747,198,9222,25,307,10644,198,669,55530,25,220,6769,279,9086,1151,3200,383,866,1834,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1860,506,3140,799,1834,279,8598,3309,854,369,2843,1056,220,15,24,25,15,15,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,11179,8598,3309,854,369,220,15,24,25,15,15,466,6575,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,20,198,887,25,3979,3273,27612,198,9222,25,5284,198,669,55530,25,220,4199,17765,513,279,3979,6237,411,1936,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,2083,4492,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,44149,4492,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,90112,4492,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"sick_days","type":"choice","question_span":[267,274],"option_spans":[[285,295],[301,311],[317,327],[333,343]],"option_ids":["0","1","2","3"],"logits":[-3.285473346710205,1.0415970087051392,-3.5007271766662598,-3.800096035003662],"probs":[0.012799806892871857,0.9692284464836121,0.010320954956114292,0.007650779094547033],"argmax_index":1,"argmax_id":"1","top2_margin":0.9564286395907402,"near_tie":false,"gold":"1","gold_correct":true},{"question_id":"longest_day","type":"choice","question_span":[365,374],"option_spans":[[385,404],[410,429],[435,454],[460,479]],"option_ids":["2026-09-28","2026-09-29","2026-10-01","2026-10-02"],"logits":[-0.9379734396934509,-2.1196460723876953,0.45527184009552,-3.483152151107788],"probs":[0.18473617732524872,0.05667063221335411,0.7440989017486572,0.014494218863546848],"argmax_index":2,"argmax_id":"2026-10-01","top2_margin":0.5593627244234085,"near_tie":false,"gold":"2026-10-01","gold_correct":true},{"question_id":"approved","type":"noul","question_span":[501,508],"option_spans":[[519,537],[543,561]],"option_ids":["true","false"],"logits":[-3.329871892929077,1.9910085201263428],"probs":[0.004864667542278767,0.9951353669166565],"argmax_index":1,"argmax_id":"false","top2_margin":0.9902706993743777,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"late_start","type":"noul","question_span":[584,593],"option_spans":[[604,631],[637,659]],"option_ids":["true","false"],"logits":[0.503396213054657,0.5012811422348022],"probs":[0.5005287528038025,0.4994712173938751],"argmax_index":0,"argmax_id":"true","top2_margin":0.0010575354099273682,"near_tie":true,"gold":"true","gold_correct":true},{"question_id":"hours_concern","type":"score","question_span":[682,691],"option_spans":[[702,713],[719,730],[736,747]],"option_ids":["0","1","2"],"logits":[-0.3749038577079773,0.34233754873275757,-1.7536165714263916],"probs":[0.30296823382377625,0.6207133531570435,0.07631848752498627],"argmax_index":1,"argmax_id":"1","top2_margin":0.3177451193332672,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"sick_days":{"type":"choice","choice":"1","confidence":0.9692,"probabilities":{"0":0.0128,"1":0.9692,"2":0.0103,"3":0.0077}},"longest_day":{"type":"choice","choice":"2026-10-01","confidence":0.7441,"probabilities":{"2026-09-28":0.1847,"2026-09-29":0.0567,"2026-10-01":0.7441,"2026-10-02":0.0145}},"approved":{"type":"noul","noul":0.0049},"late_start":{"type":"noul","noul":0.5005},"hours_concern":{"type":"score","score":0.7734,"confidence":0.6207,"legend":{"0":"no concern","1":"minor concern","2":"serious concern"},"probabilities":{"0":0.303,"1":0.6207,"2":0.0763}}},"usage":{"input_tokens":769,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_j02","arm":"text","set":"heldout","source":"heldout_json","tokens":335,"n_image_tokens":0,"state_tokens":137,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,13091,3079,21624,1767,290,4322,3147,17,15,17,21,12,15,24,12,17,22,2129,7718,64331,1767,290,763,19,17,1288,7095,763,19,17,1288,38605,3147,22456,10951,1354,42798,12,16,21,35303,1767,290,763,16,16,18,1288,7095,763,16,17,15,1288,38605,3147,22456,9009,65971,12,20,44,35303,1767,290,763,21,21,1288,7095,763,21,15,1288,38605,3147,22456,11668,1537,74139,7869,35303,1767,290,763,24,1288,7095,763,16,23,1288,38605,3147,22456,12,7444,8955,67185,35303,1767,290,763,18,20,1288,7095,763,18,20,1288,38605,3147,22456,12,896,939,2731,12630,8934,27573,4173,3147,39,935,77806,706,35547,2129,19951,261,663,27423,763,18,3307,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,7551,88894,198,9222,25,5576,198,669,55530,25,220,4199,1599,4965,1684,494,279,3481,1698,539,777,1056,279,23343,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,6487,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,21,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"beyond_tolerance","type":"choice","question_span":[198,212],"option_spans":[[223,233],[239,249],[255,265],[271,281],[287,297],[303,313]],"option_ids":["0","1","2","3","4","5"],"logits":[-3.9004204273223877,-2.9196300506591797,-1.1734979152679443,-0.9318141937255859,-3.0034804344177246,-4.12604284286499],"probs":[0.023999681696295738,0.06399666517972946,0.3668537139892578,0.4671483337879181,0.05884933844208717,0.019152212888002396],"argmax_index":3,"argmax_id":"3","top2_margin":0.10029461979866028,"near_tie":false,"gold":"3","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"beyond_tolerance":{"type":"choice","choice":"3","confidence":0.4671,"probabilities":{"0":0.024,"1":0.064,"2":0.3669,"3":0.4671,"4":0.0588,"5":0.0192}}},"usage":{"input_tokens":335,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_j03","arm":"text","set":"heldout","source":"heldout_json","tokens":775,"n_image_tokens":0,"state_tokens":137,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,20327,9410,8631,17,11,23,27573,12569,64331,1634,3147,5042,2129,5961,763,19,13,16,1288,1619,3147,15,22,25,18,15,35303,1634,3147,5042,2129,5961,763,20,13,15,1288,1619,3147,16,16,25,18,15,35303,1634,3147,5434,2129,5989,3147,10470,1669,217813,1238,9403,2129,5961,763,23,13,24,1288,1619,3147,16,20,25,18,15,35303,1634,3147,5434,2129,5961,763,22,13,17,1288,1619,3147,16,21,25,15,15,35303,1634,3147,5434,2129,5961,763,18,13,21,1288,1619,3147,16,24,25,18,15,8934,27573,813,3147,85,84989,36337,417,2129,10957,11847,18410,21624,2721,763,24,13,19,1288,1030,763,18,13,16,49471,3715,3147,11283,34,8934,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,91083,198,9222,25,307,10644,198,669,55530,25,220,32,5122,4732,279,5252,2020,557,13332,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,1118,44127,33146,198,9222,25,5576,198,669,55530,25,220,1597,864,13332,854,557,279,2020,1118,32689,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,15,22,25,18,15,2129,1979,816,3147,15,22,25,18,15,8934,198,32929,220,17,25,220,4754,4532,3147,16,16,25,18,15,2129,1979,816,3147,16,16,25,18,15,8934,198,32929,220,18,25,220,4754,4532,3147,16,20,25,18,15,2129,1979,816,3147,16,20,25,18,15,8934,198,32929,220,19,25,220,4754,4532,3147,16,21,25,15,15,2129,1979,816,3147,16,21,25,15,15,8934,198,32929,220,20,25,220,4754,4532,3147,16,24,25,18,15,2129,1979,816,3147,16,24,25,18,15,8934,198,32929,220,21,25,220,4754,4532,3147,35288,2129,1979,816,3147,6487,8934,198,4537,38945,198,198,27806,220,18,198,887,25,5073,198,9222,25,5576,198,669,55530,25,220,3710,1503,279,1433,4052,430,279,5073,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,32,687,40128,10458,2129,1979,816,3147,727,40128,8934,198,32929,220,17,25,220,4754,4532,3147,760,5817,557,524,9035,9971,2129,1979,816,3147,10470,8934,198,32929,220,18,25,220,4754,4532,3147,32,2279,3799,2129,1979,816,3147,12793,8934,198,32929,220,19,25,220,4754,4532,3147,32,55639,11661,2129,1979,816,3147,28549,8934,198,4537,38945,198,198,27806,220,19,198,887,25,24530,198,9222,25,307,10644,198,669,55530,25,220,1311,279,1483,4202,279,9039,369,1142,4613,279,5252,2020,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,20,198,887,25,5048,198,9222,25,5284,198,669,55530,25,220,4199,5812,369,411,364,279,9476,37903,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,27833,7464,1196,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,9997,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,11553,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,323,18706,2129,1979,816,3147,19,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"excursion","type":"noul","question_span":[198,207],"option_spans":[[218,236],[242,260]],"option_ids":["true","false"],"logits":[-1.857033610343933,-4.182061672210693],"probs":[0.9109287261962891,0.08907125890254974],"argmax_index":0,"argmax_id":"true","top2_margin":0.8218574672937393,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"first_excursion","type":"choice","question_span":[283,293],"option_spans":[[304,322],[328,346],[352,370],[376,394],[400,418],[424,434]],"option_ids":["07:30","11:30","15:30","16:00","19:30","none"],"logits":[-4.615559101104736,-4.408828258514404,-0.2694721817970276,-4.427146911621094,-4.437065124511719,-4.2234296798706055],"probs":[0.012006472796201706,0.014763779006898403,0.9266101121902466,0.0144957872107625,0.014352725818753242,0.017771132290363312],"argmax_index":2,"argmax_id":"15:30","top2_margin":0.9088389798998833,"near_tie":false,"gold":"15:30","gold_correct":true},{"question_id":"cause","type":"choice","question_span":[455,464],"option_spans":[[475,489],[495,510],[516,528],[534,546]],"option_ids":["defrost","door","power","sensor"],"logits":[-4.319239616394043,-0.20686215162277222,-4.413272380828857,-4.370155334472656],"probs":[0.0156366266310215,0.9552696943283081,0.014233286492526531,0.014860405586659908],"argmax_index":1,"argmax_id":"door","top2_margin":0.9396330676972866,"near_tie":false,"gold":"door","gold_correct":true},{"question_id":"recovered","type":"noul","question_span":[568,581],"option_spans":[[592,610],[616,634]],"option_ids":["true","false"],"logits":[-1.8721755743026733,-4.337958812713623],"probs":[0.9217079877853394,0.07829198986291885],"argmax_index":0,"argmax_id":"true","top2_margin":0.8434159979224205,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"risk","type":"score","question_span":[655,664],"option_spans":[[675,687],[693,703],[709,720],[726,736],[742,753]],"option_ids":["0","1","2","3","4"],"logits":[-2.9973902702331543,-2.417109489440918,-0.7302858233451843,-1.7584794759750366,-2.7989678382873535],"probs":[0.058447808027267456,0.10441934317350388,0.5641039609909058,0.2017531394958496,0.07127577811479568],"argmax_index":2,"argmax_id":"2","top2_margin":0.36235082149505615,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"excursion":{"type":"noul","noul":0.9109},"first_excursion":{"type":"choice","choice":"15:30","confidence":0.9266,"probabilities":{"07:30":0.012,"11:30":0.0148,"15:30":0.9266,"16:00":0.0145,"19:30":0.0144,"none":0.0178}},"cause":{"type":"choice","choice":"door","confidence":0.9553,"probabilities":{"door":0.9553,"power":0.0142,"defrost":0.0156,"sensor":0.0149}},"recovered":{"type":"noul","noul":0.9217},"risk":{"type":"score","score":2.123,"confidence":0.5641,"legend":{"0":"negligible","1":"low","2":"moderate","3":"high","4":"severe"},"probabilities":{"0":0.0584,"1":0.1044,"2":0.5641,"3":0.2018,"4":0.0713}}},"usage":{"input_tokens":775,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_j04","arm":"text","set":"heldout","source":"heldout_json","tokens":610,"n_image_tokens":0,"state_tokens":138,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,3219,3147,17,15,17,21,12,16,15,12,16,21,51,16,17,25,15,15,2129,46096,21624,66,47567,3147,10279,2980,220,19,23,3979,1518,279,21185,26,279,16051,369,8341,1238,421,2129,993,3147,17,15,17,21,12,16,15,12,16,22,2129,58476,21624,5856,763,21,15,1288,33530,763,1802,1288,6081,763,1802,49471,17742,3147,985,17053,13459,11,264,11012,8858,413,3102,2129,31864,2291,763,21,1288,1062,3147,37667,12,18,18,16,23,2129,67752,64331,29192,763,17,1288,16546,3147,22608,57638,35303,29192,763,20,1288,16546,3147,9239,12217,2528,8934,27573,2738,3147,39673,2129,1619,3147,16,24,25,18,15,2129,6917,3147,81870,455,85,2077,232049,29958,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,1845,648,47567,198,9222,25,307,10644,198,669,55530,25,220,34,1093,6095,1381,1000,2695,4161,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,4464,198,9222,25,5576,198,669,55530,25,220,4199,1599,1208,369,279,1898,364,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,17,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,18,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,32929,220,19,25,220,4754,4532,3147,65254,2129,1979,816,3147,23,8934,198,4537,38945,198,198,27806,220,18,198,887,25,9488,69078,198,9222,25,5576,198,669,55530,25,220,22365,15808,1902,279,9488,3372,440,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,21624,14165,3147,9239,12217,36064,2129,37537,3584,3147,6735,314,3626,56658,1979,816,3147,5045,8934,198,32929,220,17,25,220,4754,4532,21624,14165,3147,276,57638,2129,37537,3584,3147,74,7219,56658,1979,816,3147,528,40558,8934,198,32929,220,18,25,220,4754,4532,21624,14165,3147,64,10407,364,264,2526,1623,2129,37537,3584,3147,6735,314,3626,56658,1979,816,3147,11553,4002,1264,8934,198,32929,220,19,25,220,4754,4532,21624,14165,3147,64,5819,5792,4861,2129,37537,3584,3147,74,7219,56658,1979,816,3147,571,28468,8934,198,4537,38945,198,198,27806,220,19,198,887,25,3175,55374,17236,198,9222,25,307,10644,198,669,55530,25,220,14703,55374,17236,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"free_cancellation","type":"noul","question_span":[201,209],"option_spans":[[220,238],[244,262]],"option_ids":["true","false"],"logits":[-2.955127239227295,-1.4361668825149536],"probs":[0.17961466312408447,0.8203853368759155],"argmax_index":1,"argmax_id":"false","top2_margin":0.640770673751831,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"party","type":"choice","question_span":[283,291],"option_spans":[[302,312],[318,328],[334,344],[350,360]],"option_ids":["2","4","6","8"],"logits":[-5.078245639801025,-5.018433570861816,-0.6405230760574341,-5.020883560180664],"probs":[0.011402147822082043,0.0121049415320158,0.964417576789856,0.01207531988620758],"argmax_index":2,"argmax_id":"6","top2_margin":0.9523126352578402,"near_tie":false,"gold":"6","gold_correct":true},{"question_id":"kitchen_need","type":"choice","question_span":[382,390],"option_spans":[[401,422],[428,448],[454,480],[486,508]],"option_ids":["access","allergy","high_chair","vegan"],"logits":[-4.07225227355957,-0.4666510224342346,-4.985811710357666,-5.050808429718018],"probs":[0.025919659063220024,0.9539421796798706,0.010396220721304417,0.009741991758346558],"argmax_index":1,"argmax_id":"allergy","top2_margin":0.9280225206166506,"near_tie":false,"gold":"allergy","gold_correct":true},{"question_id":"special_occasion","type":"noul","question_span":[532,535],"option_spans":[[546,564],[570,588]],"option_ids":["true","false"],"logits":[-1.6849350929260254,-4.148594379425049],"probs":[0.9215545654296875,0.07844538986682892],"argmax_index":0,"argmax_id":"true","top2_margin":0.8431091755628586,"near_tie":false,"gold":"true","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"free_cancellation":{"type":"noul","noul":0.1796},"party":{"type":"choice","choice":"6","confidence":0.9644,"probabilities":{"2":0.0114,"4":0.0121,"6":0.9644,"8":0.0121}},"kitchen_need":{"type":"choice","choice":"allergy","confidence":0.9539,"probabilities":{"allergy":0.9539,"access":0.0259,"high_chair":0.0104,"vegan":0.0097}},"special_occasion":{"type":"noul","noul":0.9216}},"usage":{"input_tokens":610,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_j05","arm":"text","set":"heldout","source":"heldout_json","tokens":775,"n_image_tokens":0,"state_tokens":175,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,15501,3147,52897,2129,4913,33169,20477,3147,20,13,15,15,364,1754,7903,1801,777,1056,220,20,2756,1238,279,4016,1834,2129,9311,763,17,15,19,16,22,1288,60917,64331,5856,763,18,24,13,15,1288,9943,3147,17,15,17,21,12,15,21,2129,33530,4322,3147,17,15,17,21,12,15,21,12,15,16,35303,5856,763,18,24,13,15,1288,9943,3147,17,15,17,21,12,15,22,2129,33530,4322,3147,17,15,17,21,12,15,22,12,15,24,35303,5856,763,15,13,15,1288,9943,3147,17,15,17,21,12,15,23,2129,33530,4322,763,2827,35607,5856,763,22,23,13,15,1288,9943,3147,17,15,17,21,12,15,24,2129,33530,4322,3147,17,15,17,21,12,15,24,12,15,17,8934,27573,10083,21624,22906,16271,763,16,1288,29035,763,18,24,13,15,1288,591,3147,6614,463,86,14517,44014,14647,29958,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,13235,18362,198,9222,25,5576,198,669,55530,25,220,22365,2182,682,874,7903,12076,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,26286,2129,1979,816,3147,17,15,17,21,12,15,21,8934,198,32929,220,17,25,220,4754,4532,3147,27499,2129,1979,816,3147,17,15,17,21,12,15,22,8934,198,32929,220,18,25,220,4754,4532,3147,30425,2129,1979,816,3147,17,15,17,21,12,15,23,8934,198,32929,220,19,25,220,4754,4532,3147,28585,2129,1979,816,3147,17,15,17,21,12,15,24,8934,198,32929,220,20,25,220,4754,4532,3147,6487,2129,1979,816,3147,6487,8934,198,4537,38945,198,198,27806,220,17,198,887,25,492,3770,879,344,198,9222,25,307,10644,198,669,55530,25,220,760,5589,7903,557,1801,777,1056,4097,2756,1238,279,4016,1834,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,1923,25843,198,9222,25,307,10644,198,669,55530,25,220,760,5930,7903,16327,1330,14647,12079,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,46310,198,9222,25,5576,198,669,55530,25,220,84717,3200,12079,11,1204,1691,369,1990,46310,364,5361,310,5930,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,39850,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,16,16,22,13,15,15,2129,1979,816,3147,16,16,22,8934,198,32929,220,18,25,220,4754,4532,3147,18,24,13,15,15,2129,1979,816,3147,18,24,8934,198,32929,220,19,25,220,4754,4532,3147,22,23,13,15,15,2129,1979,816,3147,22,23,8934,198,4537,38945,198,198,27806,220,20,198,887,25,29541,198,9222,25,5284,198,669,55530,25,220,4199,14294,264,66929,369,411,4314,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,354,265,48641,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,54842,77699,14294,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,265,48641,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,1164,14294,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"missed_month","type":"choice","question_span":[236,243],"option_spans":[[254,270],[276,292],[298,314],[320,336],[342,352]],"option_ids":["2026-06","2026-07","2026-08","2026-09","none"],"logits":[-4.163268089294434,-4.125068187713623,0.7249463796615601,-4.349254608154297,-4.244737148284912],"probs":[0.007325610611587763,0.007610861677676439,0.9722287058830261,0.006082342471927404,0.006752463523298502],"argmax_index":2,"argmax_id":"2026-08","top2_margin":0.9646178442053497,"near_tie":false,"gold":"2026-08","gold_correct":true},{"question_id":"july_late","type":"noul","question_span":[377,391],"option_spans":[[402,420],[426,444]],"option_ids":["true","false"],"logits":[0.15768694877624512,-2.1239659786224365],"probs":[0.9073461294174194,0.09265390783548355],"argmax_index":0,"argmax_id":"true","top2_margin":0.8146922215819359,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"double_payment","type":"noul","question_span":[467,475],"option_spans":[[486,504],[510,528]],"option_ids":["true","false"],"logits":[-0.2602682113647461,-0.9631465077400208],"probs":[0.668825626373291,0.33117440342903137],"argmax_index":0,"argmax_id":"true","top2_margin":0.33765122294425964,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"owed","type":"choice","question_span":[549,563],"option_spans":[[574,584],[590,607],[613,628],[634,649]],"option_ids":["0","117","39","78"],"logits":[-1.9750995635986328,-1.5075019598007202,-0.8107072710990906,-0.44727635383605957],"probs":[0.09607750177383423,0.15335458517074585,0.30782991647720337,0.4427380859851837],"argmax_index":3,"argmax_id":"78","top2_margin":0.13490816950798035,"near_tie":false,"gold":"0","gold_correct":false},{"question_id":"reliability","type":"score","question_span":[670,678],"option_spans":[[689,701],[707,719],[725,736],[742,753]],"option_ids":["0","1","2","3"],"logits":[-1.0764234066009521,0.023096174001693726,-0.6772911548614502,-1.8600963354110718],"probs":[0.1680677980184555,0.5046610832214355,0.25051021575927734,0.07676083594560623],"argmax_index":1,"argmax_id":"1","top2_margin":0.2541508674621582,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"missed_month":{"type":"choice","choice":"2026-08","confidence":0.9722,"probabilities":{"2026-06":0.0073,"2026-07":0.0076,"2026-08":0.9722,"2026-09":0.0061,"none":0.0068}},"july_late":{"type":"noul","noul":0.9073},"double_payment":{"type":"noul","noul":0.6688},"owed":{"type":"choice","choice":"78","confidence":0.4427,"probabilities":{"0":0.0961,"39":0.3078,"78":0.4427,"117":0.1534}},"reliability":{"type":"score","score":1.236,"confidence":0.5047,"legend":{"0":"unreliable","1":"somewhat reliable","2":"reliable","3":"very reliable"},"probabilities":{"0":0.1681,"1":0.5047,"2":0.2505,"3":0.0768}}},"usage":{"input_tokens":775,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_j06","arm":"text","set":"heldout","source":"heldout_json","tokens":733,"n_image_tokens":0,"state_tokens":104,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,658,3147,217796,43092,1323,6250,2129,6308,21624,3771,57964,763,1802,1288,17207,8973,89801,8631,16,17,15,11,16,19,15,11,16,21,15,11,16,22,20,27573,36923,21624,329,40963,763,3721,1288,88652,763,3721,1288,69329,26372,763,1802,49471,45256,21624,8297,4747,6018,763,3721,1288,860,13393,763,1802,1288,18781,62143,3147,63644,56658,8775,3147,8678,2129,24826,3147,316,64959,2129,9570,648,1101,21624,15401,763,1802,1288,28068,85627,763,16,1288,11197,3147,275,29958,1288,4225,3147,20,13,17,13,15,8934,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,55671,6018,7462,198,9222,25,307,10644,198,669,55530,25,220,760,1151,321,809,3387,314,10964,513,7920,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,5832,4165,643,1772,198,9222,25,5576,198,669,55530,25,220,14749,628,1436,279,1156,579,10964,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,39351,2129,1979,816,3147,88787,8934,198,32929,220,17,25,220,4754,4532,3147,12123,382,1132,2129,1979,816,3147,63644,8934,198,32929,220,18,25,220,4754,4532,3147,7081,279,1156,2129,1979,816,3147,77,40934,8934,198,4537,38945,198,198,27806,220,18,198,887,25,8233,198,9222,25,307,10644,198,669,55530,25,220,65284,21292,513,28898,383,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,65284,21292,513,974,466,524,716,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,9842,18857,5262,198,9222,25,5576,198,669,55530,25,220,4199,1599,4593,41606,9842,22005,513,716,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,18,2129,1979,816,3147,18,8934,198,32929,220,17,25,220,4754,4532,3147,19,2129,1979,816,3147,19,8934,198,32929,220,18,25,220,4754,4532,3147,20,2129,1979,816,3147,20,8934,198,32929,220,19,25,220,4754,4532,3147,21,2129,1979,816,3147,21,8934,198,4537,38945,198,198,27806,220,20,198,887,25,4821,697,24186,198,9222,25,307,10644,198,669,55530,25,220,760,20699,54042,958,264,2086,4821,1785,494,279,799,6625,383,4034,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,21,198,887,25,11992,8016,198,9222,25,5284,198,669,55530,25,220,4199,11992,63355,513,1439,4842,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1164,1724,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,2428,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,895,330,1181,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,1164,44254,2129,1979,816,3147,19,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"hides_endpoints","type":"noul","question_span":[167,177],"option_spans":[[188,206],[212,230]],"option_ids":["true","false"],"logits":[-2.2539987564086914,1.8176424503326416],"probs":[0.016763577237725258,0.9832363724708557],"argmax_index":1,"argmax_id":"false","top2_margin":0.9664727952331305,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"route_audience","type":"choice","question_span":[254,262],"option_spans":[[273,283],[289,301],[307,320]],"option_ids":["everyone","followers","nobody"],"logits":[-3.1502130031585693,1.3623504638671875,-3.848297119140625],"probs":[0.010792987421154976,0.9838370680809021,0.005369918420910835],"argmax_index":1,"argmax_id":"followers","top2_margin":0.9730440806597471,"near_tie":false,"gold":"followers","gold_correct":true},{"question_id":"marketing","type":"noul","question_span":[342,348],"option_spans":[[359,377],[383,399]],"option_ids":["true","false"],"logits":[-3.1939778327941895,2.208608627319336],"probs":[0.0044847107492387295,0.9955152869224548],"argmax_index":1,"argmax_id":"false","top2_margin":0.9910305761732161,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"zone_boundaries","type":"choice","question_span":[422,431],"option_spans":[[442,452],[458,468],[474,484],[490,500]],"option_ids":["3","4","5","6"],"logits":[0.4011649191379547,-0.5004653930664062,-2.0633556842803955,-2.366238594055176],"probs":[0.6435909271240234,0.26123830676078796,0.0547371506690979,0.04043354466557503],"argmax_index":0,"argmax_id":"3","top2_margin":0.3823526203632355,"near_tie":false,"gold":"4","gold_correct":false},{"question_id":"unit_mismatch","type":"noul","question_span":[524,539],"option_spans":[[550,568],[574,592]],"option_ids":["true","false"],"logits":[-0.922295868396759,-0.10082033276557922],"probs":[0.30545052886009216,0.6945494413375854],"argmax_index":1,"argmax_id":"false","top2_margin":0.3890989124774933,"near_tie":false,"gold":"true","gold_correct":false},{"question_id":"privacy_level","type":"score","question_span":[614,621],"option_spans":[[632,643],[649,659],[665,676],[682,694],[700,711]],"option_ids":["0","1","2","3","4"],"logits":[0.05305349826812744,1.2046281099319458,0.9373055696487427,-1.732787847518921,-3.257082223892212],"probs":[0.14730772376060486,0.465959370136261,0.3566575050354004,0.024697015061974525,0.00537838414311409],"argmax_index":1,"argmax_id":"1","top2_margin":0.1093018651008606,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"hides_endpoints":{"type":"noul","noul":0.0168},"route_audience":{"type":"choice","choice":"followers","confidence":0.9838,"probabilities":{"everyone":0.0108,"followers":0.9838,"nobody":0.0054}},"marketing":{"type":"noul","noul":0.0045},"zone_boundaries":{"type":"choice","choice":"3","confidence":0.6436,"probabilities":{"3":0.6436,"4":0.2612,"5":0.0547,"6":0.0404}},"unit_mismatch":{"type":"noul","noul":0.3055},"privacy_level":{"type":"score","score":1.2749,"confidence":0.466,"legend":{"0":"very open","1":"open","2":"moderate","3":"cautious","4":"very cautious"},"probabilities":{"0":0.1473,"1":0.466,"2":0.3567,"3":0.0247,"4":0.0054}}},"usage":{"input_tokens":733,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_j07","arm":"text","set":"heldout","source":"heldout_json","tokens":826,"n_image_tokens":0,"state_tokens":231,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,58,4754,6766,2059,763,2827,1288,306,3147,18232,12,16,16,15,16,2129,54243,3147,17,15,17,21,12,15,24,12,17,24,2129,22845,3147,11553,2129,2935,3147,53,390,273,87920,8997,1543,17,2129,2738,3147,2428,2129,11170,3147,2665,7361,1238,28708,2560,35303,6766,2059,3147,95218,1264,72,8963,167223,2129,306,3147,18232,12,16,16,15,17,2129,54243,3147,17,15,17,21,12,15,24,12,18,15,2129,22845,3147,9997,2129,2935,3147,53,390,273,87920,10967,1406,2129,2738,3147,47869,4322,27898,2129,11170,3147,13788,3000,1447,15049,18016,35303,6766,2059,3147,95218,1264,72,8963,167223,2129,306,3147,18232,12,16,16,15,18,2129,54243,3147,17,15,17,21,12,16,15,12,15,16,2129,22845,3147,11553,2129,2935,3147,53,390,273,87920,8997,1543,17,2129,2738,3147,2428,2129,11170,3147,37,51471,2560,15239,506,220,23,15,4,35303,6766,2059,3147,239633,79766,26342,16231,2129,306,3147,18232,12,16,16,15,19,2129,54243,3147,17,15,17,21,12,16,15,12,15,17,2129,22845,3147,25252,2129,2935,3147,53,390,273,87920,8997,1543,16,2129,2738,3147,32958,2129,11170,3147,8128,278,11241,9412,11693,8934,60,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,632,37729,198,9222,25,5576,198,669,55530,25,220,22365,11390,682,18022,12244,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,18232,12,16,16,15,16,2129,1979,816,3147,18232,12,16,16,15,16,8934,198,32929,220,17,25,220,4754,4532,3147,18232,12,16,16,15,17,2129,1979,816,3147,18232,12,16,16,15,17,8934,198,32929,220,18,25,220,4754,4532,3147,18232,12,16,16,15,18,2129,1979,816,3147,18232,12,16,16,15,18,8934,198,32929,220,19,25,220,4754,4532,3147,18232,12,16,16,15,19,2129,1979,816,3147,18232,12,16,16,15,19,8934,198,32929,220,20,25,220,4754,4532,3147,28068,11390,369,12244,2129,1979,816,3147,6487,8934,198,4537,38945,198,198,27806,220,17,198,887,25,1724,21976,198,9222,25,5576,198,669,55530,25,220,4199,1599,13990,513,1724,440,1496,10304,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,15,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,16,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,17,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,18,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,19,2129,1979,816,3147,19,8934,198,4537,38945,198,198,27806,220,18,198,887,25,5904,19259,198,9222,25,5576,198,669,55530,25,220,3710,635,279,1496,2195,8508,13990,599,303,4047,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,79177,2129,1979,816,3147,37355,8934,198,32929,220,17,25,220,4754,4532,3147,32,28708,2560,2129,1979,816,3147,8537,1606,8934,198,32929,220,18,25,220,4754,4532,3147,13788,3387,2129,1979,816,3147,22772,8934,198,32929,220,19,25,220,4754,4532,3147,8128,278,11241,2129,1979,816,3147,3637,278,8934,198,4537,38945,198,198,27806,220,19,198,887,25,660,43215,198,9222,25,307,10644,198,669,55530,25,220,2324,314,279,13990,513,7629,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,20,198,887,25,21192,30258,1653,198,9222,25,5284,198,669,55530,25,220,12525,264,28708,4722,381,4661,264,21192,10142,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,53164,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,13873,2129,1979,816,3147,16,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"unassigned","type":"choice","question_span":[292,298],"option_spans":[[309,329],[335,355],[361,381],[387,407],[413,426]],"option_ids":["VR-1101","VR-1102","VR-1103","VR-1104","none"],"logits":[0.7238964438438416,-3.5315823554992676,-3.544370651245117,-3.6604292392730713,-3.3277406692504883],"probs":[0.9451282620429993,0.013407869264483452,0.013237496837973595,0.011786974966526031,0.016439441591501236],"argmax_index":0,"argmax_id":"VR-1101","top2_margin":0.928688820451498,"near_tie":false,"gold":"VR-1101","gold_correct":true},{"question_id":"open_high","type":"choice","question_span":[448,457],"option_spans":[[468,478],[484,494],[500,510],[516,526],[532,542]],"option_ids":["0","1","2","3","4"],"logits":[-4.361968517303467,-3.611048698425293,0.4668198823928833,-4.307477951049805,-4.849471569061279],"probs":[0.007701286114752293,0.016318626701831818,0.9631177186965942,0.008132577873766422,0.004729806445538998],"argmax_index":2,"argmax_id":"2","top2_margin":0.9467990919947624,"near_tie":false,"gold":"2","gold_correct":true},{"question_id":"shared_theme","type":"choice","question_span":[564,575],"option_spans":[[586,596],[602,615],[621,632],[638,651]],"option_ids":["billing","firmware","mesh","parental"],"logits":[-5.373301029205322,1.174947738647461,-5.019377708435059,-5.299900054931641],"probs":[0.0014254730194807053,0.9950096607208252,0.0020307942759245634,0.0015340398531407118],"argmax_index":1,"argmax_id":"firmware","top2_margin":0.9929788664449006,"near_tie":false,"gold":"firmware","gold_correct":true},{"question_id":"all_closed","type":"noul","question_span":[674,681],"option_spans":[[692,710],[716,734]],"option_ids":["true","false"],"logits":[-3.445786714553833,1.8452094793319702],"probs":[0.005011498928070068,0.9949885010719299],"argmax_index":1,"argmax_id":"false","top2_margin":0.9899770021438599,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"wider_incident","type":"score","question_span":[757,767],"option_spans":[[778,788],[794,804]],"option_ids":["0","1"],"logits":[-0.7826851010322571,-0.5202679634094238],"probs":[0.4347696006298065,0.5652303695678711],"argmax_index":1,"argmax_id":"1","top2_margin":0.13046076893806458,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"unassigned":{"type":"choice","choice":"VR-1101","confidence":0.9451,"probabilities":{"VR-1101":0.9451,"VR-1102":0.0134,"VR-1103":0.0132,"VR-1104":0.0118,"none":0.0164}},"open_high":{"type":"choice","choice":"2","confidence":0.9631,"probabilities":{"0":0.0077,"1":0.0163,"2":0.9631,"3":0.0081,"4":0.0047}},"shared_theme":{"type":"choice","choice":"firmware","confidence":0.995,"probabilities":{"firmware":0.995,"parental":0.0015,"mesh":0.002,"billing":0.0014}},"all_closed":{"type":"noul","noul":0.005},"wider_incident":{"type":"score","score":0.5652,"confidence":0.5652,"legend":{"0":"unlikely","1":"likely"},"probabilities":{"0":0.4348,"1":0.5652}}},"usage":{"input_tokens":826,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_j08","arm":"text","set":"heldout","source":"heldout_json","tokens":798,"n_image_tokens":0,"state_tokens":207,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,4754,13646,3147,33,12,17,21,16,16,2129,13968,3147,2114,269,9024,1519,39300,2129,4332,3147,1277,5531,35362,12,22,2129,39761,64331,29430,788,34850,763,23,13,15,18,1288,3993,763,16,17,15,13,17,1288,77,763,16,1288,87834,763,19,13,15,17,1288,2998,763,18,20,13,16,35607,29430,788,34850,763,23,13,15,22,1288,3993,763,16,16,24,13,22,1288,77,763,17,1288,87834,763,18,13,24,22,1288,2998,763,18,19,13,24,35607,29430,788,34850,763,23,13,15,16,1288,3993,763,16,17,15,13,19,1288,77,763,18,1288,87834,763,19,13,15,20,1288,2998,763,18,20,13,15,92,27573,9253,21624,29430,788,34850,21624,16197,946,763,23,13,15,1288,19951,763,15,13,15,20,49471,3993,21624,16197,946,763,16,17,15,13,15,1288,19951,763,15,13,20,49471,87834,21624,16197,946,763,19,13,15,1288,19951,763,15,13,16,49471,2998,21624,16197,946,763,18,20,13,15,1288,19951,763,15,13,17,3307,1288,3715,3147,3693,8934,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,660,15010,198,9222,25,307,10644,198,669,55530,25,220,11179,5887,369,2785,23343,383,1396,12502,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,20738,47948,198,9222,25,5576,198,669,55530,25,220,22365,12502,369,680,314,23343,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,29430,22315,2129,1979,816,3147,29430,788,34850,8934,198,32929,220,17,25,220,4754,4532,3147,3993,2129,1979,816,3147,3993,8934,198,32929,220,18,25,220,4754,4532,3147,6487,2129,1979,816,3147,6487,8934,198,32929,220,19,25,220,4754,4532,3147,87834,2129,1979,816,3147,87834,8934,198,32929,220,20,25,220,4754,4532,3147,2998,2129,1979,816,3147,2998,8934,198,4537,38945,198,198,27806,220,18,198,887,25,20738,16971,198,9222,25,5576,198,669,55530,25,220,22365,5887,14107,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,13218,220,16,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,13218,220,17,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,13218,220,18,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,2083,5887,2129,1979,816,3147,6487,8934,198,4537,38945,198,198,27806,220,19,198,887,25,43493,198,9222,25,5576,198,669,55530,25,220,3710,1220,3420,310,279,6937,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,21624,1265,3147,6084,279,6937,321,6420,777,5306,2129,9033,3147,588,5887,14107,539,264,2526,4592,56658,1979,816,3147,6084,8934,198,32929,220,17,25,220,4754,4532,21624,1265,3147,22281,279,6937,2129,9033,3147,28068,5887,15734,56658,1979,816,3147,22281,8934,198,32929,220,18,25,220,4754,4532,21624,1265,3147,2311,4461,279,6937,2129,9033,3147,3444,10156,3564,539,264,6669,4592,56658,1979,816,3147,2311,4461,8934,198,4537,38945,198,198,27806,220,20,198,887,25,21610,198,9222,25,5284,198,669,55530,25,220,4199,12506,1503,279,1817,1353,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,5200,269,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,76,832,946,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,986,18558,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,17981,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"all_pass","type":"noul","question_span":[269,278],"option_spans":[[289,307],[313,331]],"option_ids":["true","false"],"logits":[-0.0405958890914917,1.1167784929275513],"probs":[0.2391447126865387,0.7608553171157837],"argmax_index":1,"argmax_id":"false","top2_margin":0.521710604429245,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"failing_dimension","type":"choice","question_span":[353,360],"option_spans":[[371,384],[390,400],[406,416],[422,432],[438,448]],"option_ids":["hole_diameter","length","none","thickness","width"],"logits":[0.7131237387657166,-1.1258633136749268,-0.3782227635383606,-0.34384581446647644,-2.0007410049438477],"probs":[0.5239635109901428,0.08329886943101883,0.17592813074588776,0.1820811629295349,0.03472837805747986],"argmax_index":0,"argmax_id":"hole_diameter","top2_margin":0.3418823480606079,"near_tie":false,"gold":"hole_diameter","gold_correct":true},{"question_id":"failing_sample","type":"choice","question_span":[470,474],"option_spans":[[485,497],[503,515],[521,533],[539,550]],"option_ids":["1","2","3","none"],"logits":[0.9967367649078369,-0.08026993274688721,0.20481234788894653,0.885007917881012],"probs":[0.3720414638519287,0.12672236561775208,0.16852445900440216,0.33271172642707825],"argmax_index":0,"argmax_id":"1","top2_margin":0.039329737424850464,"near_tie":false,"gold":"2","gold_correct":false},{"question_id":"disposition","type":"choice","question_span":[571,578],"option_spans":[[589,617],[623,643],[649,675]],"option_ids":["hold","release","scrap"],"logits":[-0.6141442060470581,0.3575874865055084,-1.7945151329040527],"probs":[0.253184974193573,0.6690455079078674,0.07776950299739838],"argmax_index":1,"argmax_id":"release","top2_margin":0.41586053371429443,"near_tie":false,"gold":"hold","gold_correct":false},{"question_id":"capability","type":"score","question_span":[696,703],"option_spans":[[714,725],[731,743],[749,760],[766,776]],"option_ids":["0","1","2","3"],"logits":[-1.202749490737915,0.45577487349510193,0.3768841624259949,0.5574681758880615],"probs":[0.05910712108016014,0.31040436029434204,0.28685736656188965,0.34363123774528503],"argmax_index":3,"argmax_id":"3","top2_margin":0.03322687745094299,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"all_pass":{"type":"noul","noul":0.2391},"failing_dimension":{"type":"choice","choice":"hole_diameter","confidence":0.524,"probabilities":{"length":0.0833,"width":0.0347,"hole_diameter":0.524,"thickness":0.1821,"none":0.1759}},"failing_sample":{"type":"choice","choice":"1","confidence":0.372,"probabilities":{"1":0.372,"2":0.1267,"3":0.1685,"none":0.3327}},"disposition":{"type":"choice","choice":"release","confidence":0.669,"probabilities":{"release":0.669,"hold":0.2532,"scrap":0.0778}},"capability":{"type":"score","score":1.915,"confidence":0.3436,"legend":{"0":"poor","1":"marginal","2":"adequate","3":"good"},"probabilities":{"0":0.0591,"1":0.3104,"2":0.2869,"3":0.3436}}},"usage":{"input_tokens":798,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_l01","arm":"text","set":"heldout","source":"heldout_text_long","tokens":2850,"n_image_tokens":0,"state_tokens":1968,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,15238,2662,44,7142,3848,351,7615,51967,318,3189,639,539,63227,7055,1519,67942,8,198,37947,1653,24024,25,9039,91083,303,47916,23459,220,18,11,3603,314,6394,220,16,16,310,7483,220,16,17,5930,220,17,15,17,21,198,87275,539,2285,11369,645,9395,347,11,4131,6436,11,494,279,4599,6044,1785,7250,11,279,6206,1433,11,279,795,1433,10322,321,18758,440,279,5395,383,13783,13,1942,2942,513,2136,13,271,8441,198,1106,14778,220,18,10220,19088,11228,506,264,716,1406,314,471,17,19,35087,34,13,11116,1496,12,33413,16136,369,716,506,471,16,23,35087,34,11,321,264,829,18212,2109,1414,1876,5601,680,506,471,17,16,35087,34,13,561,23459,369,62139,539,1330,23835,1052,25,56885,220,16,22651,279,2706,11,321,56885,220,17,369,279,69301,4821,11,864,8211,539,4924,413,56885,220,16,17397,13,2332,31266,383,220,16,16,5930,279,23459,5467,220,19,16,17,56815,82,25,220,16,21,23,314,9561,12281,364,1330,10489,6112,11,220,16,24,21,314,19088,23133,318,15693,279,220,17,17,56815,82,314,44499,12238,506,220,17,17,25,16,20,8,321,220,19,23,314,19088,15530,29363,364,264,62944,8516,13,30347,220,20,11,1727,5817,11,8213,506,471,17,20,35087,34,321,995,220,16,24,15,4147,56815,12258,421,3603,13,198,1597,220,16,20,25,15,15,383,220,16,16,5930,56885,220,17,557,23443,364,264,12539,17504,13329,13,561,13017,28994,3481,310,6052,279,944,539,220,16,15,25,15,15,279,1727,6353,13,2233,12955,26054,557,20789,364,279,4031,11,1521,279,2531,995,978,31877,430,3238,5048,13,271,57625,198,17,16,25,19,15,561,3603,6206,8211,13,25662,2895,426,390,276,75315,11590,279,1375,1908,10517,11,864,7189,328,37088,269,220,17,680,314,2393,2980,883,220,16,15,25,15,15,16089,3158,30347,220,18,15339,471,17,19,13,18,35087,34,13,198,17,17,25,16,20,561,1483,324,8208,314,279,1834,632,31600,220,17,17,56815,82,314,19088,44499,1083,23459,220,18,13,561,8019,5817,12965,1724,364,220,16,16,4276,26,279,23459,3980,977,310,471,17,17,13,24,35087,34,321,369,1142,506,1141,716,1406,2785,220,17,15,4276,11,864,369,4472,1238,264,9403,13,198,17,18,25,20,23,12458,5497,63889,452,41101,2305,7642,43117,279,13618,4088,20372,13,1942,13635,513,9035,11,23459,220,18,15339,471,17,19,13,16,35087,34,321,1017,369,874,9561,1870,5072,383,279,57054,824,68366,13,198,15,15,25,18,21,1158,54331,220,16,21767,13,561,4599,6044,1785,17872,328,34,16,8604,3522,269,14109,1,321,279,56885,17397,13,8938,56885,220,17,369,23443,11,874,69301,4821,4829,888,13,198,15,15,25,18,22,357,328,37088,269,2327,6708,1,9663,7701,383,279,4034,303,279,2444,2956,13,561,2444,2956,369,524,5395,290,506,3603,321,279,9663,369,524,3106,12025,745,11,748,18022,15175,424,13,198,15,16,25,17,15,30347,220,18,23716,471,17,16,13,15,35087,34,13,561,1785,20653,279,829,18212,2109,1414,539,4924,310,279,383,56905,28417,362,1324,9476,303,1141,3522,1103,13,198,15,16,25,17,16,561,1414,369,12238,310,264,6170,4392,421,557,21893,1142,303,5589,539,449,22839,852,682,2394,2047,279,2724,13,561,3522,1103,303,279,1785,995,2496,978,5860,11,748,18022,15339,279,1876,13,198,15,17,25,19,22,30347,220,18,15734,279,16136,3798,314,471,16,23,35087,34,13,561,274,44791,10054,303,279,6631,2982,321,264,2018,1414,5601,310,279,1788,2235,1324,13,198,15,17,25,20,17,63889,452,41101,2305,7642,50309,279,274,44791,1345,20483,449,1906,303,23459,220,16,13,1216,11988,279,16136,6729,321,17094,426,390,276,75315,11,852,369,303,279,4998,47270,13,198,15,17,25,20,23,426,390,276,75315,40988,421,56885,220,16,682,10184,321,15818,264,7345,494,279,6729,13,561,56885,8213,364,883,220,19,15,6283,321,21767,1495,440,279,1788,14109,13,198,15,18,25,15,20,426,390,276,75315,6526,279,28417,362,28994,579,220,17,19,20580,1500,13,561,1562,6795,2640,279,383,56905,22839,11,61148,339,1587,4717,268,11,628,381,383,2660,303,883,449,6258,13,198,15,18,25,16,15,1942,20483,494,23459,220,18,369,10184,321,1141,13635,513,8341,9035,310,3222,303,279,8981,13,30347,220,18,15339,471,16,21,13,23,35087,34,13,198,15,18,25,17,20,4486,264,4392,1562,440,2285,11369,645,9395,347,11,426,390,276,75315,26731,310,3166,279,9561,12281,1118,11,1521,424,369,279,1918,1379,15739,310,23120,11,1083,279,1845,12258,303,23459,220,20,13,561,23133,321,279,29363,4565,303,23459,220,18,364,1381,13,198,15,18,25,18,16,8771,5372,9885,1875,11450,1151,6992,279,9561,12281,56815,82,13,8618,5817,8306,369,8341,430,2716,430,3102,11,321,264,12931,44242,369,17750,3808,279,67011,314,23459,220,18,13,198,15,19,25,15,24,61148,339,1587,4717,268,28712,383,2660,321,8211,14109,2149,3828,383,56885,220,16,13,198,15,19,25,18,15,220,16,19,15,314,279,220,16,21,23,9561,12281,56815,82,599,978,7481,1083,23459,220,20,13,30347,220,18,15339,471,16,19,13,24,35087,34,13,198,15,19,25,19,19,61148,339,1587,4717,268,13325,421,279,1822,3522,269,314,56885,220,16,682,47680,680,13,561,38348,995,68870,639,11,321,279,17954,1420,8545,9558,421,1902,599,5617,685,888,3679,1578,13,198,15,19,25,20,17,561,9376,220,17,23,9561,12281,56815,82,513,1381,303,23459,220,20,13,387,7229,10480,2842,11,748,279,13635,314,23459,220,18,4565,9035,2261,279,12368,13,198,15,20,25,16,20,30347,220,18,23716,1141,8166,9039,314,279,3603,11,471,16,17,13,19,35087,34,11,12076,539,2107,279,6784,21059,321,279,9209,795,5741,13,198,15,20,25,17,15,61148,339,1587,4717,268,17759,264,23116,3522,269,494,279,5416,303,1024,5075,13,198,15,20,25,17,18,1158,54331,220,16,16526,82,321,8213,13870,13,30347,220,18,8211,310,6792,13,198,15,20,25,19,15,426,390,276,75315,16561,279,13017,28994,310,3502,264,51715,3988,310,2113,56885,220,17,1142,1083,2393,13,357,51715,369,18378,364,220,15,23,25,15,15,13,198,15,21,25,18,15,30347,220,18,15339,471,16,21,13,15,35087,34,321,369,15218,539,16757,220,16,13,20,35087,34,1396,4139,6258,13,198,15,22,25,16,17,30347,220,18,16563,3559,279,16136,3798,314,471,16,23,35087,34,1495,11,321,279,16136,54703,539,4924,13,198,15,23,25,15,20,561,13017,51715,28712,321,31941,25504,279,491,17504,303,56885,220,17,13,198,15,24,25,19,15,1158,54331,220,17,369,1142,303,2393,430,279,69301,4821,11,6575,1056,279,3889,3019,13,198,16,15,25,16,20,30347,220,18,369,1142,506,1141,716,1406,314,471,17,19,35087,34,13,271,4662,14996,318,16,17,5930,11,220,15,24,25,15,15,310,220,16,18,25,15,15,8,198,1378,11369,645,9395,347,3738,6007,19274,494,5887,7171,2305,383,56815,82,303,1396,1918,1809,321,7463,1070,440,1754,5813,579,24332,13,198,49604,12281,25,279,220,16,19,15,56815,82,7481,1518,220,15,19,25,18,15,2496,15537,3294,471,16,21,35087,34,506,279,6007,13,11396,10489,6112,4054,2716,35937,310,471,16,20,35087,34,11,748,1439,56815,82,998,5698,13,561,1483,220,17,23,56815,82,7481,995,6007,19274,314,471,16,18,35087,34,13,19332,12281,421,682,78911,421,2944,31130,1141,10124,948,424,1985,740,30750,11,748,1439,220,17,23,56815,82,998,17030,321,3106,364,18676,13,198,88173,23133,25,279,6007,19274,19013,1881,471,16,20,35087,34,321,471,16,19,35087,34,13,561,18254,579,24332,5954,2716,35937,310,471,16,17,35087,34,364,23133,421,668,381,28154,1518,11834,11,748,660,220,16,24,21,56815,82,998,5698,13,198,33,851,29363,25,6007,19274,8379,471,16,18,13,20,35087,34,13,561,62944,8516,4439,364,279,220,19,23,56815,82,310,381,5467,2980,1141,25744,682,12152,3315,279,39142,369,1990,4393,13,561,56815,82,998,58205,430,383,3222,321,7481,310,279,68008,22002,314,23459,220,20,13,271,31995,520,5975,7003,383,220,16,19,5930,198,16,13,561,383,56905,3522,1103,303,279,16136,1785,369,310,381,9770,1396,6796,539,279,6206,2895,11,321,1495,14910,449,22839,27743,466,10583,13,198,17,13,357,328,37088,269,2327,6708,1,9663,668,1048,381,3106,539,1414,310,279,6206,2895,383,13783,11,524,1132,6625,383,279,2444,46150,4034,13,198,18,13,357,69301,56885,1189,381,4284,680,314,2393,364,12539,944,1132,413,264,20789,6170,26054,4821,682,978,8173,310,279,23459,1118,13,198,19,13,561,3522,1052,383,660,23835,1052,303,279,4599,668,381,12215,1381,11,321,1238,421,1396,4097,1578,13,198,20,13,12458,19024,668,381,15700,310,42366,264,56885,14109,383,279,6729,321,310,1562,279,28994,5774,13,271,14508,314,279,10142,198,17,23,56815,82,314,9561,12281,14012,318,8823,220,16,19,11,18,15,15,303,1918,869,681,279,28994,579,3603,1562,9502,10737,11,220,21,13,20,3979,314,37221,364,279,6992,13234,11,321,279,2695,314,279,62944,579,25744,6813,413,279,29363,369,17030,13,271,18656,198,760,3522,269,7652,4924,557,18541,9558,13,561,5383,3571,494,2250,31092,421,30265,685,383,799,3603,25,279,69301,56885,557,680,314,2393,440,874,3311,11,279,1118,9663,557,1132,6625,383,449,4147,4034,11,321,279,16136,21306,3804,310,264,4392,421,874,799,11354,13,4254,1056,1330,3979,5642,1881,279,8151,506,220,15,15,25,18,21,321,279,4299,4184,383,2660,6661,883,424,506,220,15,17,25,20,17,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,3578,648,3344,198,9222,25,5576,198,669,55530,25,220,3710,3727,279,10142,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,2014,19132,3522,269,383,264,56885,4490,2129,1979,816,3147,747,5444,8934,198,32929,220,17,25,220,4754,4532,3147,32,8019,5817,557,2047,1724,2129,1979,816,3147,10470,8934,198,32929,220,18,25,220,4754,4532,3147,760,65835,2279,4490,2129,1979,816,3147,12793,39952,8934,198,32929,220,19,25,220,4754,4532,3147,674,1569,7045,505,32728,680,2129,1979,816,3147,1062,61787,505,11414,570,8934,198,32929,220,20,25,220,4754,4532,3147,32,9039,11661,6348,867,37780,2129,1979,816,3147,28549,8934,198,4537,38945,198,198,27806,220,17,198,887,25,15686,51729,198,9222,25,5576,198,669,55530,25,220,3710,557,279,8166,3594,9039,12076,303,23459,220,18,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,12,16,17,13,19,35087,34,2129,1979,816,3147,12,16,17,13,19,8934,198,32929,220,17,25,220,4754,4532,3147,12,16,19,13,24,35087,34,2129,1979,816,3147,12,16,19,13,24,8934,198,32929,220,18,25,220,4754,4532,3147,12,16,23,13,15,35087,34,2129,1979,816,3147,12,16,23,13,15,8934,198,32929,220,19,25,220,4754,4532,3147,12,24,13,21,35087,34,2129,1979,816,3147,12,24,13,21,8934,198,4537,38945,198,198,27806,220,18,198,887,25,1118,4185,1243,3726,198,9222,25,307,10644,198,669,55530,25,220,760,1118,16465,16136,1414,8379,4184,852,1362,1121,383,424,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,29363,5878,1952,198,9222,25,5576,198,669,55530,25,220,3710,6716,310,279,15530,29363,56815,82,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,30326,364,18676,2129,1979,816,3147,15062,290,8934,198,32929,220,17,25,220,4754,4532,3147,39,758,2980,25744,6813,513,2725,2129,1979,816,3147,26713,8934,198,32929,220,18,25,220,4754,4532,3147,5247,303,23459,220,18,1973,866,1716,2129,1979,816,3147,2282,1201,4002,14778,62,18,8934,198,32929,220,19,25,220,4754,4532,3147,38707,310,279,5813,2129,1979,816,3147,74308,8934,198,4537,38945,198,198,27806,220,20,198,887,25,69301,34283,198,9222,25,307,10644,198,669,55530,25,220,32,69301,56885,557,5354,310,1831,888,948,56885,220,16,10184,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,21,198,887,25,3294,14335,198,9222,25,5576,198,669,55530,25,220,2381,1204,1248,557,23459,220,18,43442,1056,1141,16136,3798,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,3833,310,2250,3979,2129,1979,816,3147,16,2270,62,18,71,8934,198,32929,220,17,25,220,4754,4532,3147,19041,310,4590,3979,2129,1979,816,3147,18,2270,62,21,71,8934,198,32929,220,18,25,220,4754,4532,3147,7418,1056,4590,3979,2129,1979,816,3147,1908,62,21,71,8934,198,32929,220,19,25,220,4754,4532,3147,26561,1056,449,6258,2129,1979,816,3147,7744,62,16,71,8934,198,4537,38945,198,198,27806,220,22,198,887,25,71853,198,9222,25,5284,198,669,55530,25,220,4199,5812,557,279,10142,364,279,2478,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,44149,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,90112,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,40167,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"root_cause","type":"choice","question_span":[2030,2035],"option_spans":[[2046,2064],[2070,2085],[2091,2105],[2111,2130],[2136,2151]],"option_ids":["contactor","door","power_cut","refrigerant_leak","sensor"],"logits":[0.3581564128398895,-4.1279754638671875,-5.09027099609375,-5.076391696929932,-5.039133071899414],"probs":[0.97612464427948,0.010995198041200638,0.004200329538434744,0.004259034059941769,0.004420713521540165],"argmax_index":0,"argmax_id":"contactor","top2_margin":0.9651294462382793,"near_tie":false,"gold":"contactor","gold_correct":true},{"question_id":"peak_temperature","type":"choice","question_span":[2173,2185],"option_spans":[[2196,2216],[2222,2242],[2248,2268],[2274,2292]],"option_ids":["-12.4","-14.9","-18.0","-9.6"],"logits":[1.4114060401916504,-3.455031156539917,-4.281942367553711,-4.482337474822998],"probs":[0.986362874507904,0.007595738861709833,0.003322361968457699,0.002719045151025057],"argmax_index":0,"argmax_id":"-12.4","top2_margin":0.9787671356461942,"near_tie":false,"gold":"-12.4","gold_correct":true},{"question_id":"first_text_reached","type":"noul","question_span":[2317,2330],"option_spans":[[2341,2359],[2365,2383]],"option_ids":["true","false"],"logits":[-2.22367000579834,1.0384905338287354],"probs":[0.0368923656642437,0.9631076455116272],"argmax_index":1,"argmax_id":"false","top2_margin":0.9262152798473835,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"dough_outcome","type":"choice","question_span":[2406,2415],"option_spans":[[2426,2439],[2445,2461],[2467,2489],[2495,2508]],"option_ids":["destroyed","held","left_in_chamber_3","released"],"logits":[-4.553518772125244,0.9312105178833008,-4.045116424560547,-4.699711322784424],"probs":[0.004089806694537401,0.9855768084526062,0.006799846887588501,0.0035335577558726072],"argmax_index":1,"argmax_id":"held","top2_margin":0.9787769615650177,"near_tie":false,"gold":"held","gold_correct":true},{"question_id":"standby_ready","type":"noul","question_span":[2531,2545],"option_spans":[[2556,2574],[2580,2598]],"option_ids":["true","false"],"logits":[-3.1999361515045166,0.9465938806533813],"probs":[0.015572862699627876,0.9844271540641785],"argmax_index":1,"argmax_id":"false","top2_margin":0.9688542913645506,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"above_limit","type":"choice","question_span":[2620,2633],"option_spans":[[2644,2661],[2667,2684],[2690,2706],[2712,2728]],"option_ids":["1_to_3h","3_to_6h","over_6h","under_1h"],"logits":[-2.1164844036102295,-1.126630187034607,-1.521181344985962,-3.570629119873047],"probs":[0.17427606880664825,0.46894940733909607,0.31606370210647583,0.04071091115474701],"argmax_index":1,"argmax_id":"3_to_6h","top2_margin":0.15288570523262024,"near_tie":false,"gold":"3_to_6h","gold_correct":true},{"question_id":"seriousness","type":"score","question_span":[2749,2758],"option_spans":[[2769,2779],[2785,2796],[2802,2812],[2818,2828]],"option_ids":["0","1","2","3"],"logits":[-3.0846447944641113,-0.5650545954704285,-0.6465975046157837,-2.6322181224823],"probs":[0.0378124974668026,0.46976369619369507,0.43297797441482544,0.059445880353450775],"argmax_index":1,"argmax_id":"1","top2_margin":0.03678572177886963,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"root_cause":{"type":"choice","choice":"contactor","confidence":0.9761,"probabilities":{"contactor":0.9761,"door":0.011,"power_cut":0.0042,"refrigerant_leak":0.0043,"sensor":0.0044}},"peak_temperature":{"type":"choice","choice":"-12.4","confidence":0.9864,"probabilities":{"-18.0":0.0033,"-14.9":0.0076,"-12.4":0.9864,"-9.6":0.0027}},"first_text_reached":{"type":"noul","noul":0.0369},"dough_outcome":{"type":"choice","choice":"held","confidence":0.9856,"probabilities":{"released":0.0035,"destroyed":0.0041,"held":0.9856,"left_in_chamber_3":0.0068}},"standby_ready":{"type":"noul","noul":0.0156},"above_limit":{"type":"choice","choice":"3_to_6h","confidence":0.4689,"probabilities":{"under_1h":0.0407,"1_to_3h":0.1743,"3_to_6h":0.4689,"over_6h":0.3161}},"seriousness":{"type":"score","score":1.5141,"confidence":0.4698,"legend":{"0":"minor","1":"moderate","2":"serious","3":"critical"},"probabilities":{"0":0.0378,"1":0.4698,"2":0.433,"3":0.0594}}},"usage":{"input_tokens":2850,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_l02","arm":"text","set":"heldout","source":"heldout_text_long","tokens":2731,"n_image_tokens":0,"state_tokens":1913,"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,19302,2913,49560,26545,6014,231249,8860,50,318,22759,25,82909,321,47451,220,16,310,220,16,20,8,271,5619,22893,82,198,40571,24783,25,414,2060,22522,88441,18335,198,68834,25,68388,863,53452,26767,75500,18335,198,40089,4750,25,7703,220,19,11,95636,438,347,16706,5367,11,264,3074,32149,88,24279,314,883,220,21,19,15,9057,35058,440,264,18921,321,4590,12551,12258,11,6625,86483,2438,383,279,12037,3019,198,16741,25,5600,318,16,15,8,1578,5732,383,220,16,6492,220,17,15,17,21,318,1719,328,16741,4980,2544,871,198,6140,7765,25,220,19,23,11,15,15,15,264,1007,11,13366,314,2393,6545,11,7857,7765,321,866,3615,198,65457,12217,4031,25,279,1118,2250,318,18,8,3818,314,279,16999,198,65457,16051,25,264,2542,5951,310,4590,318,21,8,3818,314,279,2763,7765,198,3758,5310,958,25,279,2790,11,28776,11,5638,321,6080,314,40634,11228,11,440,44222,33360,18548,271,16,13,69628,362,198,16,13,16,733,411,24270,11,279,328,83618,20542,1,513,220,16,5869,11,220,16,5632,11,220,16,5589,321,220,16,6324,303,1754,1007,13,198,16,13,17,357,5604,310,279,11430,4750,5469,279,11098,24783,579,36416,321,66594,303,1070,11,694,524,279,71218,579,1765,6355,36416,13,198,16,13,18,10877,793,513,364,18554,1132,321,635,524,2222,1204,279,24270,369,1301,13,271,17,13,22994,198,17,13,16,561,11098,24783,14908,279,11430,4750,310,279,71218,364,279,16999,11,3658,440,279,1245,310,958,279,12039,18660,321,279,5904,24272,2982,15784,279,975,9854,4659,314,279,6355,5926,13,198,17,13,17,561,11098,24783,13205,279,1245,310,3599,279,11430,4750,11,1238,6866,506,3140,34227,67354,318,19,23,8,3979,6,5125,3525,303,449,12483,11,310,23331,1070,11,310,6564,680,29201,364,864,424,369,8220,11,321,310,1420,1070,310,3102,21927,466,11,2261,279,1483,4590,3818,314,279,16999,11,310,3102,491,38592,13,271,18,13,28766,198,18,13,16,561,71218,4723,2216,279,7765,539,2943,5951,44081,1320,303,11573,383,279,33252,20542,13,561,1118,7903,11,1391,14940,290,364,279,919,314,279,8079,424,14103,11,16563,4016,383,279,1834,1238,279,7765,12217,4031,10029,13,198,18,13,17,1368,866,7765,466,975,2542,369,777,1056,60163,318,16,19,8,2756,3200,11,279,71218,4723,2216,2647,383,424,506,2943,3161,318,19,11007,264,1007,3294,279,2255,4238,314,279,11098,24783,579,5883,11,494,279,4016,2321,2980,279,1834,424,369,6945,13,271,19,13,28766,9988,198,19,13,16,561,7765,4723,381,21250,383,279,17175,20921,314,279,16999,4980,2544,318,1719,328,18842,2544,1764,198,19,13,17,561,21250,7765,4723,381,279,4918,314,318,64,8,279,7765,43113,6849,1518,279,9988,2544,321,318,65,8,279,1724,2981,7765,314,279,11430,4750,506,279,9988,2544,11,430,7003,1881,279,9389,466,11,413,781,599,524,7003,2785,2250,3818,11,430,6426,539,449,9209,10258,269,13,198,19,13,18,28025,279,21250,7765,682,978,6426,11,279,71218,4723,708,383,12156,279,2235,7765,11,321,4723,2216,866,95621,2785,16570,18030,318,17,16,8,2756,1238,279,5307,13,271,20,13,5194,35576,321,21008,198,20,13,16,561,71218,4723,2216,264,6415,4191,314,279,11098,24783,579,6829,314,3221,1238,279,12039,18660,11,61859,11,17188,321,279,5904,24272,2982,13,1690,279,1118,2250,1578,314,279,16999,411,2393,6545,4723,524,381,777,1056,220,19,11,20,15,15,303,866,1007,13,198,20,13,17,561,11098,24783,4723,67176,279,4599,421,5435,279,11430,4750,2272,3807,11,13071,11,17198,11,4982,6867,321,279,975,13087,14832,11,364,279,2400,2695,314,53276,424,11,321,279,71218,4723,51548,310,279,11098,24783,279,919,314,279,14418,421,34439,310,279,11430,4750,318,1719,328,67583,7765,871,2785,16570,18030,2756,314,264,5158,7244,13,198,20,13,18,1368,449,50293,5048,3520,279,11430,4750,90248,364,958,11,874,7765,369,43113,2980,781,513,4786,364,958,1495,11,364,506,1379,2250,318,18,8,1578,13,271,21,13,3212,4568,198,21,13,16,561,11098,24783,369,8220,364,9976,279,14693,11,279,39144,11,279,4732,13876,321,279,23016,3899,314,279,11430,4750,303,1603,12368,13,198,21,13,17,561,71218,4723,2426,279,4613,314,279,11430,4750,11,2583,6220,25586,11,13635,11,10708,11,2682,6448,11,19132,42653,321,279,75959,421,8418,1132,279,11430,4750,11,303,1603,12368,321,43919,1906,11,321,4723,29180,5816,344,279,4613,303,279,1483,1007,314,279,16999,13,198,21,13,18,561,71218,1503,524,599,310,2113,279,11430,4750,1083,264,2577,1528,314,12368,1056,279,799,6625,303,279,66726,9412,314,2876,421,2107,9389,8239,383,279,16999,4980,2544,13,271,22,13,5272,198,22,13,16,561,71218,4723,958,279,11430,4750,1132,364,279,3495,5310,5272,13,198,22,13,17,561,71218,1189,4433,321,1542,8799,466,6605,69147,14609,4268,11,430,1248,430,424,318,64,8,90342,866,14014,421,279,3807,10872,466,279,2136,14403,7225,11,318,65,8,13205,449,685,4534,17864,3807,5048,14996,421,14103,279,14609,4268,11,321,318,66,8,682,1396,14609,77,59700,539,264,14554,22839,506,3140,2957,264,1007,13,198,22,13,18,561,71218,4723,524,2426,383,279,11430,4750,777,1274,95679,3558,1056,424,27517,3749,364,279,3495,5310,5272,11,321,4723,524,35992,279,975,9854,4659,440,281,18057,11,11639,466,46219,13,271,23,13,41434,778,321,56076,198,23,13,16,561,71218,4723,524,1236,866,23016,70118,310,279,11430,4750,321,4723,524,3799,1083,279,14693,466,279,4732,13876,13,198,23,13,17,561,71218,1189,1236,5138,57961,421,513,524,23016,11,1680,430,10371,685,466,4483,1441,45127,11,413,279,11098,24783,31816,303,4237,26,279,11098,24783,4723,524,24272,466,7383,1141,8855,1973,1603,2781,13,198,23,13,18,561,71218,1189,1420,1141,803,321,12070,383,279,203961,4331,314,279,11430,4750,11,303,264,1331,321,1647,421,279,11098,24783,682,11454,49340,13,271,24,13,33284,321,9170,1112,1235,198,24,13,16,561,71218,4723,524,9501,11,1172,1112,11,4191,466,919,440,18158,314,1132,264,919,314,279,11430,4750,13,198,24,13,17,561,71218,4723,524,1172,1112,279,4220,314,279,11430,4750,2892,13,198,24,13,18,561,71218,1189,9501,279,4220,314,279,11430,4750,413,279,11098,24783,6486,1141,5158,14020,49340,11,864,279,11098,24783,4723,524,24272,1973,1603,2781,11,321,413,279,71218,11590,449,67034,14988,8855,364,279,28201,314,279,491,24438,13,271,16,15,13,15164,53926,198,16,15,13,16,561,71218,1189,809,411,24270,383,279,17175,20921,314,279,16999,4980,2544,539,6866,279,11098,24783,506,3140,4590,318,21,8,3818,6,5125,303,4237,13,198,16,15,13,17,357,5125,1172,20984,220,16,15,13,16,1132,4829,2376,413,11,383,279,1387,2321,11,318,64,8,279,71218,682,6945,660,279,7765,421,682,20230,4016,1172,411,24270,11,321,318,65,8,279,71218,5888,1142,279,11430,4750,440,18022,68763,1070,321,6643,314,1141,11228,2047,4613,13,198,16,15,13,18,1061,20984,6486,279,11098,24783,874,1245,310,809,279,24270,3988,13,271,16,16,13,28766,46856,198,16,16,13,16,3095,411,24270,369,8239,279,71218,4723,2216,279,7765,16051,310,279,11098,24783,11,852,4723,3222,424,303,264,8387,2605,421,61607,2647,13,198,16,16,13,17,561,11098,24783,1189,1831,494,279,16051,866,7765,466,975,2542,421,682,978,52269,364,777,1056,60163,318,16,19,8,2756,11,321,279,71218,4723,1179,1844,279,16051,1142,685,2785,5600,318,16,15,8,3133,2756,13,198,16,16,13,18,561,16051,11,3658,440,279,2647,14857,383,424,11,4723,381,5743,2785,799,2182,1238,279,809,314,279,16999,11,2599,866,14287,279,11098,24783,682,9971,4284,494,424,13,271,16,17,13,1690,1795,9004,198,16,17,13,16,561,11098,24783,1189,311,46607,279,11430,4750,321,4300,411,24270,310,449,809,413,318,64,8,866,7765,369,1990,52269,16570,18030,318,17,16,8,2756,1238,424,16563,4016,11,3315,466,524,424,682,978,35102,28466,11,318,65,8,279,71218,13519,17758,866,975,27639,321,1503,524,2113,424,1245,2785,264,12891,854,1238,1602,3125,883,424,11,466,318,66,8,279,71218,5601,1083,14058,362,466,8305,13,271,16,18,13,65209,21953,468,198,16,18,13,16,1577,1248,430,279,71218,20589,279,7765,321,13205,310,411,24270,11,279,11098,24783,4723,524,35992,279,71218,579,958,314,279,11430,4750,13,271,16,19,13,3839,314,279,16999,198,16,19,13,16,2332,279,809,314,279,16999,279,71218,4723,1375,1142,279,11430,4750,303,279,12368,321,2876,421,411,24270,7225,11,1831,3023,1141,6355,36416,11,14609,4268,321,11590,11,321,12368,866,5383,8613,539,4483,1070,3023,13,198,16,19,13,17,1368,279,11098,24783,16561,303,4237,506,3140,4590,3818,1518,279,809,314,279,16999,11,279,71218,4723,1048,1831,680,866,5138,57961,424,682,1801,321,2113,1142,279,3889,6576,13,271,16,20,13,91966,321,7271,198,16,20,13,16,357,5125,1172,411,24270,369,2610,1132,413,424,369,5158,1441,321,2892,21893,888,303,1637,466,3106,539,12076,1675,310,279,9393,4998,314,279,4464,11955,424,13,198,16,20,13,17,1061,24270,369,25849,539,279,2253,314,279,3046,303,864,279,11430,4750,513,29099,11,321,1141,18259,7249,1189,9973,866,24240,883,424,13,271,47654,364,279,11098,24783,539,19338,64,3156,327,472,11,7300,198,47654,364,279,71218,539,1473,9227,76,25565,12615,11,7300,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,1387,50063,198,9222,25,5576,198,669,55530,25,220,4199,1691,5125,1902,279,24438,2873,310,958,279,1387,2904,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16,17,3818,2129,1979,816,3147,16,17,76,8934,198,32929,220,17,25,220,4754,4532,3147,18,3818,2129,1979,816,3147,18,76,8934,198,32929,220,18,25,220,4754,4532,3147,21,3818,2129,1979,816,3147,21,76,8934,198,32929,220,19,25,220,4754,4532,3147,24,3818,2129,1979,816,3147,24,76,8934,198,32929,220,20,25,220,4754,4532,3147,3733,369,874,1387,2904,2129,1979,816,3147,6487,8934,198,4537,38945,198,198,27806,220,17,198,887,25,3286,24633,198,9222,25,5576,198,669,55530,25,220,4199,628,279,7765,2222,506,279,3286,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,2064,4021,2222,2129,1979,816,3147,21343,8934,198,32929,220,17,25,220,4754,4532,3147,2064,10695,264,3240,1857,2129,1979,816,3147,1209,8934,198,32929,220,18,25,220,4754,4532,3147,2064,628,4565,279,1788,466,708,685,11,694,524,1441,2129,1979,816,3147,446,17861,8934,198,32929,220,19,25,220,4754,4532,3147,2064,628,708,685,466,1441,2129,1979,816,3147,446,8469,13596,8934,198,4537,38945,198,198,27806,220,18,198,887,25,1146,1112,10182,198,9222,25,307,10644,198,669,55530,25,220,760,24438,1189,1146,1112,919,314,279,32800,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,14693,198,9222,25,5576,198,669,55530,25,220,14749,1902,2426,279,14693,303,12368,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,70070,1132,2129,1979,816,3147,1283,7487,8934,198,32929,220,17,25,220,4754,4532,3147,760,38815,2129,1979,816,3147,1867,24783,8934,198,32929,220,18,25,220,4754,4532,3147,20003,11,303,5951,12863,2129,1979,816,3147,5910,8934,198,32929,220,19,25,220,4754,4532,3147,760,24438,2129,1979,816,3147,42466,8934,198,4537,38945,198,198,27806,220,20,198,887,25,16051,198,9222,25,5576,198,669,55530,25,220,4199,3349,369,279,7765,16051,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16,17,11,15,15,15,2129,1979,816,3147,16,17,15,15,15,8934,198,32929,220,17,25,220,4754,4532,3147,17,19,11,15,15,15,2129,1979,816,3147,17,19,15,15,15,8934,198,32929,220,18,25,220,4754,4532,3147,19,23,11,15,15,15,2129,1979,816,3147,19,23,15,15,15,8934,198,32929,220,19,25,220,4754,4532,3147,3733,369,874,16051,2129,1979,816,3147,6487,8934,198,4537,38945,198,198,27806,220,21,198,887,25,14609,4268,198,9222,25,307,10644,198,669,55530,25,220,760,24438,1189,1542,14609,4268,303,279,32800,413,424,19417,3529,4530,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,22,198,887,25,7915,198,9222,25,5284,198,669,55530,25,220,22365,3008,1503,279,24270,11984,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1719,38815,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,785,2405,3008,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,1719,24438,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"break_notice","type":"choice","question_span":[1974,1987],"option_spans":[[1998,2012],[2018,2030],[2036,2048],[2054,2066],[2072,2086]],"option_ids":["12m","3m","6m","9m","none"],"logits":[-2.8509163856506348,-4.342941761016846,0.8589291572570801,-3.595452308654785,-4.268115043640137],"probs":[0.023370077833533287,0.005256319418549538,0.9546091556549072,0.01109972782433033,0.005664722062647343],"argmax_index":2,"argmax_id":"6m","top2_margin":0.9312390778213739,"near_tie":false,"gold":"6m","gold_correct":true},{"question_id":"review_direction","type":"choice","question_span":[2108,2117],"option_spans":[[2128,2140],[2146,2160],[2166,2188],[2194,2211]],"option_ids":["fixed","index","up_only","up_or_down"],"logits":[-3.7120718955993652,-4.251005172729492,1.2186317443847656,-3.3934948444366455],"probs":[0.007070362567901611,0.004124638624489307,0.9790819883346558,0.00972294993698597],"argmax_index":2,"argmax_id":"up_only","top2_margin":0.9693590383976698,"near_tie":false,"gold":"up_only","gold_correct":true},{"question_id":"sublet_part","type":"noul","question_span":[2235,2245],"option_spans":[[2256,2274],[2280,2298]],"option_ids":["true","false"],"logits":[-3.0782456398010254,1.4342120885849],"probs":[0.010852394625544548,0.9891476631164551],"argmax_index":1,"argmax_id":"false","top2_margin":0.9782952684909105,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"roof","type":"choice","question_span":[2319,2327],"option_spans":[[2338,2351],[2357,2369],[2375,2389],[2395,2406]],"option_ids":["insurer","landlord","shared","tenant"],"logits":[-4.51728630065918,1.0228148698806763,-4.677211761474609,-4.687792778015137],"probs":[0.0038850160781294107,0.9895281195640564,0.003310839179903269,0.003275991650298238],"argmax_index":1,"argmax_id":"landlord","top2_margin":0.985643103485927,"near_tie":false,"gold":"landlord","gold_correct":true},{"question_id":"deposit","type":"choice","question_span":[2427,2434],"option_spans":[[2445,2464],[2470,2489],[2495,2514],[2520,2533]],"option_ids":["12000","24000","48000","none"],"logits":[-2.117283344268799,0.6829401254653931,-3.7780439853668213,-4.830696105957031],"probs":[0.05648241564631462,0.9290408492088318,0.010731340385973454,0.003745355876162648],"argmax_index":1,"argmax_id":"24000","top2_margin":0.8725584335625172,"near_tie":false,"gold":"24000","gold_correct":true},{"question_id":"kilns","type":"noul","question_span":[2556,2571],"option_spans":[[2582,2600],[2606,2624]],"option_ids":["true","false"],"logits":[0.41469866037368774,-3.406703472137451],"probs":[0.9785721302032471,0.02142786979675293],"argmax_index":0,"argmax_id":"true","top2_margin":0.9571442604064941,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"balance","type":"score","question_span":[2645,2652],"option_spans":[[2663,2674],[2680,2692],[2698,2709]],"option_ids":["0","1","2"],"logits":[-0.3689170777797699,-1.1923818588256836,-1.3956048488616943],"probs":[0.5564520955085754,0.2442314475774765,0.1993163675069809],"argmax_index":0,"argmax_id":"0","top2_margin":0.31222064793109894,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"break_notice":{"type":"choice","choice":"6m","confidence":0.9546,"probabilities":{"3m":0.0053,"6m":0.9546,"9m":0.0111,"12m":0.0234,"none":0.0057}},"review_direction":{"type":"choice","choice":"up_only","confidence":0.9791,"probabilities":{"up_only":0.9791,"up_or_down":0.0097,"index":0.0041,"fixed":0.0071}},"sublet_part":{"type":"noul","noul":0.0109},"roof":{"type":"choice","choice":"landlord","confidence":0.9895,"probabilities":{"landlord":0.9895,"tenant":0.0033,"shared":0.0033,"insurer":0.0039}},"deposit":{"type":"choice","choice":"24000","confidence":0.929,"probabilities":{"12000":0.0565,"24000":0.929,"48000":0.0107,"none":0.0037}},"kilns":{"type":"noul","noul":0.9786},"balance":{"type":"score","score":0.6429,"confidence":0.5565,"legend":{"0":"the landlord","1":"neither side","2":"the tenant"},"probabilities":{"0":0.5565,"1":0.2442,"2":0.1993}}},"usage":{"input_tokens":2731,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_i01","arm":"g256","set":"heldout","source":"heldout_image","tokens":529,"n_image_tokens":64,"state_tokens":18,"token_offset":36,"image_files":["ho_pie_720x480.png"],"image_size_in":[720,480],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"074c72a88d204b9bd27e90a9edd720d43079b8884e58714efe921653f49aa44f","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,32,13338,579,42800,4649,958,11,6507,539,7193,11,369,14336,430,264,4208,9207,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,8276,25640,198,9222,25,5576,198,669,55530,25,220,22365,12106,369,279,8276,15511,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,10025,2129,1979,816,3147,11855,8934,198,32929,220,17,25,220,4754,4532,3147,57164,2129,1979,816,3147,33426,8934,198,32929,220,18,25,220,4754,4532,3147,40582,2129,1979,816,3147,33034,8934,198,32929,220,19,25,220,4754,4532,3147,46110,2129,1979,816,3147,26964,8934,198,4537,38945,198,198,27806,220,17,198,887,25,888,39286,198,9222,25,307,10644,198,669,55530,25,220,32,3074,15511,4829,685,777,1056,4139,314,279,12307,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,23856,5916,198,9222,25,5576,198,669,55530,25,220,22365,12643,2306,16673,310,279,23856,15511,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1736,802,2129,1979,816,3147,19354,8934,198,32929,220,17,25,220,4754,4532,3147,57572,2129,1979,816,3147,4009,8934,198,32929,220,18,25,220,4754,4532,3147,29280,2129,1979,816,3147,1520,8934,198,32929,220,19,25,220,4754,4532,3147,54,18577,2129,1979,816,3147,12615,8934,198,4537,38945,198,198,27806,220,19,198,887,25,6105,37003,198,9222,25,5284,198,669,55530,25,220,49,1336,391,1092,4191,314,279,4208,369,6105,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,7744,264,8079,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,40486,264,8079,321,264,4139,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,1908,264,4139,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"biggest_slice","type":"choice","question_span":[146,153],"option_spans":[[164,174],[180,190],[196,206],[212,222]],"option_ids":["blue","grey","orange","yellow"],"logits":[-3.508026599884033,-5.220736026763916,-0.10318532586097717,-5.523584365844727],"probs":[0.03182368725538254,0.005740241147577763,0.9581957459449768,0.004240380134433508],"argmax_index":2,"argmax_id":"orange","top2_margin":0.9263720586895943,"near_tie":false,"gold":"orange","gold_correct":true},{"question_id":"over_half","type":"noul","question_span":[245,257],"option_spans":[[268,286],[292,310]],"option_ids":["true","false"],"logits":[-0.7230271100997925,0.3004370927810669],"probs":[0.2643531858921051,0.7356468439102173],"argmax_index":1,"argmax_id":"false","top2_margin":0.4712936580181122,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"smallest_label","type":"choice","question_span":[332,341],"option_spans":[[352,363],[369,379],[385,395],[401,412]],"option_ids":["heat","light","other","water"],"logits":[-5.1592841148376465,0.2018813192844391,-4.390270709991455,-4.602262020111084],"probs":[0.0045897639356553555,0.9774959087371826,0.009903042577207088,0.008011280559003353],"argmax_index":1,"argmax_id":"light","top2_margin":0.9675928661599755,"near_tie":false,"gold":"light","gold_correct":true},{"question_id":"blue_share","type":"score","question_span":[434,445],"option_spans":[[456,468],[474,489],[495,507]],"option_ids":["0","1","2"],"logits":[-3.093881130218506,0.47458580136299133,-3.1774089336395264],"probs":[0.026750806719064713,0.9486420154571533,0.02460714429616928],"argmax_index":1,"argmax_id":"1","top2_margin":0.9218912087380886,"near_tie":false,"gold":"1","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"biggest_slice":{"type":"choice","choice":"orange","confidence":0.9582,"probabilities":{"orange":0.9582,"blue":0.0318,"grey":0.0057,"yellow":0.0042}},"over_half":{"type":"noul","noul":0.2644},"smallest_label":{"type":"choice","choice":"light","confidence":0.9775,"probabilities":{"heat":0.0046,"water":0.008,"other":0.0099,"light":0.9775}},"blue_share":{"type":"score","score":0.9979,"confidence":0.9486,"legend":{"0":"under a quarter","1":"between a quarter and a half","2":"over a half"},"probabilities":{"0":0.0268,"1":0.9486,"2":0.0246}}},"usage":{"input_tokens":529,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_i01","arm":"g448","set":"heldout","source":"heldout_image","tokens":661,"n_image_tokens":196,"state_tokens":18,"token_offset":36,"image_files":["ho_pie_720x480.png"],"image_size_in":[720,480],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"e1d73f35929d8da0a1d51e9c685766b2cdc18d7e7e25f6a510e704d7c99a4eb7","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,32,13338,579,42800,4649,958,11,6507,539,7193,11,369,14336,430,264,4208,9207,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,8276,25640,198,9222,25,5576,198,669,55530,25,220,22365,12106,369,279,8276,15511,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,10025,2129,1979,816,3147,11855,8934,198,32929,220,17,25,220,4754,4532,3147,57164,2129,1979,816,3147,33426,8934,198,32929,220,18,25,220,4754,4532,3147,40582,2129,1979,816,3147,33034,8934,198,32929,220,19,25,220,4754,4532,3147,46110,2129,1979,816,3147,26964,8934,198,4537,38945,198,198,27806,220,17,198,887,25,888,39286,198,9222,25,307,10644,198,669,55530,25,220,32,3074,15511,4829,685,777,1056,4139,314,279,12307,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,23856,5916,198,9222,25,5576,198,669,55530,25,220,22365,12643,2306,16673,310,279,23856,15511,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1736,802,2129,1979,816,3147,19354,8934,198,32929,220,17,25,220,4754,4532,3147,57572,2129,1979,816,3147,4009,8934,198,32929,220,18,25,220,4754,4532,3147,29280,2129,1979,816,3147,1520,8934,198,32929,220,19,25,220,4754,4532,3147,54,18577,2129,1979,816,3147,12615,8934,198,4537,38945,198,198,27806,220,19,198,887,25,6105,37003,198,9222,25,5284,198,669,55530,25,220,49,1336,391,1092,4191,314,279,4208,369,6105,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,7744,264,8079,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,40486,264,8079,321,264,4139,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,1908,264,4139,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"biggest_slice","type":"choice","question_span":[278,285],"option_spans":[[296,306],[312,322],[328,338],[344,354]],"option_ids":["blue","grey","orange","yellow"],"logits":[-3.4125235080718994,-5.322712421417236,-0.12316480278968811,-5.445016860961914],"probs":[0.03558121994137764,0.005267884116619825,0.9544894695281982,0.004661439917981625],"argmax_index":2,"argmax_id":"orange","top2_margin":0.9189082495868206,"near_tie":false,"gold":"orange","gold_correct":true},{"question_id":"over_half","type":"noul","question_span":[377,389],"option_spans":[[400,418],[424,442]],"option_ids":["true","false"],"logits":[-0.7918766736984253,0.04384505748748779],"probs":[0.30243659019470215,0.6975634098052979],"argmax_index":1,"argmax_id":"false","top2_margin":0.3951268196105957,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"smallest_label","type":"choice","question_span":[464,473],"option_spans":[[484,495],[501,511],[517,527],[533,544]],"option_ids":["heat","light","other","water"],"logits":[-5.214592456817627,0.3628416061401367,-4.135987758636475,-4.941781044006348],"probs":[0.0037085588555783033,0.9805144667625427,0.010905290953814983,0.004871757701039314],"argmax_index":1,"argmax_id":"light","top2_margin":0.9696091758087277,"near_tie":false,"gold":"light","gold_correct":true},{"question_id":"blue_share","type":"score","question_span":[566,577],"option_spans":[[588,600],[606,621],[627,639]],"option_ids":["0","1","2"],"logits":[-2.981114625930786,0.5211405754089355,-2.789916515350342],"probs":[0.028247857466340065,0.9375525712966919,0.034199658781290054],"argmax_index":1,"argmax_id":"1","top2_margin":0.9033529125154018,"near_tie":false,"gold":"1","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"biggest_slice":{"type":"choice","choice":"orange","confidence":0.9545,"probabilities":{"orange":0.9545,"blue":0.0356,"grey":0.0053,"yellow":0.0047}},"over_half":{"type":"noul","noul":0.3024},"smallest_label":{"type":"choice","choice":"light","confidence":0.9805,"probabilities":{"heat":0.0037,"water":0.0049,"other":0.0109,"light":0.9805}},"blue_share":{"type":"score","score":1.006,"confidence":0.9376,"legend":{"0":"under a quarter","1":"between a quarter and a half","2":"over a half"},"probabilities":{"0":0.0282,"1":0.9376,"2":0.0342}}},"usage":{"input_tokens":661,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_i02","arm":"g256","set":"heldout","source":"heldout_image","tokens":570,"n_image_tokens":64,"state_tokens":20,"token_offset":36,"image_files":["ho_table_1000x600.png"],"image_size_in":[1000,600],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"63732a7dba47ee0a847f2a34c1f12b9f918e7359039910c7565eb7c6e7047e00","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,19542,3147,47,1656,382,76,540,2129,7945,3147,10071,685,26791,383,279,24125,1898,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,4700,9997,839,33126,198,9222,25,5576,198,669,55530,25,220,22365,5171,1503,279,5061,9997,839,5257,5106,494,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16786,220,16,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,16786,220,17,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,16786,220,18,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,16786,220,19,2129,1979,816,3147,19,8934,198,4537,38945,198,198,27806,220,17,198,887,25,28689,198,9222,25,5576,198,669,55530,25,220,8791,1503,279,28689,24125,708,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,2196,634,2829,68,2129,1979,816,3147,1329,2829,68,8934,198,32929,220,17,25,220,4754,4532,3147,38,2798,917,633,2129,1979,816,3147,70,2798,917,633,8934,198,32929,220,18,25,220,4754,4532,3147,25642,9997,839,2129,1979,816,3147,12610,9997,839,8934,198,32929,220,19,25,220,4754,4532,3147,51,3434,3621,1634,2129,1979,816,3147,83,3434,3621,1634,8934,198,4537,38945,198,198,27806,220,18,198,887,25,5171,62,18,198,9222,25,5576,198,669,55530,25,220,4199,1599,11184,1365,958,5171,220,18,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,6487,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,4537,38945,198,198,27806,220,19,198,887,25,1238,1054,472,198,9222,25,307,10644,198,669,55530,25,220,760,1898,11140,264,24125,2843,1056,220,15,24,25,15,15,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"harlowick_platform","type":"choice","question_span":[150,161],"option_spans":[[172,184],[190,202],[208,220],[226,238]],"option_ids":["1","2","3","4"],"logits":[0.897986114025116,-4.707549571990967,-4.621336936950684,-4.861471176147461],"probs":[0.9892774820327759,0.0036380193196237087,0.003965579438954592,0.003119016531854868],"argmax_index":0,"argmax_id":"1","top2_margin":0.9853119025938213,"near_tie":false,"gold":"1","gold_correct":true},{"question_id":"earliest","type":"choice","question_span":[259,266],"option_spans":[[277,292],[298,314],[320,334],[340,356]],"option_ids":["emberdowne","glenvarrow","harlowick","tumbrelby"],"logits":[-4.685932636260986,-0.12817353010177612,-4.358098983764648,-4.608298301696777],"probs":[0.010117542929947376,0.9649054408073425,0.01404272485524416,0.010934306308627129],"argmax_index":1,"argmax_id":"glenvarrow","top2_margin":0.9508627159520984,"near_tie":false,"gold":"glenvarrow","gold_correct":true},{"question_id":"platform_3","type":"choice","question_span":[379,388],"option_spans":[[399,409],[415,425],[431,441],[447,457]],"option_ids":["0","1","2","3"],"logits":[-4.488337516784668,-3.8162689208984375,-0.2855433225631714,-4.313675880432129],"probs":[0.014080134220421314,0.027572903782129288,0.9415796995162964,0.016767233610153198],"argmax_index":2,"argmax_id":"2","top2_margin":0.9140067957341671,"near_tie":false,"gold":"2","gold_correct":true},{"question_id":"after_nine","type":"noul","question_span":[481,495],"option_spans":[[506,524],[530,548]],"option_ids":["true","false"],"logits":[-0.31948068737983704,-3.8522603511810303],"probs":[0.9716061949729919,0.028393803164362907],"argmax_index":0,"argmax_id":"true","top2_margin":0.943212391808629,"near_tie":false,"gold":"true","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"harlowick_platform":{"type":"choice","choice":"1","confidence":0.9893,"probabilities":{"1":0.9893,"2":0.0036,"3":0.004,"4":0.0031}},"earliest":{"type":"choice","choice":"glenvarrow","confidence":0.9649,"probabilities":{"glenvarrow":0.9649,"harlowick":0.014,"tumbrelby":0.0109,"emberdowne":0.0101}},"platform_3":{"type":"choice","choice":"2","confidence":0.9416,"probabilities":{"0":0.0141,"1":0.0276,"2":0.9416,"3":0.0168}},"after_nine":{"type":"noul","noul":0.9716}},"usage":{"input_tokens":570,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_i02","arm":"g448","set":"heldout","source":"heldout_image","tokens":702,"n_image_tokens":196,"state_tokens":20,"token_offset":36,"image_files":["ho_table_1000x600.png"],"image_size_in":[1000,600],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"a96a0c1c561180eced11d0035fa3082326629299adcfc51d047004563797205d","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,19542,3147,47,1656,382,76,540,2129,7945,3147,10071,685,26791,383,279,24125,1898,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,4700,9997,839,33126,198,9222,25,5576,198,669,55530,25,220,22365,5171,1503,279,5061,9997,839,5257,5106,494,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16786,220,16,2129,1979,816,3147,16,8934,198,32929,220,17,25,220,4754,4532,3147,16786,220,17,2129,1979,816,3147,17,8934,198,32929,220,18,25,220,4754,4532,3147,16786,220,18,2129,1979,816,3147,18,8934,198,32929,220,19,25,220,4754,4532,3147,16786,220,19,2129,1979,816,3147,19,8934,198,4537,38945,198,198,27806,220,17,198,887,25,28689,198,9222,25,5576,198,669,55530,25,220,8791,1503,279,28689,24125,708,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,2196,634,2829,68,2129,1979,816,3147,1329,2829,68,8934,198,32929,220,17,25,220,4754,4532,3147,38,2798,917,633,2129,1979,816,3147,70,2798,917,633,8934,198,32929,220,18,25,220,4754,4532,3147,25642,9997,839,2129,1979,816,3147,12610,9997,839,8934,198,32929,220,19,25,220,4754,4532,3147,51,3434,3621,1634,2129,1979,816,3147,83,3434,3621,1634,8934,198,4537,38945,198,198,27806,220,18,198,887,25,5171,62,18,198,9222,25,5576,198,669,55530,25,220,4199,1599,11184,1365,958,5171,220,18,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,6487,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,588,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,19186,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,4537,38945,198,198,27806,220,19,198,887,25,1238,1054,472,198,9222,25,307,10644,198,669,55530,25,220,760,1898,11140,264,24125,2843,1056,220,15,24,25,15,15,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"harlowick_platform","type":"choice","question_span":[282,293],"option_spans":[[304,316],[322,334],[340,352],[358,370]],"option_ids":["1","2","3","4"],"logits":[0.6071804761886597,-4.543316841125488,-4.545759677886963,-4.8027753829956055],"probs":[0.9842028617858887,0.0057049524039030075,0.0056910328567028046,0.004401194863021374],"argmax_index":0,"argmax_id":"1","top2_margin":0.9784979093819857,"near_tie":false,"gold":"1","gold_correct":true},{"question_id":"earliest","type":"choice","question_span":[391,398],"option_spans":[[409,424],[430,446],[452,466],[472,488]],"option_ids":["emberdowne","glenvarrow","harlowick","tumbrelby"],"logits":[-4.614399433135986,-0.1071513295173645,-4.359019756317139,-4.438200950622559],"probs":[0.01062072068452835,0.9630013108253479,0.013710836879909039,0.012667064554989338],"argmax_index":1,"argmax_id":"glenvarrow","top2_margin":0.9492904739454389,"near_tie":false,"gold":"glenvarrow","gold_correct":true},{"question_id":"platform_3","type":"choice","question_span":[511,520],"option_spans":[[531,541],[547,557],[563,573],[579,589]],"option_ids":["0","1","2","3"],"logits":[-4.442370414733887,-3.9658191204071045,-0.5411005020141602,-4.022770404815674],"probs":[0.018657730892300606,0.030048472806811333,0.9229088425636292,0.028384990990161896],"argmax_index":2,"argmax_id":"2","top2_margin":0.8928603697568178,"near_tie":false,"gold":"2","gold_correct":true},{"question_id":"after_nine","type":"noul","question_span":[613,627],"option_spans":[[638,656],[662,680]],"option_ids":["true","false"],"logits":[-0.3887670636177063,-3.8327274322509766],"probs":[0.9690505266189575,0.030949482694268227],"argmax_index":0,"argmax_id":"true","top2_margin":0.9381010439246893,"near_tie":false,"gold":"true","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"harlowick_platform":{"type":"choice","choice":"1","confidence":0.9842,"probabilities":{"1":0.9842,"2":0.0057,"3":0.0057,"4":0.0044}},"earliest":{"type":"choice","choice":"glenvarrow","confidence":0.963,"probabilities":{"glenvarrow":0.963,"harlowick":0.0137,"tumbrelby":0.0127,"emberdowne":0.0106}},"platform_3":{"type":"choice","choice":"2","confidence":0.9229,"probabilities":{"0":0.0187,"1":0.03,"2":0.9229,"3":0.0284}},"after_nine":{"type":"noul","noul":0.9691}},"usage":{"input_tokens":702,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_i03","arm":"g256","set":"heldout","source":"heldout_image","tokens":541,"n_image_tokens":64,"state_tokens":7,"token_offset":36,"image_files":["ho_sign_200x150.png"],"image_size_in":[200,150],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"1162fd21a61d7e887bf26c13786dead299b765b3996a04c54feda607d487fa6c","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,32,5404,15734,411,5459,1777,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,3798,198,9222,25,5576,198,669,55530,25,220,3710,3798,1503,279,1777,1420,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16,15,15,2129,1979,816,3147,16,15,15,8934,198,32929,220,17,25,220,4754,4532,3147,16,17,15,2129,1979,816,3147,16,17,15,8934,198,32929,220,18,25,220,4754,4532,3147,17,15,2129,1979,816,3147,17,15,8934,198,32929,220,19,25,220,4754,4532,3147,18,15,2129,1979,816,3147,18,15,8934,198,32929,220,20,25,220,4754,4532,3147,19,15,2129,1979,816,3147,19,15,8934,198,32929,220,21,25,220,4754,4532,3147,20,15,2129,1979,816,3147,20,15,8934,198,32929,220,22,25,220,4754,4532,3147,21,15,2129,1979,816,3147,21,15,8934,198,32929,220,23,25,220,4754,4532,3147,22,15,2129,1979,816,3147,22,15,8934,198,32929,220,24,25,220,4754,4532,3147,23,15,2129,1979,816,3147,23,15,8934,198,32929,220,16,15,25,220,4754,4532,3147,24,15,2129,1979,816,3147,24,15,8934,198,4537,38945,198,198,27806,220,17,198,887,25,20454,198,9222,25,5576,198,669,55530,25,220,3710,369,279,20454,314,279,1777,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,57087,48085,2129,1979,816,3147,40314,5922,8934,198,32929,220,17,25,220,4754,4532,3147,26605,2129,1979,816,3147,1014,8934,198,32929,220,18,25,220,4754,4532,3147,32168,2129,1979,816,3147,36234,8934,198,32929,220,19,25,220,4754,4532,3147,21211,4095,2129,1979,816,3147,53264,8934,198,4537,38945,198,198,27806,220,18,198,887,25,67579,19212,198,9222,25,307,10644,198,669,55530,25,220,90601,506,220,22,15,25710,2785,279,3798,6625,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"limit","type":"choice","question_span":[134,141],"option_spans":[[152,166],[172,186],[192,204],[210,222],[228,240],[246,258],[264,276],[282,294],[300,312],[319,331]],"option_ids":["100","120","20","30","40","50","60","70","80","90"],"logits":[-4.553650856018066,-4.660093307495117,-4.715177536010742,-4.662933826446533,-4.677515029907227,-4.4358229637146,0.6220638751983643,-4.316461086273193,-4.5895819664001465,-4.647860050201416],"probs":[0.00538443960249424,0.004840755369514227,0.004581316839903593,0.004827022086828947,0.00475715147331357,0.006057763006538153,0.9526310563087463,0.0068257516250014305,0.005194402299821377,0.004900334868580103],"argmax_index":6,"argmax_id":"60","top2_margin":0.9458053046837449,"near_tie":false,"gold":"60","gold_correct":true},{"question_id":"outline","type":"choice","question_span":[352,360],"option_spans":[[371,383],[389,399],[405,415],[421,432]],"option_ids":["octagon","round","square","triangle"],"logits":[-4.9417853355407715,-1.1671185493469238,-5.121089935302734,-5.117566108703613],"probs":[0.021618053317070007,0.9421792030334473,0.018069475889205933,0.018133262172341347],"argmax_index":1,"argmax_id":"round","top2_margin":0.9205611497163773,"near_tie":false,"gold":"round","gold_correct":true},{"question_id":"seventy_ok","type":"noul","question_span":[455,466],"option_spans":[[477,495],[501,519]],"option_ids":["true","false"],"logits":[-4.288346767425537,-0.5070703029632568],"probs":[0.02228561043739319,0.9777144193649292],"argmax_index":1,"argmax_id":"false","top2_margin":0.955428808927536,"near_tie":false,"gold":"false","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"limit":{"type":"choice","choice":"60","confidence":0.9526,"probabilities":{"20":0.0046,"30":0.0048,"40":0.0048,"50":0.0061,"60":0.9526,"70":0.0068,"80":0.0052,"90":0.0049,"100":0.0054,"120":0.0048}},"outline":{"type":"choice","choice":"round","confidence":0.9422,"probabilities":{"round":0.9422,"triangle":0.0181,"square":0.0181,"octagon":0.0216}},"seventy_ok":{"type":"noul","noul":0.0223}},"usage":{"input_tokens":541,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_i03","arm":"g448","set":"heldout","source":"heldout_image","tokens":673,"n_image_tokens":196,"state_tokens":7,"token_offset":36,"image_files":["ho_sign_200x150.png"],"image_size_in":[200,150],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"47469361376ad364b95d60e675952df6aa4acf102cad3381e3b269729d78efa4","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,32,5404,15734,411,5459,1777,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,3798,198,9222,25,5576,198,669,55530,25,220,3710,3798,1503,279,1777,1420,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16,15,15,2129,1979,816,3147,16,15,15,8934,198,32929,220,17,25,220,4754,4532,3147,16,17,15,2129,1979,816,3147,16,17,15,8934,198,32929,220,18,25,220,4754,4532,3147,17,15,2129,1979,816,3147,17,15,8934,198,32929,220,19,25,220,4754,4532,3147,18,15,2129,1979,816,3147,18,15,8934,198,32929,220,20,25,220,4754,4532,3147,19,15,2129,1979,816,3147,19,15,8934,198,32929,220,21,25,220,4754,4532,3147,20,15,2129,1979,816,3147,20,15,8934,198,32929,220,22,25,220,4754,4532,3147,21,15,2129,1979,816,3147,21,15,8934,198,32929,220,23,25,220,4754,4532,3147,22,15,2129,1979,816,3147,22,15,8934,198,32929,220,24,25,220,4754,4532,3147,23,15,2129,1979,816,3147,23,15,8934,198,32929,220,16,15,25,220,4754,4532,3147,24,15,2129,1979,816,3147,24,15,8934,198,4537,38945,198,198,27806,220,17,198,887,25,20454,198,9222,25,5576,198,669,55530,25,220,3710,369,279,20454,314,279,1777,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,57087,48085,2129,1979,816,3147,40314,5922,8934,198,32929,220,17,25,220,4754,4532,3147,26605,2129,1979,816,3147,1014,8934,198,32929,220,18,25,220,4754,4532,3147,32168,2129,1979,816,3147,36234,8934,198,32929,220,19,25,220,4754,4532,3147,21211,4095,2129,1979,816,3147,53264,8934,198,4537,38945,198,198,27806,220,18,198,887,25,67579,19212,198,9222,25,307,10644,198,669,55530,25,220,90601,506,220,22,15,25710,2785,279,3798,6625,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"limit","type":"choice","question_span":[266,273],"option_spans":[[284,298],[304,318],[324,336],[342,354],[360,372],[378,390],[396,408],[414,426],[432,444],[451,463]],"option_ids":["100","120","20","30","40","50","60","70","80","90"],"logits":[-4.616436004638672,-4.696988105773926,-4.7531609535217285,-4.726918697357178,-4.73721170425415,-4.551877975463867,0.5937036871910095,-4.374990463256836,-4.625844955444336,-4.711438179016113],"probs":[0.005210073199123144,0.0048068491742014885,0.0045442781411111355,0.004665108863264322,0.004617337137460709,0.005557519383728504,0.9540667533874512,0.006632882170379162,0.005161281209439039,0.004737889394164085],"argmax_index":6,"argmax_id":"60","top2_margin":0.947433871217072,"near_tie":false,"gold":"60","gold_correct":true},{"question_id":"outline","type":"choice","question_span":[484,492],"option_spans":[[503,515],[521,531],[537,547],[553,564]],"option_ids":["octagon","round","square","triangle"],"logits":[-4.952449798583984,-0.9934857487678528,-5.14432954788208,-5.095734119415283],"probs":[0.018150491639971733,0.9511404633522034,0.014981530606746674,0.015727544203400612],"argmax_index":1,"argmax_id":"round","top2_margin":0.9329899717122316,"near_tie":false,"gold":"round","gold_correct":true},{"question_id":"seventy_ok","type":"noul","question_span":[587,598],"option_spans":[[609,627],[633,651]],"option_ids":["true","false"],"logits":[-4.399203300476074,-0.5585627555847168],"probs":[0.021028155460953712,0.9789718389511108],"argmax_index":1,"argmax_id":"false","top2_margin":0.9579436834901571,"near_tie":false,"gold":"false","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"limit":{"type":"choice","choice":"60","confidence":0.9541,"probabilities":{"20":0.0045,"30":0.0047,"40":0.0046,"50":0.0056,"60":0.9541,"70":0.0066,"80":0.0052,"90":0.0047,"100":0.0052,"120":0.0048}},"outline":{"type":"choice","choice":"round","confidence":0.9511,"probabilities":{"round":0.9511,"triangle":0.0157,"square":0.015,"octagon":0.0182}},"seventy_ok":{"type":"noul","noul":0.021}},"usage":{"input_tokens":673,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_i04","arm":"g256","set":"heldout","source":"heldout_image","tokens":437,"n_image_tokens":64,"state_tokens":12,"token_offset":36,"image_files":["ho_thermometer_160x400.png"],"image_size_in":[160,400],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"95326a575ee6bb01f0574759eb4abde071eb40c288f2c7ca738eca16189073c1","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,64130,3147,403,10470,89016,2129,12088,3147,67387,59275,8934,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5122,198,9222,25,5576,198,669,55530,25,220,3710,9039,1503,279,89016,1420,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,8823,220,16,20,2129,1979,816,3147,16,20,8934,198,32929,220,17,25,220,4754,4532,3147,8823,220,17,20,2129,1979,816,3147,17,20,8934,198,32929,220,18,25,220,4754,4532,3147,8823,220,18,20,2129,1979,816,3147,18,20,8934,198,32929,220,19,25,220,4754,4532,3147,8823,220,19,20,2129,1979,816,3147,19,20,8934,198,4537,38945,198,198,27806,220,17,198,887,25,3559,18772,198,9222,25,307,10644,198,669,55530,25,220,760,5122,369,3559,6942,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,39026,198,9222,25,5284,198,669,55530,25,220,4199,7948,369,424,11,2019,539,279,5122,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,10279,19752,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,40797,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,76,676,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,10307,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"reading","type":"choice","question_span":[139,146],"option_spans":[[157,171],[177,191],[197,211],[217,231]],"option_ids":["15","25","35","45"],"logits":[-3.0900797843933105,-1.7568778991699219,1.3087131977081299,-2.5198535919189453],"probs":[0.011374692432582378,0.04314614087343216,0.925361156463623,0.020118048414587975],"argmax_index":2,"argmax_id":"35","top2_margin":0.8822150155901909,"near_tie":false,"gold":"35","gold_correct":true},{"question_id":"below_zero","type":"noul","question_span":[254,260],"option_spans":[[271,289],[295,313]],"option_ids":["true","false"],"logits":[-3.8248190879821777,1.8737088441848755],"probs":[0.0033397041261196136,0.9966603517532349],"argmax_index":1,"argmax_id":"false","top2_margin":0.9933206476271152,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"warmth","type":"score","question_span":[334,344],"option_spans":[[355,366],[372,382],[388,399],[405,415]],"option_ids":["0","1","2","3"],"logits":[-4.11448860168457,-1.3652338981628418,-0.5080658197402954,-1.1960772275924683],"probs":[0.013893346302211285,0.21716661751270294,0.5117483139038086,0.25719165802001953],"argmax_index":2,"argmax_id":"2","top2_margin":0.25455665588378906,"near_tie":false,"gold":"3","gold_correct":false}],"systemone_response":{"model":"clef-flash","answers":{"reading":{"type":"choice","choice":"35","confidence":0.9254,"probabilities":{"15":0.0114,"25":0.0431,"35":0.9254,"45":0.0201}},"below_zero":{"type":"noul","noul":0.0033},"warmth":{"type":"score","score":2.0122,"confidence":0.5117,"legend":{"0":"freezing","1":"cool","2":"mild","3":"hot"},"probabilities":{"0":0.0139,"1":0.2172,"2":0.5117,"3":0.2572}}},"usage":{"input_tokens":437,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_i04","arm":"g448","set":"heldout","source":"heldout_image","tokens":569,"n_image_tokens":196,"state_tokens":12,"token_offset":36,"image_files":["ho_thermometer_160x400.png"],"image_size_in":[160,400],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"02b47252416bce7cdcd3a3295cccf19c56c473746ccdbc6bf9973b132cba4168","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,64130,3147,403,10470,89016,2129,12088,3147,67387,59275,8934,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,5122,198,9222,25,5576,198,669,55530,25,220,3710,9039,1503,279,89016,1420,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,8823,220,16,20,2129,1979,816,3147,16,20,8934,198,32929,220,17,25,220,4754,4532,3147,8823,220,17,20,2129,1979,816,3147,17,20,8934,198,32929,220,18,25,220,4754,4532,3147,8823,220,18,20,2129,1979,816,3147,18,20,8934,198,32929,220,19,25,220,4754,4532,3147,8823,220,19,20,2129,1979,816,3147,19,20,8934,198,4537,38945,198,198,27806,220,17,198,887,25,3559,18772,198,9222,25,307,10644,198,669,55530,25,220,760,5122,369,3559,6942,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,39026,198,9222,25,5284,198,669,55530,25,220,4199,7948,369,424,11,2019,539,279,5122,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,10279,19752,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,40797,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,76,676,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,10307,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"reading","type":"choice","question_span":[271,278],"option_spans":[[289,303],[309,323],[329,343],[349,363]],"option_ids":["15","25","35","45"],"logits":[-2.6599154472351074,-0.9549970030784607,1.3946359157562256,-3.1883702278137207],"probs":[0.015444081276655197,0.08495689183473587,0.8904945254325867,0.009104520082473755],"argmax_index":2,"argmax_id":"35","top2_margin":0.8055376335978508,"near_tie":false,"gold":"35","gold_correct":true},{"question_id":"below_zero","type":"noul","question_span":[386,392],"option_spans":[[403,421],[427,445]],"option_ids":["true","false"],"logits":[-3.8532285690307617,1.9849202632904053],"probs":[0.0029057636857032776,0.9970942735671997],"argmax_index":1,"argmax_id":"false","top2_margin":0.9941885098814964,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"warmth","type":"score","question_span":[466,476],"option_spans":[[487,498],[504,514],[520,531],[537,547]],"option_ids":["0","1","2","3"],"logits":[-3.9894585609436035,-1.0499372482299805,0.09225502610206604,-1.4359924793243408],"probs":[0.010868941433727741,0.20549684762954712,0.6439511775970459,0.13968297839164734],"argmax_index":2,"argmax_id":"2","top2_margin":0.4384543299674988,"near_tie":false,"gold":"3","gold_correct":false}],"systemone_response":{"model":"clef-flash","answers":{"reading":{"type":"choice","choice":"35","confidence":0.8905,"probabilities":{"15":0.0154,"25":0.085,"35":0.8905,"45":0.0091}},"below_zero":{"type":"noul","noul":0.0029},"warmth":{"type":"score","score":1.9124,"confidence":0.644,"legend":{"0":"freezing","1":"cool","2":"mild","3":"hot"},"probabilities":{"0":0.0109,"1":0.2055,"2":0.644,"3":0.1397}}},"usage":{"input_tokens":569,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_i05","arm":"g256","set":"heldout","source":"heldout_image","tokens":701,"n_image_tokens":64,"state_tokens":9,"token_offset":36,"image_files":["ho_calendar_760x700.png"],"image_size_in":[760,700],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"d97bafb4f8390a78aa4825b9927eed4c85aaa4a28b4bfec0fb1a88d185441afa","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,3833,2081,314,264,6784,12786,369,5064,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,1118,30250,1247,198,9222,25,5576,198,669,55530,25,220,1860,864,45022,1503,279,2182,3060,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,33475,2129,1979,816,3147,188335,8934,198,32929,220,17,25,220,4754,4532,3147,34072,2129,1979,816,3147,1588,8934,198,32929,220,18,25,220,4754,4532,3147,32558,2129,1979,816,3147,35264,8934,198,32929,220,19,25,220,4754,4532,3147,34871,2129,1979,816,3147,38211,8934,198,32929,220,20,25,220,4754,4532,3147,36539,2129,1979,816,3147,199231,8934,198,32929,220,21,25,220,4754,4532,3147,38867,2129,1979,816,3147,83,356,8934,198,32929,220,22,25,220,4754,4532,3147,39181,2129,1979,816,3147,65092,8934,198,4537,38945,198,198,27806,220,17,198,887,25,4086,806,198,9222,25,5576,198,669,55530,25,220,22365,2321,369,4086,806,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1719,220,16,19,337,2129,1979,816,3147,16,19,8934,198,32929,220,17,25,220,4754,4532,3147,1719,220,17,16,267,2129,1979,816,3147,17,16,8934,198,32929,220,18,25,220,4754,4532,3147,1719,220,17,23,337,2129,1979,816,3147,17,23,8934,198,32929,220,19,25,220,4754,4532,3147,1719,220,22,337,2129,1979,816,3147,22,8934,198,4537,38945,198,198,27806,220,18,198,887,25,2182,4956,198,9222,25,5576,198,669,55530,25,220,4199,1599,2756,1503,279,2182,6625,599,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,17,23,2129,1979,816,3147,17,23,8934,198,32929,220,17,25,220,4754,4532,3147,17,24,2129,1979,816,3147,17,24,8934,198,32929,220,18,25,220,4754,4532,3147,18,15,2129,1979,816,3147,18,15,8934,198,32929,220,19,25,220,4754,4532,3147,18,16,2129,1979,816,3147,18,16,8934,198,4537,38945,198,198,27806,220,19,198,887,25,88603,30250,400,198,9222,25,307,10644,198,669,55530,25,220,760,88603,2424,314,2756,5469,264,7483,466,264,7043,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,20,198,887,25,88603,3079,198,9222,25,5284,198,669,55530,25,220,4199,1599,2756,513,88603,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,6487,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,588,310,2250,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,32897,310,4590,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,77420,466,777,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"first_weekday","type":"choice","question_span":[138,146],"option_spans":[[157,167],[173,183],[189,199],[205,215],[221,231],[237,248],[254,264]],"option_ids":["fri","mon","sat","sun","thu","tue","wed"],"logits":[-2.785665988922119,0.40149974822998047,-3.030606985092163,-2.8934242725372314,-2.684436559677124,-1.9534865617752075,-2.296431541442871],"probs":[0.03131243214011192,0.758377194404602,0.02450982667505741,0.028113696724176407,0.03464815765619278,0.07196606695652008,0.05107267573475838],"argmax_index":1,"argmax_id":"mon","top2_margin":0.686411127448082,"near_tie":false,"gold":"mon","gold_correct":true},{"question_id":"circled","type":"choice","question_span":[286,292],"option_spans":[[303,318],[324,339],[345,360],[366,379]],"option_ids":["14","21","28","7"],"logits":[1.0718114376068115,-3.874310255050659,-4.462723731994629,-3.9891598224639893],"probs":[0.9828990697860718,0.006989330053329468,0.003880527336150408,0.006230989471077919],"argmax_index":0,"argmax_id":"14","top2_margin":0.9759097397327423,"near_tie":false,"gold":"14","gold_correct":true},{"question_id":"month_length","type":"choice","question_span":[401,410],"option_spans":[[421,433],[439,451],[457,469],[475,487]],"option_ids":["28","29","30","31"],"logits":[1.1878530979156494,-2.42960524559021,-3.1047449111938477,-3.2113656997680664],"probs":[0.9498416185379028,0.025504039600491524,0.012983736582100391,0.011670645326375961],"argmax_index":0,"argmax_id":"28","top2_margin":0.9243375789374113,"near_tie":false,"gold":"28","gold_correct":true},{"question_id":"shaded_weekend","type":"noul","question_span":[511,523],"option_spans":[[534,552],[558,576]],"option_ids":["true","false"],"logits":[-1.3491235971450806,0.5485743880271912],"probs":[0.13036924600601196,0.869630753993988],"argmax_index":1,"argmax_id":"false","top2_margin":0.7392615079879761,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"shaded_count","type":"score","question_span":[598,604],"option_spans":[[615,625],[631,643],[649,661],[667,679]],"option_ids":["0","1","2","3"],"logits":[-3.1390182971954346,-3.1964261531829834,-0.7756245136260986,-0.3811075687408447],"probs":[0.03528826683759689,0.033319491893053055,0.3750068247318268,0.5563853979110718],"argmax_index":3,"argmax_id":"3","top2_margin":0.181378573179245,"near_tie":false,"gold":"2","gold_correct":false}],"systemone_response":{"model":"clef-flash","answers":{"first_weekday":{"type":"choice","choice":"mon","confidence":0.7584,"probabilities":{"mon":0.7584,"tue":0.072,"wed":0.0511,"thu":0.0346,"fri":0.0313,"sat":0.0245,"sun":0.0281}},"circled":{"type":"choice","choice":"14","confidence":0.9829,"probabilities":{"7":0.0062,"14":0.9829,"21":0.007,"28":0.0039}},"month_length":{"type":"choice","choice":"28","confidence":0.9498,"probabilities":{"28":0.9498,"29":0.0255,"30":0.013,"31":0.0117}},"shaded_weekend":{"type":"noul","noul":0.1304},"shaded_count":{"type":"score","score":2.4525,"confidence":0.5564,"legend":{"0":"none","1":"one to three","2":"four to six","3":"seven or more"},"probabilities":{"0":0.0353,"1":0.0333,"2":0.375,"3":0.5564}}},"usage":{"input_tokens":701,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_i05","arm":"g448","set":"heldout","source":"heldout_image","tokens":833,"n_image_tokens":196,"state_tokens":9,"token_offset":36,"image_files":["ho_calendar_760x700.png"],"image_size_in":[760,700],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"9dde07efc92bb64a4e644fe627d98efee8852540d0a29d9a0f53d75a12e03e8a","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,3833,2081,314,264,6784,12786,369,5064,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,1118,30250,1247,198,9222,25,5576,198,669,55530,25,220,1860,864,45022,1503,279,2182,3060,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,33475,2129,1979,816,3147,188335,8934,198,32929,220,17,25,220,4754,4532,3147,34072,2129,1979,816,3147,1588,8934,198,32929,220,18,25,220,4754,4532,3147,32558,2129,1979,816,3147,35264,8934,198,32929,220,19,25,220,4754,4532,3147,34871,2129,1979,816,3147,38211,8934,198,32929,220,20,25,220,4754,4532,3147,36539,2129,1979,816,3147,199231,8934,198,32929,220,21,25,220,4754,4532,3147,38867,2129,1979,816,3147,83,356,8934,198,32929,220,22,25,220,4754,4532,3147,39181,2129,1979,816,3147,65092,8934,198,4537,38945,198,198,27806,220,17,198,887,25,4086,806,198,9222,25,5576,198,669,55530,25,220,22365,2321,369,4086,806,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1719,220,16,19,337,2129,1979,816,3147,16,19,8934,198,32929,220,17,25,220,4754,4532,3147,1719,220,17,16,267,2129,1979,816,3147,17,16,8934,198,32929,220,18,25,220,4754,4532,3147,1719,220,17,23,337,2129,1979,816,3147,17,23,8934,198,32929,220,19,25,220,4754,4532,3147,1719,220,22,337,2129,1979,816,3147,22,8934,198,4537,38945,198,198,27806,220,18,198,887,25,2182,4956,198,9222,25,5576,198,669,55530,25,220,4199,1599,2756,1503,279,2182,6625,599,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,17,23,2129,1979,816,3147,17,23,8934,198,32929,220,17,25,220,4754,4532,3147,17,24,2129,1979,816,3147,17,24,8934,198,32929,220,18,25,220,4754,4532,3147,18,15,2129,1979,816,3147,18,15,8934,198,32929,220,19,25,220,4754,4532,3147,18,16,2129,1979,816,3147,18,16,8934,198,4537,38945,198,198,27806,220,19,198,887,25,88603,30250,400,198,9222,25,307,10644,198,669,55530,25,220,760,88603,2424,314,2756,5469,264,7483,466,264,7043,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,20,198,887,25,88603,3079,198,9222,25,5284,198,669,55530,25,220,4199,1599,2756,513,88603,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,6487,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,588,310,2250,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,32897,310,4590,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,77420,466,777,2129,1979,816,3147,18,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"first_weekday","type":"choice","question_span":[270,278],"option_spans":[[289,299],[305,315],[321,331],[337,347],[353,363],[369,380],[386,396]],"option_ids":["fri","mon","sat","sun","thu","tue","wed"],"logits":[-3.385533332824707,0.3842563331127167,-3.6725759506225586,-3.338017463684082,-3.481034994125366,-2.9808177947998047,-3.3202099800109863],"probs":[0.02014286071062088,0.8736147880554199,0.015116807073354721,0.021123066544532776,0.018308185040950775,0.030191650614142418,0.02150258608162403],"argmax_index":1,"argmax_id":"mon","top2_margin":0.8434231374412775,"near_tie":false,"gold":"mon","gold_correct":true},{"question_id":"circled","type":"choice","question_span":[418,424],"option_spans":[[435,450],[456,471],[477,492],[498,511]],"option_ids":["14","21","28","7"],"logits":[0.5825479030609131,-4.137327194213867,-4.5755438804626465,-4.419884204864502],"probs":[0.9790574312210083,0.008729560300707817,0.005632192362099886,0.006580814253538847],"argmax_index":0,"argmax_id":"14","top2_margin":0.9703278709203005,"near_tie":false,"gold":"14","gold_correct":true},{"question_id":"month_length","type":"choice","question_span":[533,542],"option_spans":[[553,565],[571,583],[589,601],[607,619]],"option_ids":["28","29","30","31"],"logits":[1.104544758796692,-3.2261195182800293,-3.6257870197296143,-3.769329071044922],"probs":[0.9712264537811279,0.012780179269611835,0.008569656871259212,0.0074237617664039135],"argmax_index":0,"argmax_id":"28","top2_margin":0.9584462745115161,"near_tie":false,"gold":"28","gold_correct":true},{"question_id":"shaded_weekend","type":"noul","question_span":[643,655],"option_spans":[[666,684],[690,708]],"option_ids":["true","false"],"logits":[-1.93511962890625,1.0548971891403198],"probs":[0.047878921031951904,0.9521211385726929],"argmax_index":1,"argmax_id":"false","top2_margin":0.904242217540741,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"shaded_count","type":"score","question_span":[730,736],"option_spans":[[747,757],[763,775],[781,793],[799,811]],"option_ids":["0","1","2","3"],"logits":[-3.4661896228790283,-4.010661602020264,-0.29667240381240845,-2.21461820602417],"probs":[0.03463566303253174,0.02009381353855133,0.824190080165863,0.12108045816421509],"argmax_index":2,"argmax_id":"2","top2_margin":0.703109622001648,"near_tie":false,"gold":"2","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"first_weekday":{"type":"choice","choice":"mon","confidence":0.8736,"probabilities":{"mon":0.8736,"tue":0.0302,"wed":0.0215,"thu":0.0183,"fri":0.0201,"sat":0.0151,"sun":0.0211}},"circled":{"type":"choice","choice":"14","confidence":0.9791,"probabilities":{"7":0.0066,"14":0.9791,"21":0.0087,"28":0.0056}},"month_length":{"type":"choice","choice":"28","confidence":0.9712,"probabilities":{"28":0.9712,"29":0.0128,"30":0.0086,"31":0.0074}},"shaded_weekend":{"type":"noul","noul":0.0479},"shaded_count":{"type":"score","score":2.0317,"confidence":0.8242,"legend":{"0":"none","1":"one to three","2":"four to six","3":"seven or more"},"probabilities":{"0":0.0346,"1":0.0201,"2":0.8242,"3":0.1211}}},"usage":{"input_tokens":833,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_i06","arm":"g256","set":"heldout","source":"heldout_image","tokens":628,"n_image_tokens":64,"state_tokens":16,"token_offset":36,"image_files":["ho_digits_960x240.png"],"image_size_in":[960,240],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"5faa81694bcef6d3c2bc6f56b144b556d391a5daa3a0523fa7698fd03b5deaf9","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,5920,3147,12615,22247,2129,7945,3147,10053,1092,279,22247,2940,4774,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,4688,46790,198,9222,25,5576,198,669,55530,25,220,3710,369,279,4688,15262,494,279,2047,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,15,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,16,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,17,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,18,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,19,2129,1979,816,3147,19,8934,198,32929,220,21,25,220,4754,4532,3147,20,2129,1979,816,3147,20,8934,198,32929,220,22,25,220,4754,4532,3147,21,2129,1979,816,3147,21,8934,198,32929,220,23,25,220,4754,4532,3147,22,2129,1979,816,3147,22,8934,198,32929,220,24,25,220,4754,4532,3147,23,2129,1979,816,3147,23,8934,198,32929,220,16,15,25,220,4754,4532,3147,24,2129,1979,816,3147,24,8934,198,4537,38945,198,198,27806,220,17,198,887,25,15262,3079,198,9222,25,5576,198,669,55530,25,220,4199,1599,17955,1503,279,2940,1420,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,17,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,18,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,32929,220,19,25,220,4754,4532,3147,77420,2129,1979,816,3147,22,8934,198,32929,220,20,25,220,4754,4532,3147,65254,2129,1979,816,3147,23,8934,198,4537,38945,198,198,27806,220,18,198,887,25,682,3343,1002,198,9222,25,307,10644,198,669,55530,25,220,760,15262,220,22,7701,383,279,2940,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,2438,11847,198,9222,25,307,10644,198,669,55530,25,220,760,1245,3444,15262,22294,383,264,2438,3870,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"third_digit","type":"choice","question_span":[144,153],"option_spans":[[164,174],[180,190],[196,206],[212,222],[228,238],[244,254],[260,270],[276,286],[292,302],[309,319]],"option_ids":["0","1","2","3","4","5","6","7","8","9"],"logits":[-2.797727108001709,-3.695978879928589,-1.924776315689087,-2.1009974479675293,-2.4233720302581787,-3.5489346981048584,-3.763709783554077,-3.734691858291626,-2.7976701259613037,1.2495546340942383],"probs":[0.014981632120907307,0.006101733539253473,0.03586549684405327,0.030070802196860313,0.021784091368317604,0.007068279664963484,0.00570214306935668,0.005870033986866474,0.01498248241841793,0.8575733304023743],"argmax_index":9,"argmax_id":"9","top2_margin":0.821707833558321,"near_tie":false,"gold":"9","gold_correct":true},{"question_id":"digit_count","type":"choice","question_span":[341,349],"option_spans":[[360,370],[376,386],[392,402],[408,418],[424,434]],"option_ids":["4","5","6","7","8"],"logits":[-1.8881171941757202,-0.21181395649909973,0.6551870107650757,-0.22375354170799255,-1.1355544328689575],"probs":[0.037775568664073944,0.20193901658058167,0.4805670380592346,0.19954228401184082,0.08017608523368835],"argmax_index":2,"argmax_id":"6","top2_margin":0.27862802147865295,"near_tie":false,"gold":"6","gold_correct":true},{"question_id":"has_seven","type":"noul","question_span":[458,467],"option_spans":[[478,496],[502,520]],"option_ids":["true","false"],"logits":[-2.5705060958862305,2.0326976776123047],"probs":[0.009920286014676094,0.9900797605514526],"argmax_index":1,"argmax_id":"false","top2_margin":0.9801594745367765,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"red_last","type":"noul","question_span":[543,553],"option_spans":[[564,582],[588,606]],"option_ids":["true","false"],"logits":[1.185051679611206,-2.428867816925049],"probs":[0.9737609624862671,0.02623898722231388],"argmax_index":0,"argmax_id":"true","top2_margin":0.9475219752639532,"near_tie":false,"gold":"true","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"third_digit":{"type":"choice","choice":"9","confidence":0.8576,"probabilities":{"0":0.015,"1":0.0061,"2":0.0359,"3":0.0301,"4":0.0218,"5":0.0071,"6":0.0057,"7":0.0059,"8":0.015,"9":0.8576}},"digit_count":{"type":"choice","choice":"6","confidence":0.4806,"probabilities":{"4":0.0378,"5":0.2019,"6":0.4806,"7":0.1995,"8":0.0802}},"has_seven":{"type":"noul","noul":0.0099},"red_last":{"type":"noul","noul":0.9738}},"usage":{"input_tokens":628,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_i06","arm":"g448","set":"heldout","source":"heldout_image","tokens":760,"n_image_tokens":196,"state_tokens":16,"token_offset":36,"image_files":["ho_digits_960x240.png"],"image_size_in":[960,240],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"71c862edc7427912d1b917abb7981fca56ed2eeaac6cae5b6d033e9c5b40081d","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,5920,3147,12615,22247,2129,7945,3147,10053,1092,279,22247,2940,4774,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,4688,46790,198,9222,25,5576,198,669,55530,25,220,3710,369,279,4688,15262,494,279,2047,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,15,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,16,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,17,2129,1979,816,3147,17,8934,198,32929,220,19,25,220,4754,4532,3147,18,2129,1979,816,3147,18,8934,198,32929,220,20,25,220,4754,4532,3147,19,2129,1979,816,3147,19,8934,198,32929,220,21,25,220,4754,4532,3147,20,2129,1979,816,3147,20,8934,198,32929,220,22,25,220,4754,4532,3147,21,2129,1979,816,3147,21,8934,198,32929,220,23,25,220,4754,4532,3147,22,2129,1979,816,3147,22,8934,198,32929,220,24,25,220,4754,4532,3147,23,2129,1979,816,3147,23,8934,198,32929,220,16,15,25,220,4754,4532,3147,24,2129,1979,816,3147,24,8934,198,4537,38945,198,198,27806,220,17,198,887,25,15262,3079,198,9222,25,5576,198,669,55530,25,220,4199,1599,17955,1503,279,2940,1420,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,17,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,18,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,32929,220,19,25,220,4754,4532,3147,77420,2129,1979,816,3147,22,8934,198,32929,220,20,25,220,4754,4532,3147,65254,2129,1979,816,3147,23,8934,198,4537,38945,198,198,27806,220,18,198,887,25,682,3343,1002,198,9222,25,307,10644,198,669,55530,25,220,760,15262,220,22,7701,383,279,2940,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,2438,11847,198,9222,25,307,10644,198,669,55530,25,220,760,1245,3444,15262,22294,383,264,2438,3870,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"third_digit","type":"choice","question_span":[276,285],"option_spans":[[296,306],[312,322],[328,338],[344,354],[360,370],[376,386],[392,402],[408,418],[424,434],[441,451]],"option_ids":["0","1","2","3","4","5","6","7","8","9"],"logits":[-3.2989723682403564,-3.8699638843536377,0.20333430171012878,-3.0692882537841797,-3.6547975540161133,-3.9343435764312744,-4.454713344573975,-4.170815467834473,-2.993717908859253,1.1441030502319336],"probs":[0.008040431886911392,0.004542562644928694,0.26687735319137573,0.010116488672792912,0.005633085500448942,0.0042593288235366344,0.002531321719288826,0.0033623476047068834,0.010910621844232082,0.6837264895439148],"argmax_index":9,"argmax_id":"9","top2_margin":0.41684913635253906,"near_tie":false,"gold":"9","gold_correct":true},{"question_id":"digit_count","type":"choice","question_span":[473,481],"option_spans":[[492,502],[508,518],[524,534],[540,550],[556,566]],"option_ids":["4","5","6","7","8"],"logits":[-1.1988990306854248,0.6828122138977051,-1.0834273099899292,-2.033604145050049,-2.3772172927856445],"probs":[0.1060565710067749,0.6962326765060425,0.11903819441795349,0.04602881520986557,0.03264378011226654],"argmax_index":1,"argmax_id":"5","top2_margin":0.577194482088089,"near_tie":false,"gold":"6","gold_correct":false},{"question_id":"has_seven","type":"noul","question_span":[590,599],"option_spans":[[610,628],[634,652]],"option_ids":["true","false"],"logits":[-2.8392419815063477,2.050280809402466],"probs":[0.007468811701983213,0.9925312399864197],"argmax_index":1,"argmax_id":"false","top2_margin":0.9850624282844365,"near_tie":false,"gold":"false","gold_correct":true},{"question_id":"red_last","type":"noul","question_span":[675,685],"option_spans":[[696,714],[720,738]],"option_ids":["true","false"],"logits":[1.1405154466629028,-2.702315092086792],"probs":[0.9790168404579163,0.020983118563890457],"argmax_index":0,"argmax_id":"true","top2_margin":0.9580337218940258,"near_tie":false,"gold":"true","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"third_digit":{"type":"choice","choice":"9","confidence":0.6837,"probabilities":{"0":0.008,"1":0.0045,"2":0.2669,"3":0.0101,"4":0.0056,"5":0.0043,"6":0.0025,"7":0.0034,"8":0.0109,"9":0.6837}},"digit_count":{"type":"choice","choice":"5","confidence":0.6962,"probabilities":{"4":0.1061,"5":0.6962,"6":0.119,"7":0.046,"8":0.0326}},"has_seven":{"type":"noul","noul":0.0075},"red_last":{"type":"noul","noul":0.979}},"usage":{"input_tokens":760,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_i07","arm":"g256","set":"heldout","source":"heldout_image","tokens":535,"n_image_tokens":64,"state_tokens":7,"token_offset":36,"image_files":["ho_receipt_480x840.png"],"image_size_in":[480,840],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"283f618d88ebd8335dbcad7e18e50a1adef157db5ee8574129936a1884a57d36","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,32,5813,8341,411,11617,20485,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,2702,198,9222,25,5576,198,669,55530,25,220,3710,2702,1503,279,20485,1420,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16,15,13,17,15,2129,1979,816,3147,16,15,13,17,15,8934,198,32929,220,17,25,220,4754,4532,3147,16,16,13,17,15,2129,1979,816,3147,16,16,13,17,15,8934,198,32929,220,18,25,220,4754,4532,3147,16,16,13,19,15,2129,1979,816,3147,16,16,13,19,15,8934,198,32929,220,19,25,220,4754,4532,3147,16,17,13,16,15,2129,1979,816,3147,16,17,13,16,15,8934,198,4537,38945,198,198,27806,220,17,198,887,25,6945,3584,198,9222,25,5576,198,669,55530,25,220,4199,557,279,3896,6945,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1311,3575,2129,1979,816,3147,4791,8934,198,32929,220,17,25,220,4754,4532,3147,623,8091,2129,1979,816,3147,39908,8934,198,32929,220,18,25,220,4754,4532,3147,2278,264,47035,2129,1979,816,3147,76307,8934,198,4537,38945,198,198,27806,220,18,198,887,25,534,3266,467,198,9222,25,5576,198,669,55530,25,220,22365,1455,2695,279,1379,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,59032,2129,1979,816,3147,45805,8934,198,32929,220,17,25,220,4754,4532,3147,31287,2129,1979,816,3147,1065,8934,198,32929,220,18,25,220,4754,4532,3147,31469,2129,1979,816,3147,86462,8934,198,32929,220,19,25,220,4754,4532,3147,6430,64,2129,1979,816,3147,78781,8934,198,4537,38945,198,198,27806,220,19,198,887,25,2222,3586,754,198,9222,25,307,10644,198,669,55530,25,220,760,20485,4774,2222,21893,1142,310,279,5813,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"total","type":"choice","question_span":[134,141],"option_spans":[[152,170],[176,194],[200,218],[224,242]],"option_ids":["10.20","11.20","11.40","12.10"],"logits":[-2.7824220657348633,1.8800336122512817,-3.0124642848968506,-4.004148006439209],"probs":[0.009260543622076511,0.9806525707244873,0.007357500027865171,0.00272927712649107],"argmax_index":1,"argmax_id":"11.20","top2_margin":0.9713920271024108,"near_tie":false,"gold":"11.20","gold_correct":true},{"question_id":"paid_by","type":"choice","question_span":[264,270],"option_spans":[[281,292],[298,309],[315,327]],"option_ids":["card","cash","voucher"],"logits":[0.672404408454895,-3.191225051879883,-4.612207889556885],"probs":[0.9746012091636658,0.020458512008190155,0.004940251354128122],"argmax_index":0,"argmax_id":"card","top2_margin":0.9541426971554756,"near_tie":false,"gold":"card","gold_correct":true},{"question_id":"priciest","type":"choice","question_span":[350,356],"option_spans":[[367,377],[383,393],[399,409],[415,426]],"option_ids":["cake","roll","soup","tea"],"logits":[-1.5414553880691528,-5.240172386169434,0.25859570503234863,-3.8987927436828613],"probs":[0.13948199152946472,0.003452914534136653,0.8438599705696106,0.013205030001699924],"argmax_index":2,"argmax_id":"soup","top2_margin":0.7043779790401459,"near_tie":false,"gold":"soup","gold_correct":true},{"question_id":"change_shown","type":"noul","question_span":[450,460],"option_spans":[[471,489],[495,513]],"option_ids":["true","false"],"logits":[-2.1705026626586914,1.126098394393921],"probs":[0.03568797558546066,0.9643120169639587],"argmax_index":1,"argmax_id":"false","top2_margin":0.9286240413784981,"near_tie":false,"gold":"false","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"total":{"type":"choice","choice":"11.20","confidence":0.9807,"probabilities":{"10.20":0.0093,"11.20":0.9807,"11.40":0.0074,"12.10":0.0027}},"paid_by":{"type":"choice","choice":"card","confidence":0.9746,"probabilities":{"cash":0.0205,"card":0.9746,"voucher":0.0049}},"priciest":{"type":"choice","choice":"soup","confidence":0.8439,"probabilities":{"soup":0.8439,"roll":0.0035,"tea":0.0132,"cake":0.1395}},"change_shown":{"type":"noul","noul":0.0357}},"usage":{"input_tokens":535,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_i07","arm":"g448","set":"heldout","source":"heldout_image","tokens":667,"n_image_tokens":196,"state_tokens":7,"token_offset":36,"image_files":["ho_receipt_480x840.png"],"image_size_in":[480,840],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"06a7b5db6727f4c4b606a3ee04fd2703269fd5ce0f7a5748c9e7dd70048f5bc7","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,32,5813,8341,411,11617,20485,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,2702,198,9222,25,5576,198,669,55530,25,220,3710,2702,1503,279,20485,1420,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,16,15,13,17,15,2129,1979,816,3147,16,15,13,17,15,8934,198,32929,220,17,25,220,4754,4532,3147,16,16,13,17,15,2129,1979,816,3147,16,16,13,17,15,8934,198,32929,220,18,25,220,4754,4532,3147,16,16,13,19,15,2129,1979,816,3147,16,16,13,19,15,8934,198,32929,220,19,25,220,4754,4532,3147,16,17,13,16,15,2129,1979,816,3147,16,17,13,16,15,8934,198,4537,38945,198,198,27806,220,17,198,887,25,6945,3584,198,9222,25,5576,198,669,55530,25,220,4199,557,279,3896,6945,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1311,3575,2129,1979,816,3147,4791,8934,198,32929,220,17,25,220,4754,4532,3147,623,8091,2129,1979,816,3147,39908,8934,198,32929,220,18,25,220,4754,4532,3147,2278,264,47035,2129,1979,816,3147,76307,8934,198,4537,38945,198,198,27806,220,18,198,887,25,534,3266,467,198,9222,25,5576,198,669,55530,25,220,22365,1455,2695,279,1379,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,59032,2129,1979,816,3147,45805,8934,198,32929,220,17,25,220,4754,4532,3147,31287,2129,1979,816,3147,1065,8934,198,32929,220,18,25,220,4754,4532,3147,31469,2129,1979,816,3147,86462,8934,198,32929,220,19,25,220,4754,4532,3147,6430,64,2129,1979,816,3147,78781,8934,198,4537,38945,198,198,27806,220,19,198,887,25,2222,3586,754,198,9222,25,307,10644,198,669,55530,25,220,760,20485,4774,2222,21893,1142,310,279,5813,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"total","type":"choice","question_span":[266,273],"option_spans":[[284,302],[308,326],[332,350],[356,374]],"option_ids":["10.20","11.20","11.40","12.10"],"logits":[-2.6150717735290527,1.723196029663086,-2.96506929397583,-3.891719341278076],"probs":[0.012729376554489136,0.9747492074966431,0.008970261551439762,0.0035511264577507973],"argmax_index":1,"argmax_id":"11.20","top2_margin":0.9620198309421539,"near_tie":false,"gold":"11.20","gold_correct":true},{"question_id":"paid_by","type":"choice","question_span":[396,402],"option_spans":[[413,424],[430,441],[447,459]],"option_ids":["card","cash","voucher"],"logits":[0.8176729083061218,-3.1089515686035156,-4.305391311645508],"probs":[0.9749745726585388,0.019216837361454964,0.005808645393699408],"argmax_index":0,"argmax_id":"card","top2_margin":0.9557577352970839,"near_tie":false,"gold":"card","gold_correct":true},{"question_id":"priciest","type":"choice","question_span":[482,488],"option_spans":[[499,509],[515,525],[531,541],[547,558]],"option_ids":["cake","roll","soup","tea"],"logits":[-2.0296225547790527,-5.3733720779418945,0.23168747127056122,-3.696061611175537],"probs":[0.09242250770330429,0.0032629151828587055,0.8868542313575745,0.017460331320762634],"argmax_index":2,"argmax_id":"soup","top2_margin":0.7944317236542702,"near_tie":false,"gold":"soup","gold_correct":true},{"question_id":"change_shown","type":"noul","question_span":[582,592],"option_spans":[[603,621],[627,645]],"option_ids":["true","false"],"logits":[-2.2617058753967285,0.8138405084609985],"probs":[0.044127292931079865,0.9558727145195007],"argmax_index":1,"argmax_id":"false","top2_margin":0.9117454215884209,"near_tie":false,"gold":"false","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"total":{"type":"choice","choice":"11.20","confidence":0.9747,"probabilities":{"10.20":0.0127,"11.20":0.9747,"11.40":0.009,"12.10":0.0036}},"paid_by":{"type":"choice","choice":"card","confidence":0.975,"probabilities":{"cash":0.0192,"card":0.975,"voucher":0.0058}},"priciest":{"type":"choice","choice":"soup","confidence":0.8869,"probabilities":{"soup":0.8869,"roll":0.0033,"tea":0.0175,"cake":0.0924}},"change_shown":{"type":"noul","noul":0.0441}},"usage":{"input_tokens":667,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_i08","arm":"g256","set":"heldout","source":"heldout_image","tokens":581,"n_image_tokens":64,"state_tokens":12,"token_offset":36,"image_files":["ho_floorplan_1200x900.png"],"image_size_in":[1200,900],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"20e5bf98afda0f1b44b404c140331cc37af376e1bd8fb3769e9fdd7b1b00adf4","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,7945,3147,47086,279,4998,6576,303,279,12945,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,8276,14242,198,9222,25,5576,198,669,55530,25,220,22365,6725,3433,682,279,7526,6220,2982,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1537,9798,56,2129,1979,816,3147,69336,8934,198,32929,220,17,25,220,4754,4532,3147,7227,82886,2129,1979,816,3147,59183,8934,198,32929,220,18,25,220,4754,4532,3147,26200,5314,2129,1979,816,3147,25668,8934,198,32929,220,19,25,220,4754,4532,3147,42495,2129,1979,816,3147,4173,8934,198,32929,220,20,25,220,4754,4532,3147,24652,2129,1979,816,3147,23277,8934,198,4537,38945,198,198,27806,220,17,198,887,25,19204,198,9222,25,5576,198,669,55530,25,220,25741,864,3433,1503,279,4732,5817,1724,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1537,9798,56,2129,1979,816,3147,69336,8934,198,32929,220,17,25,220,4754,4532,3147,7227,82886,2129,1979,816,3147,59183,8934,198,32929,220,18,25,220,4754,4532,3147,26200,5314,2129,1979,816,3147,25668,8934,198,32929,220,19,25,220,4754,4532,3147,42495,2129,1979,816,3147,4173,8934,198,32929,220,20,25,220,4754,4532,3147,24652,2129,1979,816,3147,23277,8934,198,4537,38945,198,198,27806,220,18,198,887,25,25700,14380,49024,198,9222,25,307,10644,198,669,55530,25,220,760,35433,321,279,3436,4191,264,6784,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,3433,3079,198,9222,25,5576,198,669,55530,25,220,4199,1599,6725,12258,1503,279,3019,1420,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,17,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,18,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,19,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,32929,220,20,25,220,4754,4532,3147,77420,2129,1979,816,3147,22,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"biggest_space","type":"choice","question_span":[140,149],"option_spans":[[160,172],[178,189],[195,206],[212,222],[228,238]],"option_ids":["lobby","meeting","office","store","wc"],"logits":[-3.1571807861328125,-3.3978464603424072,-0.5657036900520325,-3.511955738067627,-4.632758617401123],"probs":[0.0622449554502964,0.04893103614449501,0.8309375643730164,0.04365433380007744,0.014232099056243896],"argmax_index":2,"argmax_id":"office","top2_margin":0.76869260892272,"near_tie":false,"gold":"office","gold_correct":true},{"question_id":"entrance","type":"choice","question_span":[259,268],"option_spans":[[279,291],[297,308],[314,325],[331,341],[347,357]],"option_ids":["lobby","meeting","office","store","wc"],"logits":[-0.39451950788497925,-5.077653408050537,-4.817508220672607,-5.093421936035156,-4.852109909057617],"probs":[0.9597446918487549,0.008877617307007313,0.011515321210026741,0.008738728240132332,0.011123686097562313],"argmax_index":0,"argmax_id":"lobby","top2_margin":0.9482293706387281,"near_tie":false,"gold":"lobby","gold_correct":true},{"question_id":"wc_store_wall","type":"noul","question_span":[381,390],"option_spans":[[401,419],[425,443]],"option_ids":["true","false"],"logits":[-0.6502922177314758,-3.4860048294067383],"probs":[0.9445754289627075,0.055424563586711884],"argmax_index":0,"argmax_id":"true","top2_margin":0.8891508653759956,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"space_count","type":"choice","question_span":[465,474],"option_spans":[[485,495],[501,511],[517,527],[533,543],[549,559]],"option_ids":["3","4","5","6","7"],"logits":[-4.252464771270752,-2.548898458480835,-0.5014726519584656,-2.919556140899658,-4.158664703369141],"probs":[0.018536647781729698,0.10183114558458328,0.7889806628227234,0.07029203325510025,0.020359544083476067],"argmax_index":2,"argmax_id":"5","top2_margin":0.6871495172381401,"near_tie":false,"gold":"5","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"biggest_space":{"type":"choice","choice":"office","confidence":0.8309,"probabilities":{"meeting":0.0489,"office":0.8309,"lobby":0.0622,"wc":0.0142,"store":0.0437}},"entrance":{"type":"choice","choice":"lobby","confidence":0.9597,"probabilities":{"meeting":0.0089,"office":0.0115,"lobby":0.9597,"wc":0.0111,"store":0.0087}},"wc_store_wall":{"type":"noul","noul":0.9446},"space_count":{"type":"choice","choice":"5","confidence":0.789,"probabilities":{"3":0.0185,"4":0.1018,"5":0.789,"6":0.0703,"7":0.0204}}},"usage":{"input_tokens":581,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_i08","arm":"g448","set":"heldout","source":"heldout_image","tokens":713,"n_image_tokens":196,"state_tokens":12,"token_offset":36,"image_files":["ho_floorplan_1200x900.png"],"image_size_in":[1200,900],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"53532566979fa5e91a3a5a9081acd9bca639bc80b86f76fa60e4393e84828116","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,7945,3147,47086,279,4998,6576,303,279,12945,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,8276,14242,198,9222,25,5576,198,669,55530,25,220,22365,6725,3433,682,279,7526,6220,2982,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1537,9798,56,2129,1979,816,3147,69336,8934,198,32929,220,17,25,220,4754,4532,3147,7227,82886,2129,1979,816,3147,59183,8934,198,32929,220,18,25,220,4754,4532,3147,26200,5314,2129,1979,816,3147,25668,8934,198,32929,220,19,25,220,4754,4532,3147,42495,2129,1979,816,3147,4173,8934,198,32929,220,20,25,220,4754,4532,3147,24652,2129,1979,816,3147,23277,8934,198,4537,38945,198,198,27806,220,17,198,887,25,19204,198,9222,25,5576,198,669,55530,25,220,25741,864,3433,1503,279,4732,5817,1724,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1537,9798,56,2129,1979,816,3147,69336,8934,198,32929,220,17,25,220,4754,4532,3147,7227,82886,2129,1979,816,3147,59183,8934,198,32929,220,18,25,220,4754,4532,3147,26200,5314,2129,1979,816,3147,25668,8934,198,32929,220,19,25,220,4754,4532,3147,42495,2129,1979,816,3147,4173,8934,198,32929,220,20,25,220,4754,4532,3147,24652,2129,1979,816,3147,23277,8934,198,4537,38945,198,198,27806,220,18,198,887,25,25700,14380,49024,198,9222,25,307,10644,198,669,55530,25,220,760,35433,321,279,3436,4191,264,6784,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,3433,3079,198,9222,25,5576,198,669,55530,25,220,4199,1599,6725,12258,1503,279,3019,1420,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,26952,2129,1979,816,3147,18,8934,198,32929,220,17,25,220,4754,4532,3147,32897,2129,1979,816,3147,19,8934,198,32929,220,18,25,220,4754,4532,3147,50897,2129,1979,816,3147,20,8934,198,32929,220,19,25,220,4754,4532,3147,48671,2129,1979,816,3147,21,8934,198,32929,220,20,25,220,4754,4532,3147,77420,2129,1979,816,3147,22,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"biggest_space","type":"choice","question_span":[272,281],"option_spans":[[292,304],[310,321],[327,338],[344,354],[360,370]],"option_ids":["lobby","meeting","office","store","wc"],"logits":[-3.269683837890625,-3.4861958026885986,-0.635803759098053,-3.6718640327453613,-4.7130279541015625],"probs":[0.06010328233242035,0.04840255156159401,0.8371008634567261,0.040200695395469666,0.014192594215273857],"argmax_index":2,"argmax_id":"office","top2_margin":0.7769975811243057,"near_tie":false,"gold":"office","gold_correct":true},{"question_id":"entrance","type":"choice","question_span":[391,400],"option_spans":[[411,423],[429,440],[446,457],[463,473],[479,489]],"option_ids":["lobby","meeting","office","store","wc"],"logits":[-0.48431360721588135,-5.142092227935791,-4.880463123321533,-5.16994571685791,-4.872228622436523],"probs":[0.9583448171615601,0.009092309512197971,0.011811315082013607,0.00884255114942789,0.011908977292478085],"argmax_index":0,"argmax_id":"lobby","top2_margin":0.946435839869082,"near_tie":false,"gold":"lobby","gold_correct":true},{"question_id":"wc_store_wall","type":"noul","question_span":[513,522],"option_spans":[[533,551],[557,575]],"option_ids":["true","false"],"logits":[-0.9848035573959351,-3.954789161682129],"probs":[0.9511995315551758,0.04880039021372795],"argmax_index":0,"argmax_id":"true","top2_margin":0.9023991413414478,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"space_count","type":"choice","question_span":[597,606],"option_spans":[[617,627],[633,643],[649,659],[665,675],[681,691]],"option_ids":["3","4","5","6","7"],"logits":[-4.360629558563232,-2.9317216873168945,-0.49320346117019653,-2.9589755535125732,-4.280754089355469],"probs":[0.017200350761413574,0.07179663330316544,0.8225060701370239,0.06986632198095322,0.01863059587776661],"argmax_index":2,"argmax_id":"5","top2_margin":0.7507094368338585,"near_tie":false,"gold":"5","gold_correct":true}],"systemone_response":{"model":"clef-flash","answers":{"biggest_space":{"type":"choice","choice":"office","confidence":0.8371,"probabilities":{"meeting":0.0484,"office":0.8371,"lobby":0.0601,"wc":0.0142,"store":0.0402}},"entrance":{"type":"choice","choice":"lobby","confidence":0.9583,"probabilities":{"meeting":0.0091,"office":0.0118,"lobby":0.9583,"wc":0.0119,"store":0.0088}},"wc_store_wall":{"type":"noul","noul":0.9512},"space_count":{"type":"choice","choice":"5","confidence":0.8225,"probabilities":{"3":0.0172,"4":0.0718,"5":0.8225,"6":0.0699,"7":0.0186}}},"usage":{"input_tokens":713,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_p01","arm":"g256","set":"heldout","source":"heldout_photo","tokens":634,"n_image_tokens":64,"state_tokens":10,"token_offset":36,"image_files":["ho_photo_lighthouse.png"],"image_size_in":[960,960],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"70d99abd72d8ccc1fe3ceaa1381ece1c86328fc34b1e6e4e0b4aa8fcdb040f1f","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,34080,1732,883,279,4599,6625,303,279,6588,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,27056,4322,198,9222,25,307,10644,198,669,55530,25,220,760,27056,506,279,1844,314,279,20618,369,46325,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,854,3456,16271,198,9222,25,5576,198,669,55530,25,220,4350,557,279,6345,1379,4222,4284,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1860,264,19177,11,888,3701,1834,2129,1979,816,3147,33426,16271,8934,198,32929,220,17,25,220,4754,4532,3147,42138,4937,1247,303,9613,6797,2129,1979,816,3147,15632,1247,8934,198,32929,220,18,25,220,4754,4532,3147,623,279,5956,314,264,6121,3603,2129,1979,816,3147,9011,8934,198,32929,220,19,25,220,4754,4532,3147,1597,89687,466,37119,11,440,279,12515,1990,26170,12647,2129,1979,816,3147,15103,33743,8934,198,4537,38945,198,198,27806,220,18,198,887,25,9117,41815,474,198,9222,25,307,10644,198,669,55530,25,220,760,9117,628,381,3753,4661,279,20618,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,20618,54370,198,9222,25,5576,198,669,55530,25,220,3710,12106,369,279,20618,23235,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,14004,2129,1979,816,3147,11124,8934,198,32929,220,17,25,220,4754,4532,3147,5844,321,4022,20254,2129,1979,816,3147,1114,42954,8934,198,32929,220,18,25,220,4754,4532,3147,13672,2129,1979,816,3147,5603,8934,198,32929,220,19,25,220,4754,4532,3147,46110,2129,1979,816,3147,26964,8934,198,4537,38945,198,198,27806,220,20,198,887,25,34607,22967,198,9222,25,5284,198,669,55530,25,220,4199,8434,1503,279,1925,1353,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,258,264,12655,6092,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,266,279,6607,314,264,13721,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,57901,639,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"lamp_on","type":"noul","question_span":[139,150],"option_spans":[[161,179],[185,203]],"option_ids":["true","false"],"logits":[-1.9387848377227783,-4.749064922332764],"probs":[0.9432287812232971,0.0567711777985096],"argmax_index":0,"argmax_id":"true","top2_margin":0.8864576034247875,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"time_of_day","type":"choice","question_span":[226,234],"option_spans":[[245,262],[268,284],[290,306],[312,333]],"option_ids":["grey_day","midday","night","twilight"],"logits":[-4.876680850982666,-4.726728439331055,-3.1574995517730713,0.11803555488586426],"probs":[0.006436130963265896,0.007477364502847195,0.03591334819793701,0.9501730799674988],"argmax_index":3,"argmax_id":"twilight","top2_margin":0.9142597317695618,"near_tie":false,"gold":"twilight","gold_correct":true},{"question_id":"sea_behind","type":"noul","question_span":[357,366],"option_spans":[[377,395],[401,419]],"option_ids":["true","false"],"logits":[-2.0789542198181152,-4.6626505851745605],"probs":[0.9298049211502075,0.07019509375095367],"argmax_index":0,"argmax_id":"true","top2_margin":0.8596098273992538,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"tower_colour","type":"choice","question_span":[441,448],"option_spans":[[459,469],[475,489],[495,505],[511,521]],"option_ids":["black","red_white","white","yellow"],"logits":[-5.275908470153809,-4.541167259216309,-1.0301802158355713,-5.270671367645264],"probs":[0.01353238895535469,0.028214257210493088,0.9446499943733215,0.013603451661765575],"argmax_index":2,"argmax_id":"white","top2_margin":0.9164357371628284,"near_tie":false,"gold":"white","gold_correct":true},{"question_id":"remoteness","type":"score","question_span":[543,550],"option_spans":[[561,574],[580,595],[601,612]],"option_ids":["0","1","2"],"logits":[-4.654239654541016,-3.974550485610962,-1.6135694980621338],"probs":[0.041854098439216614,0.08258920162916183,0.8755567073822021],"argmax_index":2,"argmax_id":"2","top2_margin":0.7929675057530403,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"lamp_on":{"type":"noul","noul":0.9432},"time_of_day":{"type":"choice","choice":"twilight","confidence":0.9502,"probabilities":{"midday":0.0075,"twilight":0.9502,"night":0.0359,"grey_day":0.0064}},"sea_behind":{"type":"noul","noul":0.9298},"tower_colour":{"type":"choice","choice":"white","confidence":0.9446,"probabilities":{"white":0.9446,"red_white":0.0282,"black":0.0135,"yellow":0.0136}},"remoteness":{"type":"score","score":1.8337,"confidence":0.8756,"legend":{"0":"in a busy town","1":"at the edge of a village","2":"isolated"},"probabilities":{"0":0.0419,"1":0.0826,"2":0.8756}}},"usage":{"input_tokens":634,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_p01","arm":"g448","set":"heldout","source":"heldout_photo","tokens":766,"n_image_tokens":196,"state_tokens":10,"token_offset":36,"image_files":["ho_photo_lighthouse.png"],"image_size_in":[960,960],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"c00abf38a111b0af44102b7d68049246ffa488925da72f3bdce3d3999e2d7dab","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,34080,1732,883,279,4599,6625,303,279,6588,13,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,27056,4322,198,9222,25,307,10644,198,669,55530,25,220,760,27056,506,279,1844,314,279,20618,369,46325,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,17,198,887,25,854,3456,16271,198,9222,25,5576,198,669,55530,25,220,4350,557,279,6345,1379,4222,4284,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,1860,264,19177,11,888,3701,1834,2129,1979,816,3147,33426,16271,8934,198,32929,220,17,25,220,4754,4532,3147,42138,4937,1247,303,9613,6797,2129,1979,816,3147,15632,1247,8934,198,32929,220,18,25,220,4754,4532,3147,623,279,5956,314,264,6121,3603,2129,1979,816,3147,9011,8934,198,32929,220,19,25,220,4754,4532,3147,1597,89687,466,37119,11,440,279,12515,1990,26170,12647,2129,1979,816,3147,15103,33743,8934,198,4537,38945,198,198,27806,220,18,198,887,25,9117,41815,474,198,9222,25,307,10644,198,669,55530,25,220,760,9117,628,381,3753,4661,279,20618,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,19,198,887,25,20618,54370,198,9222,25,5576,198,669,55530,25,220,3710,12106,369,279,20618,23235,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,14004,2129,1979,816,3147,11124,8934,198,32929,220,17,25,220,4754,4532,3147,5844,321,4022,20254,2129,1979,816,3147,1114,42954,8934,198,32929,220,18,25,220,4754,4532,3147,13672,2129,1979,816,3147,5603,8934,198,32929,220,19,25,220,4754,4532,3147,46110,2129,1979,816,3147,26964,8934,198,4537,38945,198,198,27806,220,20,198,887,25,34607,22967,198,9222,25,5284,198,669,55530,25,220,4199,8434,1503,279,1925,1353,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,258,264,12655,6092,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,266,279,6607,314,264,13721,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,57901,639,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"lamp_on","type":"noul","question_span":[271,282],"option_spans":[[293,311],[317,335]],"option_ids":["true","false"],"logits":[-1.9242303371429443,-4.81666374206543],"probs":[0.9474711418151855,0.052528876811265945],"argmax_index":0,"argmax_id":"true","top2_margin":0.8949422650039196,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"time_of_day","type":"choice","question_span":[358,366],"option_spans":[[377,394],[400,416],[422,438],[444,465]],"option_ids":["grey_day","midday","night","twilight"],"logits":[-4.9305548667907715,-4.81920862197876,-3.5496044158935547,0.1001175045967102],"probs":[0.006284076254814863,0.00702422671020031,0.025002341717481613,0.961689293384552],"argmax_index":3,"argmax_id":"twilight","top2_margin":0.9366869516670704,"near_tie":false,"gold":"twilight","gold_correct":true},{"question_id":"sea_behind","type":"noul","question_span":[489,498],"option_spans":[[509,527],[533,551]],"option_ids":["true","false"],"logits":[-2.133479595184326,-4.700721740722656],"probs":[0.9287233352661133,0.07127664983272552],"argmax_index":0,"argmax_id":"true","top2_margin":0.8574466854333878,"near_tie":false,"gold":"true","gold_correct":true},{"question_id":"tower_colour","type":"choice","question_span":[573,580],"option_spans":[[591,601],[607,621],[627,637],[643,653]],"option_ids":["black","red_white","white","yellow"],"logits":[-5.283841609954834,-4.674393177032471,-1.0173951387405396,-5.26944637298584],"probs":[0.013311704620718956,0.024485770612955093,0.9486978054046631,0.013504715636372566],"argmax_index":2,"argmax_id":"white","top2_margin":0.924212034791708,"near_tie":false,"gold":"white","gold_correct":true},{"question_id":"remoteness","type":"score","question_span":[675,682],"option_spans":[[693,706],[712,727],[733,744]],"option_ids":["0","1","2"],"logits":[-4.723003387451172,-4.078141689300537,-1.6123697757720947],"probs":[0.03946177661418915,0.07520323246717453,0.8853350281715393],"argmax_index":2,"argmax_id":"2","top2_margin":0.8101317957043648,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"lamp_on":{"type":"noul","noul":0.9475},"time_of_day":{"type":"choice","choice":"twilight","confidence":0.9617,"probabilities":{"midday":0.007,"twilight":0.9617,"night":0.025,"grey_day":0.0063}},"sea_behind":{"type":"noul","noul":0.9287},"tower_colour":{"type":"choice","choice":"white","confidence":0.9487,"probabilities":{"white":0.9487,"red_white":0.0245,"black":0.0133,"yellow":0.0135}},"remoteness":{"type":"score","score":1.8459,"confidence":0.8853,"legend":{"0":"in a busy town","1":"at the edge of a village","2":"isolated"},"probabilities":{"0":0.0395,"1":0.0752,"2":0.8853}}},"usage":{"input_tokens":766,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_p02","arm":"g256","set":"heldout","source":"heldout_photo","tokens":620,"n_image_tokens":64,"state_tokens":10,"token_offset":36,"image_files":["ho_photo_sunflowers.png"],"image_size_in":[960,1280],"image_size_fed":[256,256],"resized_hw":[256,256],"grid_thw":[1,16,16],"merged_hw":[8,8],"rope_shift_amount":56,"pixel_values_shape":[256,1536],"pixel_values_sha256":"f8d5a8894da942ec64ffb0e5983254959543f41b5aa8e6245c697f2bae707961","image_embeds_shape":[64,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,7945,3147,72240,279,78438,10462,40192,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,6541,278,54370,198,9222,25,5576,198,669,55530,25,220,3710,12106,513,279,92334,314,279,18698,303,3929,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,71978,2129,1979,816,3147,54613,8934,198,32929,220,17,25,220,4754,4532,3147,5844,2129,1979,816,3147,1114,8934,198,32929,220,18,25,220,4754,4532,3147,13672,2129,1979,816,3147,5603,8934,198,32929,220,19,25,220,4754,4532,3147,46110,2129,1979,816,3147,26964,8934,198,4537,38945,198,198,27806,220,17,198,887,25,25158,198,9222,25,307,10644,198,669,55530,25,220,2014,25158,369,11362,383,799,314,279,18698,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,11911,54370,198,9222,25,5576,198,669,55530,25,220,3710,12106,369,279,5956,314,279,7526,21655,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,10025,2129,1979,816,3147,11855,8934,198,32929,220,17,25,220,4754,4532,3147,46776,466,6121,18016,2129,1979,816,3147,62284,8934,198,32929,220,18,25,220,4754,4532,3147,18978,2129,1979,816,3147,12865,8934,198,32929,220,19,25,220,4754,4532,3147,13672,2129,1979,816,3147,5603,8934,198,4537,38945,198,198,27806,220,19,198,887,25,6047,198,9222,25,5576,198,669,55530,25,220,8791,513,279,18698,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,2575,26422,4082,10583,321,975,10462,2129,1979,816,3147,403,26422,8934,198,32929,220,17,25,220,4754,4532,3147,623,12104,9002,2129,1979,816,3147,88104,8934,198,32929,220,18,25,220,4754,4532,3147,623,11456,2129,1979,816,3147,71944,8934,198,32929,220,19,25,220,4754,4532,3147,623,264,89192,52239,2129,1979,816,3147,85,507,8934,198,4537,38945,198,198,27806,220,20,198,887,25,16940,198,9222,25,5284,198,669,55530,25,220,4199,27044,369,279,51930,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,94834,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,77518,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"petal_colour","type":"choice","question_span":[139,150],"option_spans":[[161,171],[177,187],[193,203],[209,219]],"option_ids":["purple","red","white","yellow"],"logits":[-5.134452819824219,-5.118758201599121,-5.202085971832275,-0.49137184023857117],"probs":[0.009362042881548405,0.009510135278105736,0.008749795146286488,0.9723780155181885],"argmax_index":3,"argmax_id":"yellow","top2_margin":0.9628678802400827,"near_tie":false,"gold":"yellow","gold_correct":true},{"question_id":"insect","type":"noul","question_span":[241,251],"option_spans":[[262,280],[286,304]],"option_ids":["true","false"],"logits":[-2.3114304542541504,1.117574691772461],"probs":[0.03140117600560188,0.9685987830162048],"argmax_index":1,"argmax_id":"false","top2_margin":0.937197607010603,"near_tie":false,"gold":"true","gold_correct":false},{"question_id":"centre_colour","type":"choice","question_span":[326,336],"option_spans":[[347,357],[363,376],[382,392],[398,408]],"option_ids":["blue","brown","green","white"],"logits":[-4.920816898345947,-0.17515140771865845,-4.441528797149658,-4.714609622955322],"probs":[0.008408427238464355,0.9676785469055176,0.013578973710536957,0.010334023274481297],"argmax_index":1,"argmax_id":"brown","top2_margin":0.9540995731949806,"near_tie":false,"gold":"brown","gold_correct":true},{"question_id":"setting","type":"choice","question_span":[429,434],"option_spans":[[445,462],[468,480],[486,497],[503,517]],"option_ids":["outdoors","sand","snow","vase"],"logits":[-0.03647449612617493,-4.91420841217041,-4.9091081619262695,-4.604005813598633],"probs":[0.9749904274940491,0.007423821836709976,0.007461781147867441,0.010123875923454762],"argmax_index":0,"argmax_id":"outdoors","top2_margin":0.9648665515705943,"near_tie":false,"gold":"outdoors","gold_correct":true},{"question_id":"density","type":"score","question_span":[538,544],"option_spans":[[555,565],[571,582],[588,598]],"option_ids":["0","1","2"],"logits":[-2.998471260070801,-1.566981315612793,-0.7220994234085083],"probs":[0.066996268928051,0.2803747057914734,0.652629017829895],"argmax_index":2,"argmax_id":"2","top2_margin":0.37225431203842163,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"petal_colour":{"type":"choice","choice":"yellow","confidence":0.9724,"probabilities":{"yellow":0.9724,"white":0.0087,"purple":0.0094,"red":0.0095}},"insect":{"type":"noul","noul":0.0314},"centre_colour":{"type":"choice","choice":"brown","confidence":0.9677,"probabilities":{"brown":0.9677,"green":0.0136,"blue":0.0084,"white":0.0103}},"setting":{"type":"choice","choice":"outdoors","confidence":0.975,"probabilities":{"vase":0.0101,"outdoors":0.975,"sand":0.0074,"snow":0.0075}},"density":{"type":"score","score":1.5856,"confidence":0.6526,"legend":{"0":"sparse","1":"moderate","2":"dense"},"probabilities":{"0":0.067,"1":0.2804,"2":0.6526}}},"usage":{"input_tokens":620,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}},{"id":"ho_p02","arm":"g448","set":"heldout","source":"heldout_photo","tokens":752,"n_image_tokens":196,"state_tokens":10,"token_offset":36,"image_files":["ho_photo_sunflowers.png"],"image_size_in":[960,1280],"image_size_fed":[448,448],"resized_hw":[448,448],"grid_thw":[1,28,28],"merged_hw":[14,14],"rope_shift_amount":182,"pixel_values_shape":[784,1536],"pixel_values_sha256":"cff04b7ecda76a23808628b8b08346be0a7ec9ea0219a4844212d9c239876150","image_embeds_shape":[196,4096],"ids":[248045,8678,198,4274,279,4434,1528,321,10485,13,95462,1396,2002,50407,13,8618,4087,1902,381,6681,799,314,421,2002,579,5252,2519,13,248046,198,248045,846,198,23852,25,198,248053,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248056,248054,198,4754,7945,3147,72240,279,78438,10462,40192,1149,92,271,3423,34651,426,32349,25,198,198,27806,220,16,198,887,25,6541,278,54370,198,9222,25,5576,198,669,55530,25,220,3710,12106,513,279,92334,314,279,18698,303,3929,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,71978,2129,1979,816,3147,54613,8934,198,32929,220,17,25,220,4754,4532,3147,5844,2129,1979,816,3147,1114,8934,198,32929,220,18,25,220,4754,4532,3147,13672,2129,1979,816,3147,5603,8934,198,32929,220,19,25,220,4754,4532,3147,46110,2129,1979,816,3147,26964,8934,198,4537,38945,198,198,27806,220,17,198,887,25,25158,198,9222,25,307,10644,198,669,55530,25,220,2014,25158,369,11362,383,799,314,279,18698,13,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,760,38749,369,804,466,279,4087,369,9542,46292,1979,816,3147,1802,8934,198,32929,220,17,25,220,4754,4532,3147,760,38749,369,867,466,279,4087,369,874,46292,1979,816,3147,3721,8934,198,4537,38945,198,198,27806,220,18,198,887,25,11911,54370,198,9222,25,5576,198,669,55530,25,220,3710,12106,369,279,5956,314,279,7526,21655,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,10025,2129,1979,816,3147,11855,8934,198,32929,220,17,25,220,4754,4532,3147,46776,466,6121,18016,2129,1979,816,3147,62284,8934,198,32929,220,18,25,220,4754,4532,3147,18978,2129,1979,816,3147,12865,8934,198,32929,220,19,25,220,4754,4532,3147,13672,2129,1979,816,3147,5603,8934,198,4537,38945,198,198,27806,220,19,198,887,25,6047,198,9222,25,5576,198,669,55530,25,220,8791,513,279,18698,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,2575,26422,4082,10583,321,975,10462,2129,1979,816,3147,403,26422,8934,198,32929,220,17,25,220,4754,4532,3147,623,12104,9002,2129,1979,816,3147,88104,8934,198,32929,220,18,25,220,4754,4532,3147,623,11456,2129,1979,816,3147,71944,8934,198,32929,220,19,25,220,4754,4532,3147,623,264,89192,52239,2129,1979,816,3147,85,507,8934,198,4537,38945,198,198,27806,220,20,198,887,25,16940,198,9222,25,5284,198,669,55530,25,220,4199,27044,369,279,51930,30,198,51535,1425,49555,25,198,32929,220,16,25,220,4754,4532,3147,94834,2129,1979,816,3147,15,8934,198,32929,220,17,25,220,4754,4532,3147,2508,56684,2129,1979,816,3147,16,8934,198,32929,220,18,25,220,4754,4532,3147,77518,2129,1979,816,3147,17,8934,198,4537,38945,198,198,248046,198,248045,74455,198,248068,271,248069,271,25310,3118,7295,34651,41402,23451,50,25],"in_decoder_gates":true,"questions":[{"question_id":"petal_colour","type":"choice","question_span":[271,282],"option_spans":[[293,303],[309,319],[325,335],[341,351]],"option_ids":["purple","red","white","yellow"],"logits":[-5.268555164337158,-5.277965068817139,-5.313487529754639,-0.1695554107427597],"probs":[0.005995036568492651,0.005938888527452946,0.005731627810746431,0.9823344945907593],"argmax_index":3,"argmax_id":"yellow","top2_margin":0.9763394580222666,"near_tie":false,"gold":"yellow","gold_correct":true},{"question_id":"insect","type":"noul","question_span":[373,383],"option_spans":[[394,412],[418,436]],"option_ids":["true","false"],"logits":[-2.51741361618042,1.4968559741973877],"probs":[0.01773589290678501,0.9822641015052795],"argmax_index":1,"argmax_id":"false","top2_margin":0.9645282085984945,"near_tie":false,"gold":"true","gold_correct":false},{"question_id":"centre_colour","type":"choice","question_span":[458,468],"option_spans":[[479,489],[495,508],[514,524],[530,540]],"option_ids":["blue","brown","green","white"],"logits":[-4.828190326690674,0.1938069462776184,-4.3376688957214355,-4.674854278564453],"probs":[0.006430333014577627,0.975571870803833,0.01050181221216917,0.00749594671651721],"argmax_index":1,"argmax_id":"brown","top2_margin":0.9650700585916638,"near_tie":false,"gold":"brown","gold_correct":true},{"question_id":"setting","type":"choice","question_span":[561,566],"option_spans":[[577,594],[600,612],[618,629],[635,649]],"option_ids":["outdoors","sand","snow","vase"],"logits":[0.3681952655315399,-5.062819957733154,-5.1942596435546875,-4.724268913269043],"probs":[0.9858424067497253,0.004316657781600952,0.0037849845830351114,0.006055897101759911],"argmax_index":0,"argmax_id":"outdoors","top2_margin":0.9797865096479654,"near_tie":false,"gold":"outdoors","gold_correct":true},{"question_id":"density","type":"score","question_span":[670,676],"option_spans":[[687,697],[703,714],[720,730]],"option_ids":["0","1","2"],"logits":[-2.7540669441223145,-1.2832974195480347,-0.7492126822471619],"probs":[0.07826199382543564,0.3406418263912201,0.5810961723327637],"argmax_index":2,"argmax_id":"2","top2_margin":0.24045434594154358,"near_tie":false,"gold":null,"gold_correct":null}],"systemone_response":{"model":"clef-flash","answers":{"petal_colour":{"type":"choice","choice":"yellow","confidence":0.9823,"probabilities":{"yellow":0.9823,"white":0.0057,"purple":0.006,"red":0.0059}},"insect":{"type":"noul","noul":0.0177},"centre_colour":{"type":"choice","choice":"brown","confidence":0.9756,"probabilities":{"brown":0.9756,"green":0.0105,"blue":0.0064,"white":0.0075}},"setting":{"type":"choice","choice":"outdoors","confidence":0.9858,"probabilities":{"vase":0.0061,"outdoors":0.9858,"sand":0.0043,"snow":0.0038}},"density":{"type":"score","score":1.5028,"confidence":0.5811,"legend":{"0":"sparse","1":"moderate","2":"dense"},"probabilities":{"0":0.0783,"1":0.3406,"2":0.5811}}},"usage":{"input_tokens":752,"output_tokens":0}},"checks":{"encode_equal_systemone_batch":true,"systemone_answers_equal_hook_path":true,"systemone_model_field":true,"systemone_usage_input_tokens":true,"head_rerun_max_abs_diff":0.0,"head_rerun_bit_equal":true,"spans_decode_to_rendered_text":true,"rope_closed_form_equal":true}}],"summary":{"records":{"fixture":186,"heldout":30,"total":216},"runs":{"fixture":214,"heldout":40,"total":254},"questions":{"fixture":405,"heldout":188},"decoder_gate_set":{"fixture_runs":213,"fixture_questions":401,"heldout_runs":40,"heldout_questions":188,"near_ties":["img_01/g256/clutter","img_01/native/clutter","img_06/g448/in_red_zone","photo_01/g256/kinds","ho_t07/text/questions_asked","ho_j01/text/late_start"]},"oracle_summary_fixture":{"complete":true,"runs":214,"expected_runs":214,"near_tie_threshold":0.02,"by_source_arm":{"cc0_photo/g256":{"runs":4,"records":4,"questions":16,"types":{"noul":5,"choice":7,"score":4},"tokens_p50":503.5,"tokens_p99":554.47,"tokens_max":556,"max_options":6,"near_tie":1,"near_tie_ids":["photo_01/g256/kinds"],"gold_questions":13,"gold_correct":11,"gold_accuracy":0.8461538461538461,"wall_s_p50":13.040722354002355,"wall_s_max":14.455852874991251,"all_checks_pass":true,"head_rerun_bit_equal_runs":4,"head_rerun_max_abs_diff":0.0},"cc0_photo/g448":{"runs":4,"records":4,"questions":16,"types":{"noul":5,"choice":7,"score":4},"tokens_p50":635.5,"tokens_p99":686.47,"tokens_max":688,"max_options":6,"near_tie":0,"near_tie_ids":[],"gold_questions":13,"gold_correct":12,"gold_accuracy":0.9230769230769231,"wall_s_p50":14.013799771506456,"wall_s_max":14.184826875003637,"all_checks_pass":true,"head_rerun_bit_equal_runs":4,"head_rerun_max_abs_diff":0.0},"cc0_photo/native":{"runs":4,"records":4,"questions":16,"types":{"noul":5,"choice":7,"score":4},"tokens_p50":1039.5,"tokens_p99":1672.4699999999998,"tokens_max":1692,"max_options":6,"near_tie":0,"near_tie_ids":[],"gold_questions":13,"gold_correct":12,"gold_accuracy":0.9230769230769231,"wall_s_p50":16.668746395997005,"wall_s_max":21.603107415998238,"all_checks_pass":true,"head_rerun_bit_equal_runs":4,"head_rerun_max_abs_diff":0.0},"own_image/g256":{"runs":10,"records":10,"questions":34,"types":{"noul":10,"choice":15,"score":9},"tokens_p50":464.5,"tokens_p99":554.76,"tokens_max":558,"max_options":10,"near_tie":1,"near_tie_ids":["img_01/g256/clutter"],"gold_questions":25,"gold_correct":22,"gold_accuracy":0.88,"wall_s_p50":12.303918728997814,"wall_s_max":14.050204874991323,"all_checks_pass":true,"head_rerun_bit_equal_runs":10,"head_rerun_max_abs_diff":0.0},"own_image/g448":{"runs":10,"records":10,"questions":34,"types":{"noul":10,"choice":15,"score":9},"tokens_p50":596.5,"tokens_p99":686.76,"tokens_max":690,"max_options":10,"near_tie":1,"near_tie_ids":["img_06/g448/in_red_zone"],"gold_questions":25,"gold_correct":24,"gold_accuracy":0.96,"wall_s_p50":14.213438062499336,"wall_s_max":15.057918749997043,"all_checks_pass":true,"head_rerun_bit_equal_runs":10,"head_rerun_max_abs_diff":0.0},"own_image/native":{"runs":10,"records":10,"questions":34,"types":{"noul":10,"choice":15,"score":9},"tokens_p50":581.5,"tokens_p99":1258.76,"tokens_max":1262,"max_options":10,"near_tie":1,"near_tie_ids":["img_01/native/clutter"],"gold_questions":25,"gold_correct":23,"gold_accuracy":0.92,"wall_s_p50":13.478594770997006,"wall_s_max":18.227609458001098,"all_checks_pass":true,"head_rerun_bit_equal_runs":10,"head_rerun_max_abs_diff":0.0},"own_json/text":{"runs":12,"records":12,"questions":45,"types":{"noul":15,"choice":19,"score":11},"tokens_p50":550.5,"tokens_p99":613.69,"tokens_max":616,"max_options":6,"near_tie":0,"near_tie_ids":[],"gold_questions":32,"gold_correct":32,"gold_accuracy":1.0,"wall_s_p50":11.593206562502019,"wall_s_max":13.094212417010567,"all_checks_pass":true,"head_rerun_bit_equal_runs":12,"head_rerun_max_abs_diff":0.0},"own_text/text":{"runs":13,"records":13,"questions":50,"types":{"noul":19,"choice":20,"score":11},"tokens_p50":493.0,"tokens_p99":923.4399999999996,"tokens_max":967,"max_options":10,"near_tie":0,"near_tie_ids":[],"gold_questions":43,"gold_correct":43,"gold_accuracy":1.0,"wall_s_p50":10.141154458993697,"wall_s_max":14.840472332987702,"all_checks_pass":true,"head_rerun_bit_equal_runs":13,"head_rerun_max_abs_diff":0.0},"own_text_long/text":{"runs":3,"records":3,"questions":16,"types":{"noul":5,"choice":8,"score":3},"tokens_p50":2494.0,"tokens_p99":2600.82,"tokens_max":2603,"max_options":5,"near_tie":0,"near_tie_ids":[],"gold_questions":14,"gold_correct":14,"gold_accuracy":1.0,"wall_s_p50":22.9717632910033,"wall_s_max":23.452956665991223,"all_checks_pass":true,"head_rerun_bit_equal_runs":3,"head_rerun_max_abs_diff":0.0},"semif_authored144/text":{"runs":144,"records":144,"questions":144,"types":{"noul":0,"choice":144,"score":0},"tokens_p50":190.0,"tokens_p99":211.14,"tokens_max":214,"max_options":3,"near_tie":0,"near_tie_ids":[],"gold_questions":144,"gold_correct":130,"gold_accuracy":0.9027777777777778,"wall_s_p50":11.867934833499021,"wall_s_max":13.645276333001675,"all_checks_pass":true,"head_rerun_bit_equal_runs":144,"head_rerun_max_abs_diff":0.0}},"own_subset":{"runs":70,"records":42,"questions":261,"types":{"noul":84,"choice":113,"score":64},"tokens_p50":558.0,"tokens_p99":2527.7900000000004,"tokens_max":2603,"max_options":10,"near_tie":4,"near_tie_ids":["img_01/g256/clutter","img_01/native/clutter","img_06/g448/in_red_zone","photo_01/g256/kinds"],"gold_questions":203,"gold_correct":193,"gold_accuracy":0.9507389162561576,"wall_s_p50":13.192888042001869,"wall_s_max":23.452956665991223,"all_checks_pass":true,"head_rerun_bit_equal_runs":70,"head_rerun_max_abs_diff":0.0},"semif":{"runs":144,"records":144,"questions":144,"types":{"noul":0,"choice":144,"score":0},"tokens_p50":190.0,"tokens_p99":211.14,"tokens_max":214,"max_options":3,"near_tie":0,"near_tie_ids":[],"gold_questions":144,"gold_correct":130,"gold_accuracy":0.9027777777777778,"wall_s_p50":11.867934833499021,"wall_s_max":13.645276333001675,"all_checks_pass":true,"head_rerun_bit_equal_runs":144,"head_rerun_max_abs_diff":0.0},"own_subset_text_and_g256":{"runs":42,"records":42,"questions":161,"types":{"noul":54,"choice":69,"score":38},"tokens_p50":498.0,"tokens_p99":2558.3099999999995,"tokens_max":2603,"max_options":10,"near_tie":2,"near_tie_ids":["img_01/g256/clutter","photo_01/g256/kinds"],"gold_questions":127,"gold_correct":122,"gold_accuracy":0.9606299212598425,"wall_s_p50":11.593206562502019,"wall_s_max":23.452956665991223,"all_checks_pass":true,"head_rerun_bit_equal_runs":42,"head_rerun_max_abs_diff":0.0},"all":{"runs":214,"records":186,"questions":405,"types":{"noul":84,"choice":257,"score":64},"tokens_p50":200.0,"tokens_p99":2298.390000000003,"tokens_max":2603,"max_options":10,"near_tie":4,"near_tie_ids":["img_01/g256/clutter","img_01/native/clutter","img_06/g448/in_red_zone","photo_01/g256/kinds"],"gold_questions":347,"gold_correct":323,"gold_accuracy":0.930835734870317,"wall_s_p50":11.881351687494316,"wall_s_max":23.452956665991223,"all_checks_pass":true,"head_rerun_bit_equal_runs":214,"head_rerun_max_abs_diff":0.0},"semif_by_family":{"evidence_interpretation":{"n":48,"correct":46},"rule_application":{"n":48,"correct":42},"candidate_selection":{"n":48,"correct":42}},"independent_forwards":[{"id":"own_t01","arm":"text","logits_bit_equal_to_systemone_run":true,"per_question":[true,true,true],"wall_s":11.078098917001626,"role":"determinism (fixture record 0)"},{"id":"own_j01","arm":"text","logits_bit_equal_to_systemone_run":true,"per_question":[true,true,true],"wall_s":11.280761083005928,"role":"independent model(batch) call"},{"id":"img_01","arm":"g256","logits_bit_equal_to_systemone_run":true,"per_question":[true,true,true],"wall_s":12.406592958010151,"role":"independent model(batch) call"},{"id":"img_04","arm":"native","logits_bit_equal_to_systemone_run":true,"per_question":[true,true,true,true],"wall_s":18.270908541991957,"role":"independent model(batch) call"},{"id":"semif_a3f18f3a63d45345942b","arm":"text","logits_bit_equal_to_systemone_run":true,"per_question":[true],"wall_s":10.354115416994318,"role":"independent model(batch) call"}]},"oracle_summary_heldout":{"complete":true,"runs":40,"expected_runs":40,"near_tie_threshold":0.02,"by_source_arm":{"heldout_image/g256":{"runs":8,"records":8,"questions":31,"types":{"noul":9,"choice":19,"score":3},"tokens_p50":555.5,"tokens_p99":695.89,"tokens_max":701,"max_options":10,"near_tie":0,"near_tie_ids":[],"gold_questions":31,"gold_correct":29,"gold_accuracy":0.9354838709677419,"wall_s_p50":12.851172167007462,"wall_s_max":13.47028058298747,"all_checks_pass":true,"head_rerun_bit_equal_runs":8,"head_rerun_max_abs_diff":0.0},"heldout_image/g448":{"runs":8,"records":8,"questions":31,"types":{"noul":9,"choice":19,"score":3},"tokens_p50":687.5,"tokens_p99":827.89,"tokens_max":833,"max_options":10,"near_tie":0,"near_tie_ids":[],"gold_questions":31,"gold_correct":29,"gold_accuracy":0.9354838709677419,"wall_s_p50":13.839468541991664,"wall_s_max":14.820220916997641,"all_checks_pass":true,"head_rerun_bit_equal_runs":8,"head_rerun_max_abs_diff":0.0},"heldout_json/text":{"runs":8,"records":8,"questions":36,"types":{"noul":13,"choice":17,"score":6},"tokens_p50":772.0,"tokens_p99":824.04,"tokens_max":826,"max_options":6,"near_tie":1,"near_tie_ids":["ho_j01/text/late_start"],"gold_questions":30,"gold_correct":25,"gold_accuracy":0.8333333333333334,"wall_s_p50":13.984552937501576,"wall_s_max":14.210936207993655,"all_checks_pass":true,"head_rerun_bit_equal_runs":8,"head_rerun_max_abs_diff":0.0},"heldout_photo/g256":{"runs":2,"records":2,"questions":10,"types":{"noul":3,"choice":5,"score":2},"tokens_p50":627.0,"tokens_p99":633.86,"tokens_max":634,"max_options":4,"near_tie":0,"near_tie_ids":[],"gold_questions":8,"gold_correct":7,"gold_accuracy":0.875,"wall_s_p50":13.235288791998755,"wall_s_max":13.326044541987358,"all_checks_pass":true,"head_rerun_bit_equal_runs":2,"head_rerun_max_abs_diff":0.0},"heldout_photo/g448":{"runs":2,"records":2,"questions":10,"types":{"noul":3,"choice":5,"score":2},"tokens_p50":759.0,"tokens_p99":765.86,"tokens_max":766,"max_options":4,"near_tie":0,"near_tie_ids":[],"gold_questions":8,"gold_correct":7,"gold_accuracy":0.875,"wall_s_p50":14.461410999996588,"wall_s_max":14.47575070898165,"all_checks_pass":true,"head_rerun_bit_equal_runs":2,"head_rerun_max_abs_diff":0.0},"heldout_text/text":{"runs":10,"records":10,"questions":56,"types":{"noul":20,"choice":25,"score":11},"tokens_p50":773.0,"tokens_p99":1066.29,"tokens_max":1086,"max_options":7,"near_tie":1,"near_tie_ids":["ho_t07/text/questions_asked"],"gold_questions":46,"gold_correct":43,"gold_accuracy":0.9347826086956522,"wall_s_p50":13.993519312003627,"wall_s_max":15.59947616700083,"all_checks_pass":true,"head_rerun_bit_equal_runs":10,"head_rerun_max_abs_diff":0.0},"heldout_text_long/text":{"runs":2,"records":2,"questions":14,"types":{"noul":4,"choice":8,"score":2},"tokens_p50":2790.5,"tokens_p99":2848.81,"tokens_max":2850,"max_options":5,"near_tie":0,"near_tie_ids":[],"gold_questions":12,"gold_correct":12,"gold_accuracy":1.0,"wall_s_p50":25.28669147900655,"wall_s_max":26.345506292011123,"all_checks_pass":true,"head_rerun_bit_equal_runs":2,"head_rerun_max_abs_diff":0.0}},"own_subset":{"runs":40,"records":30,"questions":188,"types":{"noul":61,"choice":98,"score":29},"tokens_p50":717.0,"tokens_p99":2803.59,"tokens_max":2850,"max_options":10,"near_tie":2,"near_tie_ids":["ho_t07/text/questions_asked","ho_j01/text/late_start"],"gold_questions":166,"gold_correct":152,"gold_accuracy":0.9156626506024096,"wall_s_p50":13.711458645993844,"wall_s_max":26.345506292011123,"all_checks_pass":true,"head_rerun_bit_equal_runs":40,"head_rerun_max_abs_diff":0.0},"semif":{"runs":0,"records":0,"questions":0,"types":{"noul":0,"choice":0,"score":0},"tokens_p50":null,"tokens_p99":null,"tokens_max":null,"max_options":null,"near_tie":0,"near_tie_ids":[],"gold_questions":0,"gold_correct":0,"gold_accuracy":null,"wall_s_p50":null,"wall_s_max":null,"all_checks_pass":true,"head_rerun_bit_equal_runs":0,"head_rerun_max_abs_diff":null},"own_subset_text_and_g256":{"runs":30,"records":30,"questions":147,"types":{"noul":49,"choice":74,"score":24},"tokens_p50":724.0,"tokens_p99":2815.4900000000002,"tokens_max":2850,"max_options":10,"near_tie":2,"near_tie_ids":["ho_t07/text/questions_asked","ho_j01/text/late_start"],"gold_questions":127,"gold_correct":116,"gold_accuracy":0.9133858267716536,"wall_s_p50":13.56940066649986,"wall_s_max":26.345506292011123,"all_checks_pass":true,"head_rerun_bit_equal_runs":30,"head_rerun_max_abs_diff":0.0},"all":{"runs":40,"records":30,"questions":188,"types":{"noul":61,"choice":98,"score":29},"tokens_p50":717.0,"tokens_p99":2803.59,"tokens_max":2850,"max_options":10,"near_tie":2,"near_tie_ids":["ho_t07/text/questions_asked","ho_j01/text/late_start"],"gold_questions":166,"gold_correct":152,"gold_accuracy":0.9156626506024096,"wall_s_p50":13.711458645993844,"wall_s_max":26.345506292011123,"all_checks_pass":true,"head_rerun_bit_equal_runs":40,"head_rerun_max_abs_diff":0.0},"semif_by_family":{},"independent_forwards":[{"id":"ho_t01","arm":"text","logits_bit_equal_to_systemone_run":true,"per_question":[true,true,true,true,true],"wall_s":13.666526582994265,"role":"independent model(batch) call"},{"id":"ho_j02","arm":"text","logits_bit_equal_to_systemone_run":true,"per_question":[true],"wall_s":11.042339042003732,"role":"independent model(batch) call"},{"id":"ho_i01","arm":"g256","logits_bit_equal_to_systemone_run":true,"per_question":[true,true,true,true],"wall_s":12.25833658297779,"role":"independent model(batch) call"},{"id":"ho_p01","arm":"g448","logits_bit_equal_to_systemone_run":true,"per_question":[true,true,true,true,true],"wall_s":14.310894666996319,"role":"independent model(batch) call"}]}},"determinism":{"fixture":[{"id":"own_t01","arm":"text","logits_bit_equal_to_systemone_run":true,"per_question":[true,true,true],"wall_s":11.078098917001626,"role":"determinism (fixture record 0)"},{"id":"own_j01","arm":"text","logits_bit_equal_to_systemone_run":true,"per_question":[true,true,true],"wall_s":11.280761083005928,"role":"independent model(batch) call"},{"id":"img_01","arm":"g256","logits_bit_equal_to_systemone_run":true,"per_question":[true,true,true],"wall_s":12.406592958010151,"role":"independent model(batch) call"},{"id":"img_04","arm":"native","logits_bit_equal_to_systemone_run":true,"per_question":[true,true,true,true],"wall_s":18.270908541991957,"role":"independent model(batch) call"},{"id":"semif_a3f18f3a63d45345942b","arm":"text","logits_bit_equal_to_systemone_run":true,"per_question":[true],"wall_s":10.354115416994318,"role":"independent model(batch) call"}],"heldout":[{"id":"ho_t01","arm":"text","logits_bit_equal_to_systemone_run":true,"per_question":[true,true,true,true,true],"wall_s":13.666526582994265,"role":"independent model(batch) call"},{"id":"ho_j02","arm":"text","logits_bit_equal_to_systemone_run":true,"per_question":[true],"wall_s":11.042339042003732,"role":"independent model(batch) call"},{"id":"ho_i01","arm":"g256","logits_bit_equal_to_systemone_run":true,"per_question":[true,true,true,true],"wall_s":12.25833658297779,"role":"independent model(batch) call"},{"id":"ho_p01","arm":"g448","logits_bit_equal_to_systemone_run":true,"per_question":[true,true,true,true,true],"wall_s":14.310894666996319,"role":"independent model(batch) call"}]},"trimmed":{"from_lane_files":{"fixtures/records.json":{"sha256":"84e24684b5ce29e8da7582a23acf83c923a02fc874681c9247c735ddaaadecbc","bytes":231726},"fixtures/images_meta.json":{"sha256":"e630446ee9564b1ed76f43cb6be5d5680c482a925748f8b489cf633352bd5bbb","bytes":9103},"fixtures/heldout/records.json":{"sha256":"e271fe203f026fd7c14e031ec25951516e64375c7b17a695ebbbba1e0d6ce7f7","bytes":77764},"fixtures/heldout/images_meta.json":{"sha256":"bbb0230d97e50d56368f51eca1560981ca676930aa3773b34b645f77601aa2dc","bytes":6400},"oracle/records_oracle.json":{"sha256":"d8adaa746a5d575e654212ca78f5634af334aaa62eb53ef150d5d5925b1fa7c0","bytes":1388820},"oracle/heldout/records_oracle.json":{"sha256":"85e4dda56890dff47a163bdd74c85291f5d3be0944403dd1f59ba2f5b9faaa3f","bytes":558837},"results/oracle_summary.json":{"sha256":"b660331ef505c547abb1339bb55febf5c1dd61ccb90222b90df6454eebc81a6e","bytes":9377},"oracle/heldout/oracle_summary.json":{"sha256":"1ac5b3a77680c87771acecf0873317315c60db546fc71955c1fdc5f1fef7b582","bytes":7152},"results/layout.json":{"sha256":"0952a5a4fe3e035c6f7bf9ab056db215fc57ff23963b33daa8010f8f922c7cf9","bytes":23043}},"removed":["per-run wall_s, head_rerun_s, device","hidden_states (shape notes of the 2 hidden-state dump runs)","mm_token_type_ids_media (1 exactly on the image-pad tokens, derivable from ids)","the SemIf jsonl's local path (upstream repo, commit and sha256 kept)"],"serialization":"compact json (no indentation)"}}