{"schema":"coreai-d1-omni-fixtures/1","what":"The public records of the d1-omni-600M Core AI port's fixture (v3) and the publisher's fp32 CPU numbers on every question of them, in text mode and, for a record with an image or a clip, in that media mode. The records are the HF repository's reference/records.json records, the rows its reference/oracle.json questions; the images and clips are named here and stored there (reference/images/, reference/audio/).","model":{"hf_id":"LiquidAI/d1-omni-600M","revision":"414f8d6438174f5b2133a9c21a478fc42625e308","license":"LFM Open License v1.0","oracle":{"reference":"AutoModel.from_pretrained(snapshot, trust_remote_code=True, dtype=torch.float32) (modeling_d1.D1OmniModel), CPU fp32, probabilities(state, questions, images, audio) per request","source_sha256":{"encoder.py":"5f2e20319ee42a7367febf06c7653d7b3f9b28ef99a64cb9ec7739563dcfcffe","modeling_d1.py":"2b71a5c909ec5d3aa6307378a41a0fd24af1e17d5c1c1f8d510582147051a686","prompt.py":"a6b29a55ec8345f1fc1fdbfcc4b64d80d473dc4316f095a62c51cb6cce1194cf","vision.py":"43ad71029f82e20fd26b97574627773d47639d7a3742c8b04fe5bf55f41c4c28","audio.py":"c5ff09d52d6a079e79d0505c89dbd0582ba2bc9abf2b26595d8d5b1d0c347bdf","config.json":"dfccae241822f81752426629c88c8b8d7efdd03dd19418b6645c604fad43d619"},"weights_sha256":"0713bb05270c2685ad106522f4092bceeeb3a93cf79b401f399a712296c911e1","versions":{"python":"3.12.11","platform":"macOS-27.0-arm64-arm-64bit","torch":"2.9.0","transformers":"5.19.0","numpy":"2.5.3","tokenizers":"0.23.2","safetensors":"0.8.0","huggingface_hub":"1.33.0","soundfile":"0.14.0","torchvision":"0.24.0","pillow":"12.3.0"}},"oracle_files":[{"file":"records_ref_audio.json","sha256":"c483f8c511cfcdb05a8211e495a7476b11f9318899a1125c387c9e722a6d8fbc","written":"2026-10-08T11:20:06+0900","fixtures_sha256":"18b4ddff73a8044b63f949c69245210e1af8a26c48c5fdb785fd91a7807ead5b"},{"file":"records_ref_images.json","sha256":"681d2d10eef35ecb8980a3caf93d922a77a17e88ee58795468e78b915ac91b08","written":"2026-10-08T08:38:10+0900","fixtures_sha256":"07ea38a2c0f3d8da4afe4dd4ae79e083c46404cb8ab0ba150b9c0a3fca83ae62"},{"file":"records_ref.json","sha256":"e7dba6f44d0452a6aea3e401746d417503056d6f468ff72b56c5511883436c5f","written":"2026-10-08T04:10:36+0900","fixtures_sha256":"c0781185cf9b86210024f94fa645f047af843bfaaf7d80c703c4272634030123"}]},"contract":{"row":"ids = [1, 17] + enc(state)[:room] + [18] + enc(instructions)[:max(16, budget)] + per option ([19, 16] + enc(' ' + text)[:per] + [20]) + [21]; enc(s) = the tokenizer's ids of escape(s) without special tokens; marker_positions = the positions of the 16s in ids (host.py, the publisher's prompt.encode)","graph":"positions 0 .. P-1 = the media prefix (P = prefix_rows), P .. P+len(ids)-1 = ids; the decision graph of the smallest window (L) that holds P + len(ids) returns a score per position; the host reads scores[P + marker] (host.py graph_inputs, probabilities_from_logits)","window":"host.bucket_for(positions, host.ALL_BUCKETS) = the shipped decision buckets (64 ... 4096); window_r10 = host.bucket_for(positions) over (256 ... 4096), the set before round 12","raw_logits":"the head's marker logits in the model's option order (noul: [false, true])","probabilities_raw":"softmax(raw_logits), the model's option order, no temperature","probabilities":"the reported probabilities: text rows softmax(raw_logits / temperature) with the temperature of temperature_key from config.json (type default 1.0); image and audio rows no temperature; noul reported as [yes, no] = [true, false] (the model's order reversed)","argmax":"the option key at argmax_index of probabilities (noul: 'true' or 'false')","near_tie":"the oracle's top-2 probability margin <= 0.02 (FACTS §7: argmax compared apart)","official_answer":"the publisher's response for this question (`system_one` answer shape)","qtype":{"choice":0,"score":1,"noul":2}},"sources":{"card":{"file":"README.md","sha256":"a18d4aa959198a001bc35be2309b3b21a265473fa59275e23823862c9544b411","section":"How to use","repo":"LiquidAI/d1-omni-600M","revision":"414f8d6438174f5b2133a9c21a478fc42625e308"},"semif_authored144":{"sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","upstream_repo":"github.com/TheoLeeCJ/SemIf","upstream_path":"benchmarks/data/authored144.jsonl","upstream_commit":"ca3ba65f142967030ecb453346e94d6f476a69df","copyright_notice":"Copyright (c) 2026 TheoLeeCJ","rows":144,"question_id":"decision","license":"MIT, Copyright (c) 2026 TheoLeeCJ","license_notice":{"file":"LICENSE-SemIf","sha256":"f765f2140f8507a8f0d81ec0fd2c4bd72fe6a066841ef27883ff876a76bf61be","text":"MIT License\n\nCopyright (c) 2026 TheoLeeCJ\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"}},"transfer_v4":{"path_in_repo":"evals/v4/transfer-v4/development.jsonl","sha256":"ff374c49c6c9f15f8a56fb274b4a4857d20497eb8dd1ac07ce01560e682a5f2e","lines":764,"author_repo":{"repo":"github.com/jaredpalmer/kev","tag":"kev-1.0","commit":"6b719c3c3f367295f6ef336f4f751cf5ff970abc"},"records_here":"tv4_000 ... tv4_059: the file's first 60 lines (all mmlu)","mmlu":{"hf_id":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","license":"MIT, Copyright (c) 2020 Dan Hendrycks","license_from":"https://raw.githubusercontent.com/hendrycks/test/master/LICENSE","license_notice":{"file":"LICENSE-MMLU-MIT.txt","sha256":"5d841b9eee8bf0d721359b79fbae956a8b426d3139bfefee32149c7d4f2a1a9c","text":"MIT License\n\nCopyright (c) 2020 Dan Hendrycks\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"}}},"own":{"license":"written for this port (Apache-2.0, as the conversion code)","records":["long_3400","own_L01","own_L03","own_a01","own_j01","own_j02","own_m01","own_t01","own_t04","own_t06","own_t07","own_t10"]},"images":{"what":"img_01: own model output (FLUX.2 klein 4B, Apache-2.0); img_02, img_03: CC0 1.0 photographs (Wikimedia Commons), downscaled; origin and resize in the HF repository's reference/images/manifest.json"},"audio":{"what":"aud_01 ... aud_15: speech synthesized with hexgrad/Kokoro-82M (Apache-2.0) from scripts written for these ports; the scripts in the HF repository's reference/audio/script.json"}},"media":[{"file":"audio/aud_01.wav","bytes":308078,"sha256":"ad576e24573dce521c9104d0acb05697395811743b17f0c58882c17d9a27feb4"},{"file":"audio/aud_02.wav","bytes":280878,"sha256":"d3128b99481a46e3631fb7fc396c54551a46398da271ed48f0a36c5217467563"},{"file":"audio/aud_03.wav","bytes":280078,"sha256":"c78244d4ae793e00dc0d82a609f72d14bdd1fe93705e9a99483a9bf3dadf4c6d"},{"file":"audio/aud_04.wav","bytes":89678,"sha256":"efbf3b4287c1ecbfa13b7fc9ab1a52e2a7a5d7b09ad2f8319f0a9bbba11b92b3"},{"file":"audio/aud_05.wav","bytes":92078,"sha256":"98d200490be9134d28d81f257bdf9e8d99520bb00ac20d68d26e748b0dd28c6c"},{"file":"audio/aud_06.wav","bytes":257678,"sha256":"b5b2522295c3cf334ef97fde4c44678ae8017958a2f4395cab0cf8a8ce7f964c"},{"file":"audio/aud_07.wav","bytes":270478,"sha256":"cc8e1ed29d98c65e2420106ef620f0c84ef94db37bc410d3fb341128f2dfe538"},{"file":"audio/aud_08.wav","bytes":508078,"sha256":"3a06935972640ca5f9d2758c35b5de504e2589734d805089f77d3b7742b95bd6"},{"file":"audio/aud_09.wav","bytes":438478,"sha256":"6a9e400bd5dc10207c0482f22ad275ffcc23804e2fd765470c9604aaad0f2428"},{"file":"audio/aud_10.wav","bytes":687278,"sha256":"e5c5dd555900c9cf5c78bac7cc0a9ab7dd0592aa6425d9af526d873940c73e6a"},{"file":"audio/aud_11.wav","bytes":671278,"sha256":"6d861cce1c398b7ea1783f99ae39bfcf082f082206feade3290570920231a8bb"},{"file":"audio/aud_12.wav","bytes":894478,"sha256":"e012012e8c3dbcd7b9b09d6c208db22bd7e8ccdbd1597985cdc09048a1fbf040"},{"file":"audio/aud_13.wav","bytes":910478,"sha256":"77e9d3c6bd3a7ed47ae819a8c8632c03da017312dd366a0832ec8f2bfb32926f"},{"file":"audio/aud_14.wav","bytes":11978,"sha256":"ccb80a9da1e09508c72795a5c0d442f024b95a1199c73e6f00699890a1bf730c"},{"file":"audio/aud_15.wav","bytes":1156878,"sha256":"d2ed03414e9da96d1055afcd92d8b0476d8f1f17c87f70ae9ce57434fb9fafa9"},{"file":"images/img_01.png","bytes":167859,"sha256":"ae59d05efc3d6990f1a02b728508cd786d140bdccf666f9a8a11c53d1e2191b9"},{"file":"images/img_02.png","bytes":146091,"sha256":"1864e1c215f23774abbd74860272c5f8e9ce439e75857b48b4cabaabf09195b7"},{"file":"images/img_03.png","bytes":566869,"sha256":"28080dd775920ac0b701d6640eb08e0500542946a19c84ab80e263e595224312"}],"fixtures":[{"id":"card_text","source":"card","public":true,"request":{"state":"I was charged twice this month, please refund one of them.","questions":{"refund":{"type":"noul","instructions":"Is the customer asking for a refund?"},"team":{"type":"choice","instructions":"Which team should handle this?","criteria":{"billing":"Charges, refunds, invoices","technical":"App or site faults","fraud":"Suspected unauthorised use"}},"urgency":{"type":"score","instructions":"How urgent is this?","criteria":["Can wait","Today","Blocking the customer now"]}}},"media":null,"gold":null,"note":"the card's text example (three named questions over one state)","provenance":{"file":"README.md","repo":"LiquidAI/d1-omni-600M","revision":"414f8d6438174f5b2133a9c21a478fc42625e308","sha256":"a18d4aa959198a001bc35be2309b3b21a265473fa59275e23823862c9544b411","section":"How to use"}},{"id":"card_batch_00","source":"card","public":true,"request":{"state":"Where is my parcel? It was due Monday.","questions":{"team":{"type":"choice","instructions":"Which team should handle this?","criteria":{"billing":"Charges, refunds, invoices","technical":"App or site faults","fraud":"Suspected unauthorised use"}}}},"media":null,"gold":null,"note":"the card's system_one_batch example, ticket 0","provenance":{"file":"README.md","repo":"LiquidAI/d1-omni-600M","revision":"414f8d6438174f5b2133a9c21a478fc42625e308","sha256":"a18d4aa959198a001bc35be2309b3b21a265473fa59275e23823862c9544b411","section":"How to use"}},{"id":"card_batch_01","source":"card","public":true,"request":{"state":"The app crashes when I open settings.","questions":{"team":{"type":"choice","instructions":"Which team should handle this?","criteria":{"billing":"Charges, refunds, invoices","technical":"App or site faults","fraud":"Suspected unauthorised use"}}}},"media":null,"gold":null,"note":"the card's system_one_batch example, ticket 1","provenance":{"file":"README.md","repo":"LiquidAI/d1-omni-600M","revision":"414f8d6438174f5b2133a9c21a478fc42625e308","sha256":"a18d4aa959198a001bc35be2309b3b21a265473fa59275e23823862c9544b411","section":"How to use"}},{"id":"tv4_000","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"security studies","question":"Which of the following did not emerge as a trend of the defence trade in the 20th century?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"The arms trade shifts from being a private to a government controlled enterprise.","b":"An increase in the defence trade.","c":"A decrease in the arms trade.","d":"A growth in the number and variety of weapons traded."}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/13292","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":0,"src":{"answer":"mmlu"},"_meta":{"row":13292,"text_sha256":"e4f765d62c6ed44999ddbaa502f1b865f19197d1e492af2abef30ac5dfd47750","row_sha256":"010a5fda50061ef619d06b589cf689715a9774f817996fb54d6d627b8499f882","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13292","group_id":"mmlu/test/13292","variant":"clean"},"line_sha256":"65e5e955976c82abb47df3b3287409cf867c6af264235c0c32f606cb8681de83"}},{"id":"tv4_001","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"clinical knowledge","question":"What is hypochlorhydria?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Increased amount of gastric acid.","b":"Gastric ulceration.","c":"Reduced amount of gastric acid.","d":"Pernicious anaemia."}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/588","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":1,"src":{"answer":"mmlu"},"_meta":{"row":588,"text_sha256":"6f26999eaee4ab138243c58aa5c0360817cb583c90caaac957438ec0b09f9d90","row_sha256":"010f9883d7f34928b7cf5b3fe2ecf5f7e662cf3cc29611c5790da5e6b28ef96a","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/588","group_id":"mmlu/test/588","variant":"clean"},"line_sha256":"56f810d754b4cb990275166ed819a6fdd67b05710dacb06ed9731548a5fea1dc"}},{"id":"tv4_002","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school microeconomics","question":"Which of the following is correct?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"In the long run, all inputs are variable.","b":"In the short run, all inputs are variable.","c":"In the long run, supply is not able to adjust fully to changes in demand.","d":"In the short run, supply is able to adjust fully to changes in demand."}}}},"media":null,"gold":{"answer":"a"},"note":"mmlu mmlu/test/4589","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":2,"src":{"answer":"mmlu"},"_meta":{"row":4589,"text_sha256":"b21eb6a47835c694dbb4c55d0c6eaf127143103b5be6e901b3bc081a1875e03a","row_sha256":"0140ae7a6c63a4b72a607fe028843ded2c762ed70839dd64f1e26c4f29fd8813","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4589","group_id":"mmlu/test/4589","variant":"clean"},"line_sha256":"668f5021cf322641a26d4cd4cbe417f80011784f61eeeb6ee18c3d5683877f9a"}},{"id":"tv4_003","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"human aging","question":"One reason that older adults show high levels of optimism is that"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Many pessimists die at a younger age","b":"They do not really know what is going on","c":"This may be a warning sign for dementia","d":"Our measures have low validity"}}}},"media":null,"gold":{"answer":"a"},"note":"mmlu mmlu/test/6099","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":3,"src":{"answer":"mmlu"},"_meta":{"row":6099,"text_sha256":"d11693badfd9d9aeb495b1bbd2b030536f3badf884e3c5c968350c222af0c1bf","row_sha256":"018fc4e2f2bc5d033d1203c9298ea91deb852be048cc2234325c3f764b24db98","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6099","group_id":"mmlu/test/6099","variant":"clean"},"line_sha256":"07c8fc2c87043af4a643fa2664336d5dbe1bb67fb5263fde3b8f1e96defd5dcb"}},{"id":"tv4_004","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"human aging","question":"Centenarians are people who"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Were born before the centennial","b":"Study old age and older adults","c":"Are 100 or more years old","d":"Have not yet reached old age"}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/6139","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":4,"src":{"answer":"mmlu"},"_meta":{"row":6139,"text_sha256":"67fe59443c44fef770b56204f7dd4ac9a42a3d1d492a4249ec7e742c7c97566f","row_sha256":"022dbb0545abb6ee6edcb5b28dbfdc7099785c2a5b106aa0536906349b11988d","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6139","group_id":"mmlu/test/6139","variant":"clean"},"line_sha256":"0b68b799521da2826e8661997053ccf67c4c896684e9406459f674fda5b2f287"}},{"id":"tv4_005","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"virology","question":"The influenza virus has complicated genetics mainly characterised by which of the following?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Genetic reassortment amongst 8 genes","b":"High levels of recombination","c":"Rapid adsorption and fusion from without via a hydrophobic peptide","d":"High fidelity RNA replicase enzyme"}}}},"media":null,"gold":{"answer":"b"},"note":"mmlu mmlu/test/13810","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":5,"src":{"answer":"mmlu"},"_meta":{"row":13810,"text_sha256":"d6ab903bb64d97ef4252906a9c35b394c2b135cc82173b9a98a562ae4941e714","row_sha256":"03bd43e6287a70dfe5cc0e5fe2bf7bfe2cdf49e4a895d91a13fdde9bbb4ff5d9","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13810","group_id":"mmlu/test/13810","variant":"clean"},"line_sha256":"72a72e39855ff80bac41e07ab26fdefde7d730c0ecd25723e41241d5d572ab53"}},{"id":"tv4_006","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"professional medicine","question":"A 57-year-old woman comes to the physician because of an 8-week history of difficulty sleeping, fatigue, and muscle tension. During this period, she also has had memory lapses, difficulty concentrating, and has been reprimanded at work for arriving late. Over the past 2 weeks, she has had three episodes of palpitations and shortness of breath that have awakened her from sleep. Her pulse is 80/min, and blood pressure is 110/90 mm Hg. Physical examination shows no abnormalities. Mental status examination shows a depressed mood and constricted affect. She says that she is no longer interested in activities that she used to enjoy. She has suicidal ideation without a plan. Her hemoglobin concentration is 11 g/dL, and serum ferritin concentration is 140 ng/mL. Which of the following is the most appropriate initial step in treatment?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Donepezil therapy","b":"Ferrous sulfate therapy","c":"Ginkgo biloba extract therapy","d":"Paroxetine therapy"}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/12434","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":6,"src":{"answer":"mmlu"},"_meta":{"row":12434,"text_sha256":"6415833a28d2dcfe5791e61f69f47411f55fa901fdbcb63f0f2a39b718ad434b","row_sha256":"04184bb438912221313ece06f12d94cc5b9eb0500c9a978a14d7e05f5fcbbedf","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12434","group_id":"mmlu/test/12434","variant":"clean"},"line_sha256":"a6b6b58f862a9d6b00840836ad6eb39240c7678f95758101815267d19d234362"}},{"id":"tv4_007","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"anatomy","question":"Gastrulation is the process of"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"mesoderm formation and occurs after neurulation.","b":"ectomesenchyme formation and occurs before neurulation.","c":"mesoderm formation and occurs before neurulation.","d":"ectomesenchyme formation and occurs after neurulation."}}}},"media":null,"gold":{"answer":"b"},"note":"mmlu mmlu/test/123","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":7,"src":{"answer":"mmlu"},"_meta":{"row":123,"text_sha256":"b91ae5229f181bae8ead6b7d5617ebf84c8a0a6e86eb7b45f9f04ff55413c613","row_sha256":"041edcea28c4c2a8b1b57e50a75d175e7efa26cce57a5e8dc554add9a2369644","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/123","group_id":"mmlu/test/123","variant":"clean"},"line_sha256":"03798a32d17a3722b4a0812045f32cb3950273c26ae67b12f804cb91b29f8245"}},{"id":"tv4_008","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school biology","question":"Which of the following properties is responsible for holding together the water molecules in a vertical column of water in xylem vessels of a plant?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"the strong adhesion property of water","b":"the strong cohesion of property of water","c":"the high heat capacity of water","d":"the strong cohesion property of cellulose in the xylem cell wall"}}}},"media":null,"gold":{"answer":"b"},"note":"mmlu mmlu/test/2785","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":8,"src":{"answer":"mmlu"},"_meta":{"row":2785,"text_sha256":"15d7296ec5506ce45f273de8dca692fd7ee3b091af4b89c5d8ff60e3aa972fd4","row_sha256":"0434623eedfb30e0c938e9f5342d1ebc1849aa07e10549d9bfccffc0d06e8ed6","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/2785","group_id":"mmlu/test/2785","variant":"clean"},"line_sha256":"eec6954816087f3349953b57bb383aad1739198ea170c995d1c6888989bd1e78"}},{"id":"tv4_009","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school us history","question":"This question refers to the following information.\n\"I marvel not a little, right worshipful, that since the first discovery of America (which is now full four score and ten years), after so great conquests and plantings of the Spaniards and Portuguese there, that we of England could never have the grace to set fast footing in such fertile and temperate places as are left as yet unpossessed of them. But . . . I conceive great hope that the time approacheth and now is that we of England may share and part stakes [divide the prize] (if we will ourselves) both with the Spaniard and the Portuguese in part of America and other regions as yet undiscovered.\n\"And surely if there were in us that desire to advance the honor of our country which ought to be in every good man, we would not all this while have [neglected] the possessing of these lands which of equity and right appertain unto us, as by the discourses that follow shall appear most plainly.\"\n—Richard Hakluyt, Divers Voyages Touching the Discovery of America and the Islands Adjacent, 1582\nBy following the ideas of Richard Hakluyt, England was eventually able to"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"drive the French and Portuguese governments into bankruptcy.","b":"conquer large parts of Africa in the eighteenth century.","c":"establish several colonies along the Atlantic coastline of North America.","d":"destroy the Dutch commercial empire."}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/5717","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":9,"src":{"answer":"mmlu"},"_meta":{"row":5717,"text_sha256":"ae688490966c0852a0980ac7935c42ea98308efbfaca49a1cd5589064dba6cac","row_sha256":"05d282bcb34f55ca7b59ecafcb9a63c6d36d97a015a1c48979f80a7d6c7a4ea6","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/5717","group_id":"mmlu/test/5717","variant":"clean"},"line_sha256":"99228b97f13d5bc63c18ca3970111a36b887f0baf745f164de45feb7ce5481db"}},{"id":"tv4_010","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"miscellaneous","question":"Which actress played a pointy-eared Vulcan in the movie 'Star Trek II: The Wrath of Khan'?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Whoopi Goldberg","b":"Jennifer Grey","c":"Kirstie Alley","d":"Helen Hunt"}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/8037","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":10,"src":{"answer":"mmlu"},"_meta":{"row":8037,"text_sha256":"6eb96bdf834743053b122379db03e7f056ad3dc8ff3f5513b290c40620d2555f","row_sha256":"064a938d631d75f6147d58bb15659a7e76b9afd06b060eced5705d607167a4dd","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/8037","group_id":"mmlu/test/8037","variant":"clean"},"line_sha256":"c7255ac19ceaddddadcd644bd539d3093a959d96bc90a689b56c313da9f5be74"}},{"id":"tv4_011","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"college computer science","question":"Which of the following statements about horizontal versus vertical microarchitecture is (are) true?\nI. Programs for horizontal architectures require more time steps than those for vertical architectures.\nII. Horizontal microinstructions are unencoded.\nIII. Horizontal microinstructions usually have a single opcode and multiple operand specifiers."},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"I only","b":"II only","c":"III only","d":"II and III only"}}}},"media":null,"gold":{"answer":"b"},"note":"mmlu mmlu/test/1047","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":11,"src":{"answer":"mmlu"},"_meta":{"row":1047,"text_sha256":"2a0d61f454a3b0592fa563c96cd9d4ec0981bf856112ed46603214e9573eac02","row_sha256":"065c2a3f0ad371d06e17d82fe4bc8bfc5855bca2a04de9c1c9adb08d551a8639","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1047","group_id":"mmlu/test/1047","variant":"clean"},"line_sha256":"ad470c6a57936d5e202ba37caad6a69604d5aa423f97612b5f4eba926186d1e4"}},{"id":"tv4_012","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school microeconomics","question":"After Julia received a raise in her income, she began purchasing more ice cream cones and fewer popsicles. For Julia, popsicles"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"disobey the law of demand","b":"are a joint product","c":"are a complementary good with ice cream","d":"are an inferior good"}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/4518","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":12,"src":{"answer":"mmlu"},"_meta":{"row":4518,"text_sha256":"72be257ffdd59cdd7a74367581cd0aa5d75169cacfe687ab8b9bfbacafc817d0","row_sha256":"06f9edd28b068873d96ccfdb1df2848b00d64e1fd317e529fc71ac37a979eb39","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4518","group_id":"mmlu/test/4518","variant":"clean"},"line_sha256":"40bbc2da9f39e86fe5336e3bd8036472357e6ed9a8a082be37f75ea777d1de0f"}},{"id":"tv4_013","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"computer security","question":"In Brumley and Boneh’s paper on side-channel attacks, why does blinding prevent the timing attack from working?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Blinding prevents the server from using the CRT optimization, which is essential to the timing attack.","b":"Blinding changes the p and q primes that are used, so an adversary cannot learn the server’s true p and q values.","c":"Blinding randomizes the ciphertext being decrypted, thus obscuring the correlation between an adversary’s input and the timing differences.","d":"Blinding adds a random amount of time to the decryption due to the multiplication and division by the blinding random value r, which obscures the timing differences used in the attack."}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/1492","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":13,"src":{"answer":"mmlu"},"_meta":{"row":1492,"text_sha256":"c76022d4ee47dfc12b1e1ba7ff196c857d494ffbcd679e8b1d655f668fd72500","row_sha256":"070ed9de720c112fbd8b78c4e6dbbd24c1734d23867fb249de507295794e6cdf","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1492","group_id":"mmlu/test/1492","variant":"clean"},"line_sha256":"dcdb88cdb38f650686756813eb037728e3dfc1d1c204bd95a5af0536ff5fa86f"}},{"id":"tv4_014","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school macroeconomics","question":"Which of the following events has no effect on GDP?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"You buy a 1957 Chevy from a friend.","b":"The Department of Transportation repaves a road.","c":"Your friends make a music CD that doesn't sell any copies.","d":"A college buys computers."}}}},"media":null,"gold":{"answer":"a"},"note":"mmlu mmlu/test/4063","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":14,"src":{"answer":"mmlu"},"_meta":{"row":4063,"text_sha256":"5cc13884a1a177fa42853b29a986afb3d31c76715d41b3ef3263fd69b6438330","row_sha256":"0733d390eb549c8c29ff274b7f6c2ef73f6dcc0629dd53b1b19b34e43f31f116","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4063","group_id":"mmlu/test/4063","variant":"clean"},"line_sha256":"b0d027a49011258e7f0ad072081b1ab6411ff341bea9f10f7f423df31f962312"}},{"id":"tv4_015","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"professional law","question":"A college student was walking home from a party. He had been drinking alcohol at the party and was very intoxicated. Local police officers stopped the student and questioned him. When he refused to answer any of their questions, the police put the student in the back seat of a police car. The police left the student in the back seat of the police car for approximately one hour, and then released him. If the student asserts a claim against the police for false imprisonment, he will most likely"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"prevail, unless the police made a valid arrest.","b":"prevail, if he consented to the confinement because he was intoxicated.","c":"not prevail, because police cannot be held liable for false imprisonment.","d":"not prevail, because the student suffered no harm."}}}},"media":null,"gold":{"answer":"a"},"note":"mmlu mmlu/test/11310","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":15,"src":{"answer":"mmlu"},"_meta":{"row":11310,"text_sha256":"08836e0191901e64734fbe0f9a786efaf3709406e58f7fa495078bcb610c53a9","row_sha256":"07cea382ce1e459ddb0a46898382abcb1762fd9b713a087f06aca048f0923b2a","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/11310","group_id":"mmlu/test/11310","variant":"clean"},"line_sha256":"e6fa25992cab53f1dac576717376a7e08b467dde58cafe18efea22dd5f8de21d"}},{"id":"tv4_016","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"jurisprudence","question":"Which proposition below is the most inconsistent with a consequentialist justification of punishment?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Punishment protects security.","b":"Punishment is a cost-effective method by which to prevent or reduce crime.","c":"Punishment protects personal safety.","d":"Punishment should fit the crime."}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/6610","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":16,"src":{"answer":"mmlu"},"_meta":{"row":6610,"text_sha256":"230af30dc37acdac5cace354b74414e19f977465f005f72fb6f13ce1c83fc01f","row_sha256":"080972877925a0e9071a647836ab7bc1b9f6fb860fd37fd74b9d115eb1a8a150","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6610","group_id":"mmlu/test/6610","variant":"clean"},"line_sha256":"5eb8cd7d0f9291ad40dfb323c2c3dbc75dc5712eb1117a38bac38fc8f81d73d1"}},{"id":"tv4_017","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"nutrition","question":"Children who are obese should:\n"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Follow a different diet to the rest of the family","b":"Eat the same and take more exercise","c":"Reduce their total energy intake, increase their activity level and reduce their screen time","d":"Eat more fruit and vegetables"}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/9462","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":17,"src":{"answer":"mmlu"},"_meta":{"row":9462,"text_sha256":"f658cc8e1232787da72bf6e03aa3f71f61d9b2e1cbd9cb34e78e29b12865dcfb","row_sha256":"0813758be42e718265867e9b104410830958994506ff946e678204da674ab60e","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/9462","group_id":"mmlu/test/9462","variant":"clean"},"line_sha256":"764e7d81d8e2293eb8261c9cfd55d4c0049e71b0c664d43957458d40e06f9479"}},{"id":"tv4_018","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school psychology","question":"Which of the following refers to the benefit of having an emotional release to reduce aggressive tendencies?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"James-Lange theory of emotion","b":"Counterconditioning","c":"Catharsis hypothesis","d":"Transference"}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/5208","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":18,"src":{"answer":"mmlu"},"_meta":{"row":5208,"text_sha256":"31a70389d9e44cad3395616eac1504fa0ce169118cd1d71dce3763e10a62c45c","row_sha256":"08cdef9dcb2506c626af30d1df6d606b616e510de7a393b065c19acf5da24497","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/5208","group_id":"mmlu/test/5208","variant":"clean"},"line_sha256":"9fdf49f65e1bbc1bfe397e1744d67d7dfad6cb9b53309bd1bd9cd70e4a36ce29"}},{"id":"tv4_019","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"moral scenarios","question":"For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I was excited to hear that my stepdad had not pulled through his operation. Scenario 2 | I called 911 and reported that my neighbor was sitting quietly at her home."},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Wrong, Wrong","b":"Wrong, Not wrong","c":"Not wrong, Wrong","d":"Not wrong, Not wrong"}}}},"media":null,"gold":{"answer":"a"},"note":"mmlu mmlu/test/9213","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":19,"src":{"answer":"mmlu"},"_meta":{"row":9213,"text_sha256":"ad7d058864a902202cc16c98a71221e2eee376a5cb319653c0581de66893e99f","row_sha256":"0ad24336bd4a190abb0870d37ed56901090b8c288422d98b2b8dae935e0a5951","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/9213","group_id":"mmlu/test/9213","variant":"clean"},"line_sha256":"a1965ab273faa7dfa644f1847e3bf6f1e2043182acd27c8ad8e6974bb53b1c64"}},{"id":"tv4_020","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"professional psychology","question":"\"According to Hersey and Blanchard's situational leadership model, \"\"employee maturity\"\" is a function of the employee’s:\""},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"stage of career development.","b":"personality.","c":"skills and willingness to assume responsibility.","d":"beliefs about the meaning and value of work."}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/12914","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":20,"src":{"answer":"mmlu"},"_meta":{"row":12914,"text_sha256":"5afdeb67f2af871c752771dd9425be2bc6967bcb203b35de05f057220a27789a","row_sha256":"0ada8ef3402be468e4a35329742fa3232f1ff88aac6612afb0d092090c50e3f3","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12914","group_id":"mmlu/test/12914","variant":"clean"},"line_sha256":"34d5162147130e4d740b5f5ffdc6f097a139b4cb782323b77a9d2b4d79244487"}},{"id":"tv4_021","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"clinical knowledge","question":"A surgical support can be used for:"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"prolapsed stomas.","b":"high output stomas.","c":"herniation around the stoma.","d":"retraction of the stoma."}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/724","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":21,"src":{"answer":"mmlu"},"_meta":{"row":724,"text_sha256":"e38af88bca0cd26b62120a5e330698e74c26820357a06f0fd8da19c39a867706","row_sha256":"0adf991fd97e7801b25429b4babec11df58b8d5dc9f6e81989c1ae1694338f59","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/724","group_id":"mmlu/test/724","variant":"clean"},"line_sha256":"24e96525cc66ce161c1854342a4600204e85d284d29eb4b16b1dd916b98b1b06"}},{"id":"tv4_022","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school mathematics","question":"A consumer report revealed the following information about three tubes of toothpaste. Bright is $60 \\%$ more expensive than Fresh and has $25\\%$ less volume than Glow. Glow is $25\\%$ less expensive than Bright and has $33\\frac{1}{3} \\%$ more volume than Fresh. Fresh costs $\\$1.00$ per unit of volume. What is the number of cents per unit of volume of Glow?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"90","b":"120","c":"60","d":"320"}}}},"media":null,"gold":{"answer":"a"},"note":"mmlu mmlu/test/4454","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":22,"src":{"answer":"mmlu"},"_meta":{"row":4454,"text_sha256":"28649f1c96e825de7daa47754c989b906cd75b4644525f722544a5d298b8468c","row_sha256":"0b7df30f69dd9f462fbac0d3c7662a9b4b38a59a4d4f11310fc1969525ae8ae8","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4454","group_id":"mmlu/test/4454","variant":"clean"},"line_sha256":"e96062586430afadc9c1e839a584945437762330cc4958483cc498b1d793653d"}},{"id":"tv4_023","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"moral disputes","question":" In Baxter's view, the first and foremost step toward a solution of our environmental problems is a clear recognition that our objective is not pure air or water but rather"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"some optimal state of pollution.","b":"an overall maximally clean environment.","c":"some maximal state of pollution.","d":"the preservation of all living species."}}}},"media":null,"gold":{"answer":"a"},"note":"mmlu mmlu/test/8228","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":23,"src":{"answer":"mmlu"},"_meta":{"row":8228,"text_sha256":"5eda8abb1505371de513af404a66c64321f24733a74fe9d7997751869896daca","row_sha256":"0bec9e299974e1bbd0299dc5b542170ed2e7f0c823d095e8c95bd34dd66c5ceb","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/8228","group_id":"mmlu/test/8228","variant":"clean"},"line_sha256":"a6b39e74e00a0b153bae5903309090d8c3fb22d15ad193eae6008ca370926641"}},{"id":"tv4_024","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"security studies","question":"What is the value of a post-structuralist critique in security studies?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"To enable us to understand the workings of political subjecthoods left out of traditional discourses on security.","b":"To examine the base of power and its construction, in greater depth than traditional analysis.","c":"To challenge the idea that smart weapons and technology are the active subjects determining security today.","d":"All of these."}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/13238","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":24,"src":{"answer":"mmlu"},"_meta":{"row":13238,"text_sha256":"5c97128bd214131f1ddac5218553879a4bd91db54f482e32c57da5913e6d0f09","row_sha256":"0cace9a6eff08851fc6f07a62a24fcb1f1c06c96dceebc71c4dc67fec7a6bf10","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13238","group_id":"mmlu/test/13238","variant":"clean"},"line_sha256":"debbb575343af7e4b0b0cc5bd8822991f487607194abcb622e85cb917e75e4cb"}},{"id":"tv4_025","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"professional law","question":"A deed executed by a woman in 2001 conveyed a tract of land for a consideration of one dollar, receipt of which was acknowledged, \"to my friend for life, but if liquor is ever sold on the tract of land, then to my cousin and his heirs, and if for any reason the interest hereby conveyed to my cousin is not valid, then I reserve the right to re-enter the tract of land and take back my property. \" In 2004, the friend died intestate before the wheat he had planted could be harvested. Who is entitled to the proceeds of the crop?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"The friend's heirs.","b":"The cousin.","c":"The woman.","d":"Divided equally between the friend's heirs and the woman."}}}},"media":null,"gold":{"answer":"a"},"note":"mmlu mmlu/test/11404","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":25,"src":{"answer":"mmlu"},"_meta":{"row":11404,"text_sha256":"c27531d6a1741c33d2f26d6e9fddc7936f384064c89cde64dcc2ccdb8481cf78","row_sha256":"0cd28fb4615e5cff84b7f6c042098abed66dd1a265c7c45364224dbff09e2c84","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/11404","group_id":"mmlu/test/11404","variant":"clean"},"line_sha256":"590d744bb4f3a367f1c5a9053a7b934ff16835b17893086a5d58597109a146f4"}},{"id":"tv4_026","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"jurisprudence","question":"Critical Legal Studies (CLS) is often compared to (or with) American realism. Which of the following statements is inaccurate?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Both are concerned with the 'law in action.'","b":"Both are anti-formalist and sceptical","c":"Both adopt a liberal ideology.","d":"Both attempt to demystify the law."}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/6593","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":26,"src":{"answer":"mmlu"},"_meta":{"row":6593,"text_sha256":"fa18d8eb79c698b5e014bb3484bc1ac5132f0d94874455a0cead140a36871b4b","row_sha256":"0d02b58d02f3e95f74b821635dfdddc34a8b2ace30bdbc508f05f915cd44bfb3","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6593","group_id":"mmlu/test/6593","variant":"clean"},"line_sha256":"bbe8c56a361539948fc980ad5f65ed6410c6ce97da8f7d1d80085a7859f15f50"}},{"id":"tv4_027","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school macroeconomics","question":"Tariffs and quotas"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"result in lower domestic prices.","b":"sometimes raise and sometimes lower the amount of the product sold domestically.","c":"lower the amount of the product sold domestically.","d":"raise the amount of the product sold domestically."}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/3841","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":27,"src":{"answer":"mmlu"},"_meta":{"row":3841,"text_sha256":"080abd0a3265a45f0136d1fa3f9b3eb724571503c178f9c082ca331d2550ee66","row_sha256":"0d43bd015715be6a656242ba73f2e7cb6e82355ad11e9c12914a2d9052eccd2b","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/3841","group_id":"mmlu/test/3841","variant":"clean"},"line_sha256":"d3b4c0e91e7a2e3af4bf3c2ee7f019c026e2e47782753822e7e00ae9021900d8"}},{"id":"tv4_028","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"professional law","question":"A plaintiff is the beneficiary of a policy issued by an insurance company insuring the life of his wife, now deceased. The policy contains a clause providing that double indemnity is payable in the event that death of the insured \"results directly, and independently of all other causes, from bodily injury effected solely through external violent and accidental means. \"The plaintiff's wife met her death in the silage shed of her farm. The death resulted from wounds caused by the discharge of a double-barreled shotgun. The plaintiff was arrested and prosecuted for the murder of his wife. After a lengthy trial, the plaintiff was acquitted of the charge. After the insurance company refused to pay the plaintiff's insurance claim, the plaintiff instituted this civil action. The complaint sets forth the policy, alleges the facts surrounding the death of the insured, avers that the death was within the policy provisions for double indemnity payment, and demands judgment accordingly. At trial, the insurance company's attorney proposes to introduce excerpts from the plaintiff's murder trial. The plaintiff's attorney objects to the introduction of such evidence. The trial judge should rule the proffered evidence"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"admissible as a declaration against interest.","b":"admissible as former testimony.","c":"inadmissible, because of collateral estoppel.","d":"inadmissible, because of double jeopardy."}}}},"media":null,"gold":{"answer":"b"},"note":"mmlu mmlu/test/11705","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":28,"src":{"answer":"mmlu"},"_meta":{"row":11705,"text_sha256":"73ee9198bb27004a774c7defd4636b24f6931b85320a3fb826f31dbd7d2bbe02","row_sha256":"0eb66519306fa8a8ecc1873b57d6bbf252d623dc5481e64207120b290c2359aa","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/11705","group_id":"mmlu/test/11705","variant":"clean"},"line_sha256":"245be49c410a548c03b05a78b3e8edcbaeace42ef8247d90f591e69f036f18e9"}},{"id":"tv4_029","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school biology","question":"Chemiosmosis occurs in I.Mitochondria II. Nuclei III. Chloroplasts"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"I only","b":"II only","c":"III only","d":"I and III"}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/2862","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":29,"src":{"answer":"mmlu"},"_meta":{"row":2862,"text_sha256":"d429e7cd3c0676ad51fe99eb08872dabbb5f9f457cfed19084009276fcfb0ab5","row_sha256":"0f055897f863481cb6f33301380858951cd7a3fad82fec2a605fcfe4551d0596","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/2862","group_id":"mmlu/test/2862","variant":"clean"},"line_sha256":"18fe39d4f3dac58636dadfce7d9023174a4a6df8c2e70e806a124f4112ff729e"}},{"id":"tv4_030","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"philosophy","question":"Hume defines virtue as:"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"a tendency to maximize happiness.","b":"a tendency to pursue justice.","c":"whatever mental action or quality gives a spectator a sentiment of approbation.","d":"whatever mental action or quality brings about the best consequences for all."}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/9772","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":30,"src":{"answer":"mmlu"},"_meta":{"row":9772,"text_sha256":"18cbaea8b014f81f1a3cae404be60d6cde12ff38a648f3ec08b8c51464b57137","row_sha256":"0f24142b228b4133d31115ae881abcd37fd4efd0afe4f91de60e11fb598211d1","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/9772","group_id":"mmlu/test/9772","variant":"clean"},"line_sha256":"7af97b0ac0cde7df1c315b6e0f142d2b0cad82cfbf556916814d2d50e21ac5b5"}},{"id":"tv4_031","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school psychology","question":"According to Erikson, a young adult's developmental crisis centers around"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"intimacy vs. isolation","b":"identity vs. role confusion","c":"autonomy vs. shame and guilt","d":"industry vs. inferiority"}}}},"media":null,"gold":{"answer":"a"},"note":"mmlu mmlu/test/5094","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":31,"src":{"answer":"mmlu"},"_meta":{"row":5094,"text_sha256":"2bafd22f31a9fff5917ac28f483d154f2a964ff090176b1ba0b4c9434403801c","row_sha256":"0f69624a2c67dc837033e8037a6c0e7f8748b6c94ed6ef265ec82be45c5cb8e7","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/5094","group_id":"mmlu/test/5094","variant":"clean"},"line_sha256":"9151dd70b6903d8570a8a095c0a3a3122264a4d700ad345ff1a1f27f8be3c2bf"}},{"id":"tv4_032","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"prehistory","question":"What is the difference between an absolute date and a relative date?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"An absolute date is known by everyone, while a relative date ranges from 30 to 50 years.","b":"A relative date is derived from radiometric dating techniques, while an absolute date is only associated with written documents.","c":"A relative date refers to a year or a range of years, while an absolute date indicates a chronological order.","d":"An absolute date refers to a year or a range of years, while a relative date indicates a chronological order."}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/10172","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":32,"src":{"answer":"mmlu"},"_meta":{"row":10172,"text_sha256":"e063a35bc796e0c4eb85c16dc5c8b1fbb82701c11168fb320be646ba28476bfc","row_sha256":"0f9a5d1f0ef7225fd0f2bd67d68d0ebc47bb8ce1d98e3a082d23ca27d4cd5182","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/10172","group_id":"mmlu/test/10172","variant":"clean"},"line_sha256":"4b326f5221f5040a86c383896719fc43d6668931e4d4ef791bfc98184ba4fea2"}},{"id":"tv4_033","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school european history","question":"This question refers to the following information.\n\"It's come! It's posted at the district mayor's office,\" a passerby shouted at me as he ran. I reached the Rue Drout in one leap. … I read the message at a glance. … \"The First Day of Mobilization Will Be Sunday, August 2 [1914].\" … It was an announcement to a million and a half Frenchmen. … War! … Dead tired but exhilarated, I got back to [my newspaper's office] and burst into the office of Georges Clemenceau, our chief. \"What is Paris saying?\" he asked me. \"It's singing, sir!\" \"It will be all right then[,\" Clemenceau replied].\nRoland Doregelès, After 50 Years, c. 1965\nFrom the passage, one may infer that Doregelès recalled that"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"mobilization for war in 1914 was greeted with great skepticism in Paris","b":"mobilization for war in 1914 was greeted with great skepticism all across Europe","c":"mobilization for war in 1914 was greeted with great enthusiasm all across Europe","d":"mobilization for war in 1914 was greeted with great enthusiasm in Paris"}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/3349","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":33,"src":{"answer":"mmlu"},"_meta":{"row":3349,"text_sha256":"2727ef590d6a7b74fc042faafcf9c9570b374400f88c35e1069784efabaa7ef7","row_sha256":"111ff188a463868073a49590f1be6da85ca9649c068f5dd639fcce931036c5c8","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/3349","group_id":"mmlu/test/3349","variant":"clean"},"line_sha256":"983164def8eb976b604ae8e11dd535f502e204e8b7f21baaf08bef38422109ac"}},{"id":"tv4_034","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school psychology","question":"A group of participants in a sleep study are to be deprived of sleep for four days. After their second sleepless night, participants may begin reporting which of the following?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Hunger","b":"Thirst","c":"Lack of coordination","d":"Hallucinations"}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/5191","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":34,"src":{"answer":"mmlu"},"_meta":{"row":5191,"text_sha256":"526fc9864f083866076d59ec085a5840d2dc3a51de7d6140a37f0b06b082b8c2","row_sha256":"1176e707d52212efe4a01cc0c680f05ffac10f895743a6f755297e64ab15f122","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/5191","group_id":"mmlu/test/5191","variant":"clean"},"line_sha256":"3f9a4d74fc7cd14b5a55c3f492d899e07d17e3f788c3d32cbfe6f67d3ad1b725"}},{"id":"tv4_035","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"econometrics","question":"If there were a leverage effect in practice, what would be the shape of the news impact curve for as model that accounted for that leverage?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"It would rise more quickly for negative disturbances than for positive ones of the same magnitude","b":"It would be symmetrical about zero","c":"It would rise less quickly for negative disturbances than for positive ones of the same magnitude","d":"It would be zero for all positive disturbances"}}}},"media":null,"gold":{"answer":"a"},"note":"mmlu mmlu/test/1808","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":35,"src":{"answer":"mmlu"},"_meta":{"row":1808,"text_sha256":"f770a51b608eda349ee8aa4125bde15c974a7c110b05eec9df0f7392b935225a","row_sha256":"123bf64fbafcd2f6fec39cf01f12d33a9bf995d806e619aa81bc351e8915a2af","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1808","group_id":"mmlu/test/1808","variant":"clean"},"line_sha256":"be29f5314f080acdb814e8b7c9d28680b173f9bf365a72b0376940dd1ed6c928"}},{"id":"tv4_036","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"professional accounting","question":"Lemon owned 2000 shares of Spectrol Corp. common stock that were purchased in year 1 at $10.50 per share. In year 4 Lemon received a 5% non-taxable dividend of Spectrol common stock. In year 5 the stock split 2-for-1. In the current year Lemon sold 800 shares. What is Lemon's basis in the 800 shares of stock sold?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"$4,000","b":"$8,000","c":"$8,400","d":"16800"}}}},"media":null,"gold":{"answer":"a"},"note":"mmlu mmlu/test/10509","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":36,"src":{"answer":"mmlu"},"_meta":{"row":10509,"text_sha256":"f56bf336aa556755eac37356eee98ec47feea09017a3d173deca0090a0671730","row_sha256":"14383703de5b130b888fbfa3e419839116b75a84137a6acc94b9f9da1a536863","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/10509","group_id":"mmlu/test/10509","variant":"clean"},"line_sha256":"19bd2669d53da414a2e36b4ec9c7366252ee730b90dd29211cb8b9da8a0eb33a"}},{"id":"tv4_037","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"virology","question":"Which of the following is the most special characteristic of the adaptive immune system?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Antibodies","b":"T cells","c":"Memory T cells","d":"Clonal expansion of pre-existing antibodies and/or T cells"}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/13713","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":37,"src":{"answer":"mmlu"},"_meta":{"row":13713,"text_sha256":"5f54dd5414d94f4e5d9b41b3446a051b265c3b6c54c95b2e5ad393b9186d495c","row_sha256":"14910a65dfdb4ef6cf2108d5ff4522050ab8408c532b968aae2b7de7aa64727b","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13713","group_id":"mmlu/test/13713","variant":"clean"},"line_sha256":"9479ee8318444cb983c72daa6d782387464eeb918fa38914d08b11f0578f76e7"}},{"id":"tv4_038","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"econometrics","question":"What is the main difference between the Dickey Fuller (DF) and Phillips-Perron (PP) approaches to unit root testing?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"ADF is a single equation approach to unit root testing while PP is a systems approach","b":"PP tests reverse the DF null and alternative hypotheses so that there is stationarity under the null hypothesis of the PP test","c":"The PP test incorporates an automatic correction for autocorrelated residuals in the test regression","d":"PP tests have good power in small samples whereas DF tests do not."}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/1807","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":38,"src":{"answer":"mmlu"},"_meta":{"row":1807,"text_sha256":"63104fec8e11568f820894e0eb75b8c1e47b4bf437e5d78f65047e8c92ad6cfb","row_sha256":"149cc7e282a8181766092e95741e7d74584e56717f89aa15d7fb2de472b1776d","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1807","group_id":"mmlu/test/1807","variant":"clean"},"line_sha256":"3fdff0512549e464e5a1431c92838b0043e0a6a1c86b0c5f21a828c8cc679a8c"}},{"id":"tv4_039","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school microeconomics","question":"The total utility from sardines is maximized when they are purchased until"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"marginal utility is zero","b":"marginal benefit equals marginal cost","c":"consumer surplus is zero","d":"distributive efficiency is achieved"}}}},"media":null,"gold":{"answer":"a"},"note":"mmlu mmlu/test/4720","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":39,"src":{"answer":"mmlu"},"_meta":{"row":4720,"text_sha256":"d9f86719ffa9dc8954a412b58b7482a7dc5bd0ea176d0f0ec31398637cf1648b","row_sha256":"14e515224ce2587fad9cf05bbfab4b5e27f211f949db0c413468ecb5c63e2ea0","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/4720","group_id":"mmlu/test/4720","variant":"clean"},"line_sha256":"aea77b9e473e5b109a9433254ee27ae00b36e5a5b7d016bad19bc8f129377d56"}},{"id":"tv4_040","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"professional psychology","question":"You receive a call from Mrs. Wang who is very upset because her 14-year-old daughter witnessed the assault of her best friend three days ago and is very distraught. The girl hasn’t slept, won’t eat, and can’t stop crying. You have limited experience working with adolescents and in providing crisis intervention services. However, there is no one else in the community who is more experienced than you are. As an ethical psychologist, you will:"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"agree to see Mrs. Wang’s daughter in therapy since you’ve had some experience providing crisis intervention experiences.","b":"inform Mrs. Wang about your lack of experience and let her decide if she wants you to provide therapy to her daughter.","c":"inform Mrs. Wang that you cannot see her daughter because of your lack of experience.","d":"see Mrs. Wang’s daughter in therapy only until the crisis has ended or until you locate alternative services."}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/12568","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":40,"src":{"answer":"mmlu"},"_meta":{"row":12568,"text_sha256":"a3a223bcafcf86355939d886cbe5cffcd1499240ff3007125d1629cfb6e6ce05","row_sha256":"15ee481e12ac1182a5e6741fc1dc006dd5acd23711420945c5cfddec3de8c7e5","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12568","group_id":"mmlu/test/12568","variant":"clean"},"line_sha256":"0968ec65c31f28779888feaaf4b7345c652623c8548b7c67cdc28393d688d39a"}},{"id":"tv4_041","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"professional law","question":"A victim was standing on a street corner waiting to make a drug buy. The drug dealer was always late, so the victim continued to wait outside in the freezing cold weather because he needed a quick fix. Suddenly a car rounded the street corner and a shot rang out. The victim slumped over and fell to the sidewalk. Immediately after being shot and before he died, the victim exclaimed, \"The defendant did it!\" The defendant was indicted and is now on trial for the victim's murder. Based on these facts alone, the admissibility of the victim's statement should be determined by"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"the jury, after a preliminary determination by the judge.","b":"the jury, without a preliminary determination by the judge.","c":"the judge, with instruction to the jury that it may consider the dying declaration if it determines that the victim knew he was dying.","d":"the judge, without assistance from the jury."}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/12163","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":41,"src":{"answer":"mmlu"},"_meta":{"row":12163,"text_sha256":"357bec3dde33cccb1b6d97f805743e7804c397287ffbe3cfc449c9f3ce049e8c","row_sha256":"16592e464ce99da5a7f2f2e16f0251575aea7595e69d2fc1092656e96904297e","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12163","group_id":"mmlu/test/12163","variant":"clean"},"line_sha256":"609a36b8975910a716c7cfbd056375cd7645e45b6f67dcd4d59d18bfb4213aaa"}},{"id":"tv4_042","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"logical fallacies","question":"The fallacy of guilt by association is a specific type of"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Fallacy of accident","b":"Laudatory personality","c":"Begging the question","d":"Circular reasoning"}}}},"media":null,"gold":{"answer":"a"},"note":"mmlu mmlu/test/6773","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":42,"src":{"answer":"mmlu"},"_meta":{"row":6773,"text_sha256":"ad92e1941705bfa22d357ace87e80108d4f552db2d04d78c4c7dbcff69e62991","row_sha256":"16f57b256c903af68e1f29638cdea0c98d34cdfe847a6899b3974dfceb24522f","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6773","group_id":"mmlu/test/6773","variant":"clean"},"line_sha256":"bc7ced7f048db613944f55accd46cf315f91444881b02b79b5bfab2929dde9d0"}},{"id":"tv4_043","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"electrical engineering","question":"Electrons in p-type material of a semi-conductor are called as"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"either minority carriers or majority carriers","b":"minority carriers","c":"majority carriers","d":"valance carriers"}}}},"media":null,"gold":{"answer":"b"},"note":"mmlu mmlu/test/1982","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":43,"src":{"answer":"mmlu"},"_meta":{"row":1982,"text_sha256":"d9ffe57f3bac0b6b5e36685e5fc5c980ca2c277cafee5911504bac119dc8a19b","row_sha256":"173b236e9e6235053ace7bfa297ab81a9dc1d8098f84e6605736ea0497db23d8","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1982","group_id":"mmlu/test/1982","variant":"clean"},"line_sha256":"bcc0b07bbdace8dc92fe41bb4000cf534b39b6eab6dc9a83658b4a4af5e46e6c"}},{"id":"tv4_044","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school government and politics","question":"The spoils system of awarding civil service jobs was replaced by the merit system as a result of the"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Truman Doctrine","b":"Fair Labor Standards Act of 1961","c":"National Industrial Recovery Act","d":"Pendleton Act of 1883"}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/3758","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":44,"src":{"answer":"mmlu"},"_meta":{"row":3758,"text_sha256":"209b997de0aab123d791890481a71f42a3242b0de191ce264d3cc16192c12b6e","row_sha256":"173e90ff9c8e472cbc5633766bc1e4310718fa6cfc5738327921b238c04c4b16","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/3758","group_id":"mmlu/test/3758","variant":"clean"},"line_sha256":"b6fe8c36da22f3a68479e11f7873243ed6934433035ee60c8c3011b29c498314"}},{"id":"tv4_045","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"econometrics","question":"Consider the following model for $y_t$:\n\n$y_t = \\mu + \\lambda t + u_t$\n\nWhich one of the following most accurately describes the process for $y_t$?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"A unit root process","b":"A stationary process","c":"A deterministic trend process","d":"A random walk with drift"}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/1857","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":45,"src":{"answer":"mmlu"},"_meta":{"row":1857,"text_sha256":"d1c63be268917217f4bf97cdd3bce1c57767d755fe12f1771a173baaf1eb0455","row_sha256":"1742bf69e66dfd9afd72eca8a80efdfea705670aa814f7c847d3d1a3e82d5bae","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/1857","group_id":"mmlu/test/1857","variant":"clean"},"line_sha256":"2d2cad279b11f0b10e6f488c94b48d030e523916ca07806dc14ec6eea4f7c9ca"}},{"id":"tv4_046","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school biology","question":"This process leads to the net production of two pyruvate, two ATP, and two NADH."},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Glycolysis","b":"Chemiosmosis","c":"Fermentation","d":"Calvin cycle"}}}},"media":null,"gold":{"answer":"a"},"note":"mmlu mmlu/test/2806","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":46,"src":{"answer":"mmlu"},"_meta":{"row":2806,"text_sha256":"47d3aa72b23e7d16cbb68a5870a03acdf468c693fccc545e641995d1b9d9b59a","row_sha256":"1797ea252aabf8e7b5d92080fb1c53b73873154cfa2f9e0ddedaae07c3b25986","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/2806","group_id":"mmlu/test/2806","variant":"clean"},"line_sha256":"09744b9091d552ca5da1f8a2594b21082be690afd31838073978bc63d968a308"}},{"id":"tv4_047","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"professional psychology","question":"A woman transmits color blindness to her son although she herself displays normal color vision. The defect transmitted by this woman represents a characteristic of her"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Y chromosome","b":"dominant genes","c":"genotype","d":"phenotype"}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/12630","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":47,"src":{"answer":"mmlu"},"_meta":{"row":12630,"text_sha256":"c4fbf107fefb1392e03a9e502108a2e09e06a2848b1b1e732af643146f9ae716","row_sha256":"17b6e12c658cc0b30180f9170090381c8cf88caf34f84ef346ea61c0076d328f","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/12630","group_id":"mmlu/test/12630","variant":"clean"},"line_sha256":"e52bb207c5a1def7483a9260dadc9490e51d564a4cfb6a8f100adc005958cdc6"}},{"id":"tv4_048","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"moral scenarios","question":"For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I urinated into a cup so I could test it for infection. Scenario 2 | I fed the hungry child some warm soup."},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Wrong, Wrong","b":"Wrong, Not wrong","c":"Not wrong, Wrong","d":"Not wrong, Not wrong"}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/8523","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":48,"src":{"answer":"mmlu"},"_meta":{"row":8523,"text_sha256":"c2d9284847102813614e2479684b859b9ad6a71aaf310c15851ed05a0c67e047","row_sha256":"17f07d503ffa0494aa586d7a4a18911b4b0d62da344c0dca33d1f6be0a925804","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/8523","group_id":"mmlu/test/8523","variant":"clean"},"line_sha256":"629f8fbf15fcb2da3b9bd0b5d7cfdafa09f9ffcf990d1e6f7f8270bccd9d1017"}},{"id":"tv4_049","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"public relations","question":"Which statement is true about focus group research?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"The participants should be selected at random.","b":"It is an example of a formal research tool for practitioners.","c":"The information can be generalized to describe a larger population.","d":"The findings generate qualitative data that can be used to develop surveys."}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/13067","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":49,"src":{"answer":"mmlu"},"_meta":{"row":13067,"text_sha256":"0b689018d753bf1e0bab2566ca49fd325829eb7f0cce0eaf324421c61f7eb07b","row_sha256":"18d05dc22cd35d391f093924dbfcf17482a64e1d2d9780a90fd9b43ca842618f","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/13067","group_id":"mmlu/test/13067","variant":"clean"},"line_sha256":"a4ce6e56a58b39a67368f28462029eadd354f148b62ce43f8717a3fc7780a10f"}},{"id":"tv4_050","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"elementary mathematics","question":"Solve the equation −17.02 = −2.3y. Check your answer."},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"7.4","b":"−7.4","c":"−19.32","d":"14.72"}}}},"media":null,"gold":{"answer":"a"},"note":"mmlu mmlu/test/2282","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":50,"src":{"answer":"mmlu"},"_meta":{"row":2282,"text_sha256":"ca192f34bda6eb5a55ae413e76503d8d9a8481bf2c86e8bad24fcf2cadfd75d2","row_sha256":"18dffaae14139a6757aa0e55718041a4ea0795baf3588fae0d85de5093163620","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/2282","group_id":"mmlu/test/2282","variant":"clean"},"line_sha256":"4ec5a97e18360e8ee46499c742858b2239e52576e7d635ba6e78632358f4c35c"}},{"id":"tv4_051","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"prehistory","question":"The argument in favor of hominids making and using stone tools by at least 3.3 million years ago is supported by:"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"cut marks on animal bones that were made by butchering with pieces of stone.","b":"the anatomy of Australopithecus africanus hands and analogies with studies of living non-human primates that demonstrate tool use.","c":"simple stone cores, anvil stones, and flakes found at the Lomekwi 3 site in West Turkana, Kenya.","d":"all of the above."}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/10179","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":51,"src":{"answer":"mmlu"},"_meta":{"row":10179,"text_sha256":"ea4396ca54518fad07b8b5961da2009fae7b812110014f09cb227ca35c027a9c","row_sha256":"1953357a0b60b7cc6360565c6031204ac39ec21a61b1f3fb6c3cf9af4632a3cb","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/10179","group_id":"mmlu/test/10179","variant":"clean"},"line_sha256":"cebedd3e0f613a4ac663adece2d01d91c41aca8ea6bb66f9cade23aad6035214"}},{"id":"tv4_052","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school chemistry","question":"The net ionic equation expected when solutions of NH4Br and AgNO3 are mixed together is"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Ag+(aq) + Br-(aq) → AgBr(s)","b":"NH4+(aq) + Ag+(aq) → Ag(NH4)3+2(aq)","c":"Br-(aq) + NO3-(aq) → NO3Br(aq)","d":"NH4Br(aq) + NO3-(aq) → NH4NO3(aq) + Br-(aq)"}}}},"media":null,"gold":{"answer":"a"},"note":"mmlu mmlu/test/2982","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":52,"src":{"answer":"mmlu"},"_meta":{"row":2982,"text_sha256":"98874b205296b4f10427e4fd13b34e861d28b2cc1eb15e895246a288e4223ed3","row_sha256":"197fc93a439226b164e6b58a655da65fbb98aa5fa9efd63e0d8adcae40ae2b9a","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/2982","group_id":"mmlu/test/2982","variant":"clean"},"line_sha256":"faa322f86bb627d8b77c5dc580be71498074247068883bbf18792998663a6823"}},{"id":"tv4_053","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"professional law","question":"An owner operated a successful retail business. He decided he wanted to retire and listed his business for sale. A buyer saw the listing and visited the retail business. The buyer told the owner that she was very interested in buying the business, but that she needed to get a loan from the bank to finance the purchase. The parties drafted a contract, signed by both parties, which stated that the buyer agreed to purchase the retail business from the owner for $250,000. The written agreement contained a provision wherein the sale would not take effect \"unless the buyer is successful in her bid to obtain a loan from a bank to finance the purchase of the business. \" Subsequently, the buyer made no effort to attempt to obtain a loan from a bank so that the sale could be finalized. After several months, the owner filed suit against the buyer for breach of contract. Which of the following will be the owner's strongest argument in support of his action against the buyer for breach of contract?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"The obtaining of a loan from a bank was not worded as a \"condition\" to the buyer's duty to buy the business.","b":"Although obtaining a loan from a bank was a condition to the buyer's duty to buy the business, the condition should be excused because its non-occurrence would cause a forfeiture to the owner.","c":"Although obtaining a loan from a bank was a condition to the buyer's duty to buy the business, it should be stricken from the contract because it is an unconscionable term.","d":"The buyer breached an implied promise to make a reasonable effort to obtain a loan from a bank."}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/10687","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":53,"src":{"answer":"mmlu"},"_meta":{"row":10687,"text_sha256":"8ad6ed1ee05fb239397328c42c3635969b0116c8cd706783db3b539d69442150","row_sha256":"1afab0887f5eed8238a62ad38710101c9a670e8fc7c0e297ee8870bc7da16ca5","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/10687","group_id":"mmlu/test/10687","variant":"clean"},"line_sha256":"90353a7da2b83810b8456a350a29e316df9128ab7c03722b50ab34c4e0032acc"}},{"id":"tv4_054","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"college chemistry","question":"Calculate the spin angular momentum of 43Ca. [I = 7⁄2]"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"2.166 x 10^-34 J s","b":"3.691 x 10^-34 J s","c":"4.185 x 10^-34 J s","d":"5.493 x 10^-34 J s"}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/931","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":54,"src":{"answer":"mmlu"},"_meta":{"row":931,"text_sha256":"bffdafb5dfcb1e38747797c28c9240ba377547c3a3b7918284db1bf3bcf291f0","row_sha256":"1b19c3c4efa3abc6ba91131741026207079d43fb02ec9d8da8d87d08eeae05ea","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/931","group_id":"mmlu/test/931","variant":"clean"},"line_sha256":"959e55bf1d932e38ed0968c99a9ec1264ab3468ccf6031c843b6b4f1f214d969"}},{"id":"tv4_055","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"professional accounting","question":"Ultra Co. uses a periodic inventory system. The following are inventory transactions for the month of January: 1/1 Beginning inventory 20000 units at $13 1/20 Purchase 30000 units at $15 1/23 Purchase 40000 units at $17 1/31 Sales at $20 per unit 50000 units Ultra uses the LIFO method to determine the value of its inventory. What amount should Ultra report as cost of goods sold on its income statement for the month of January?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"$710,000","b":"$750,000","c":"$830,000","d":"1000000"}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/10372","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":55,"src":{"answer":"mmlu"},"_meta":{"row":10372,"text_sha256":"600f6c0edf1596b591557e7d464d168fee58a76d37bccd04899ea67277ca8440","row_sha256":"1d98a7be5205e8cdca077a6afd8d4b19c680d2b6dbeed56307facafc6211e788","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/10372","group_id":"mmlu/test/10372","variant":"clean"},"line_sha256":"7b687e66af486614648d8cd236bf7caf49938061890f1551b010a220fe26c3f1"}},{"id":"tv4_056","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school geography","question":"Which of the following is NOT a feature of an export processing zone (EPZ)?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Tariff reductions on exports","b":"Raw materials imported duty-free","c":"Exemption from customs duties","d":"Strict pollution regulations"}}}},"media":null,"gold":{"answer":"d"},"note":"mmlu mmlu/test/3523","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":56,"src":{"answer":"mmlu"},"_meta":{"row":3523,"text_sha256":"0a55d50ec128876b1b93a1817c50be49da2b8d9ba9da57fe6b8e68bc7fc1f804","row_sha256":"1f67a2db1cb8de1eb8994cb200cca72336ee543d0d712feeac0f7377fc9ccc0b","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/3523","group_id":"mmlu/test/3523","variant":"clean"},"line_sha256":"ca4d4d000f48001968f8ec680408cf392bcac5adb9622b4396f0bddcf1261e07"}},{"id":"tv4_057","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"human aging","question":"Which of the following is NOT one of the more frequently named activities by those who are retired?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Volunteering","b":"Travel","c":"Redoing the home","d":"Household chores"}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/6245","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":57,"src":{"answer":"mmlu"},"_meta":{"row":6245,"text_sha256":"232d792150da6346024c6becbeb1ed6e41d7b8225d3afc5408dc51c7d2883b3f","row_sha256":"1fa4a3b6a91ab2aeb7bb098782fa05e9bad721b7a1db556e4fbb62d55ab70c04","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/6245","group_id":"mmlu/test/6245","variant":"clean"},"line_sha256":"c22f44bec87abf1b40d2db9595caa0a4088c772969cef65598625fe10127f39d"}},{"id":"tv4_058","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"moral scenarios","question":"For which of these two scenarios does the main character (who uses I/me/my) do something clearly morally wrong, according to ordinary moral standards in the US as of 2020? Scenario 1 | I helped my friend get out of the house after he lost his girlfriend. Scenario 2 | I went skinny-dipping in the community pool, while everyone was at the poolside."},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Wrong, Wrong","b":"Wrong, Not wrong","c":"Not wrong, Wrong","d":"Not wrong, Not wrong"}}}},"media":null,"gold":{"answer":"c"},"note":"mmlu mmlu/test/9182","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":58,"src":{"answer":"mmlu"},"_meta":{"row":9182,"text_sha256":"c041a95d73eaf3622c37781715d7651e4e11b576acfd78959990153d966a28b4","row_sha256":"202af939f40c369cf54f711791efd619354fd75d0a8159f50bff9d69acd88401","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/9182","group_id":"mmlu/test/9182","variant":"clean"},"line_sha256":"3aa640a4bbf743d724bdb741ab8dc613c5cc4100d1c20dead76a6eccbbd12cbe"}},{"id":"tv4_059","source":"transfer_v4_dev_head","public":true,"request":{"state":{"subject":"high school psychology","question":"Which of the following is often true of memory recall?"},"questions":{"answer":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"People are more likely to recall information that is congruent with their prior schemas.","b":"People will not recall information unless it is stored in working memory.","c":"People have no ability to recall information that has been extinguished.","d":"People will recall nonsense syllables more rapidly than they will sets of numbers."}}}},"media":null,"gold":{"answer":"a"},"note":"mmlu mmlu/test/5150","provenance":{"file":"evals/v4/transfer-v4/development.jsonl","line":59,"src":{"answer":"mmlu"},"_meta":{"row":5150,"text_sha256":"521d509885166a58de2627c79846a9aa42a22d22d83f8ffcfadf1e28d3e106e3","row_sha256":"2036db6f4f015742ccc14fe5697c9e51c566c5dd072f2b40b6f44fd892f4b76c","source":"mmlu","repo":"cais/mmlu","revision":"c30699e8356da336a370243923dbaf21066bb9fe","split":"test","id":"mmlu/test/5150","group_id":"mmlu/test/5150","variant":"clean"},"line_sha256":"0f8cfb484311049160b9dcb55d9351563aa6530ec45c142c228ce42f8bdd02ca"}},{"id":"semif_a3f18f3a63d45345942b","source":"semif_authored144","public":true,"request":{"state":"The optician ordered replacement lenses. The workshop confirms they have not yet been fitted to the customer's glasses.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the replacement lenses have been fitted.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=2fdaa8a61e6e989dd866","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":0,"line_sha256":"29b8ab32b468adfba83998a9f0161d285d4ad10a4dfd8fe310f40ae2f5a64567","semif_id":"a3f18f3a63d45345942b","family":"evidence_interpretation","variant":"original","group_id":"2fdaa8a61e6e989dd866","label":2,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_f40beba9088c8db8bbd6","source":"semif_authored144","public":true,"request":{"state":"The optician ordered replacement lenses. The workshop confirms they have not yet been fitted to the customer's glasses.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: replacement lenses were ordered.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=2fdaa8a61e6e989dd866","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":1,"line_sha256":"4ac1b71277f4c327373bff0295ba8fc36c0d2efc3cf3e5b8e3aeff3b43d81d87","semif_id":"f40beba9088c8db8bbd6","family":"evidence_interpretation","variant":"criterion_reversal","group_id":"2fdaa8a61e6e989dd866","label":0,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_0b43ea8e24e74d621f1b","source":"semif_authored144","public":true,"request":{"state":"The workshop confirms it fitted the replacement lenses to the customer's glasses.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the replacement lenses have been fitted.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=2fdaa8a61e6e989dd866","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":2,"line_sha256":"3ec20897f955bcd3f7eafbbfc3c5046e56828a44e61bc910c2e6501a91317553","semif_id":"0b43ea8e24e74d621f1b","family":"evidence_interpretation","variant":"evidence_change","group_id":"2fdaa8a61e6e989dd866","label":1,"option_order":["insufficient","supported","contradicted"]}},{"id":"semif_942ec65dac0bfb9874f9","source":"semif_authored144","public":true,"request":{"state":"The optician records a glasses prescription but no order or fitting status.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the replacement lenses have been fitted.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=2fdaa8a61e6e989dd866","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":3,"line_sha256":"0f9aaf4ac9d9718cf8bdb0d1a5f8d9a14ef51bace747b9c4c4146165d3e829cb","semif_id":"942ec65dac0bfb9874f9","family":"evidence_interpretation","variant":"missing","group_id":"2fdaa8a61e6e989dd866","label":0,"option_order":["insufficient","supported","contradicted"]}},{"id":"semif_e0c140e9222d1bce5647","source":"semif_authored144","public":true,"request":{"state":"The guest writes, \"The listing says towels are supplied. I agree: fresh towels were provided during my stay.\"","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the guest endorses the listing's towel statement.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=original group=f4bea3d3bcc01056740d","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":4,"line_sha256":"ee9d8c375b6acc354f7420edd5d92646d64e60c9bc78eeb549283a1e430680ff","semif_id":"e0c140e9222d1bce5647","family":"evidence_interpretation","variant":"original","group_id":"f4bea3d3bcc01056740d","label":0,"option_order":["supported","contradicted","insufficient"]}},{"id":"semif_8e8c3804a3c15ebb31e3","source":"semif_authored144","public":true,"request":{"state":"The guest writes, \"The listing says towels are supplied. I agree: fresh towels were provided during my stay.\"","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the guest disputes the towel statement.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=criterion_reversal group=f4bea3d3bcc01056740d","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":5,"line_sha256":"d04fb857b655f9545341ef8874d1ac7adb53915cda8c0a95bb83bd9e18f9c914","semif_id":"8e8c3804a3c15ebb31e3","family":"evidence_interpretation","variant":"criterion_reversal","group_id":"f4bea3d3bcc01056740d","label":2,"option_order":["insufficient","supported","contradicted"]}},{"id":"semif_56c419b12cd37cbe790c","source":"semif_authored144","public":true,"request":{"state":"The guest writes, \"The listing says towels are supplied, but none were provided. I dispute that statement.\"","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the guest endorses the listing's towel statement.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=evidence_change group=f4bea3d3bcc01056740d","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":6,"line_sha256":"349e0655f52d59d16af361363c7409ec2c2bbd940b1b76e2670d848b311469f2","semif_id":"56c419b12cd37cbe790c","family":"evidence_interpretation","variant":"evidence_change","group_id":"f4bea3d3bcc01056740d","label":0,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_75b470586834df8c568d","source":"semif_authored144","public":true,"request":{"state":"The guest forwards the listing but provides no comment about towels or the stay.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the guest endorses the listing's towel statement.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=f4bea3d3bcc01056740d","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":7,"line_sha256":"6f393466824b6b141fa200643f214eab240c4900031248bb95dddc1de96afe85","semif_id":"75b470586834df8c568d","family":"evidence_interpretation","variant":"missing","group_id":"f4bea3d3bcc01056740d","label":2,"option_order":["supported","contradicted","insufficient"]}},{"id":"semif_834f3b3e10cc61b33d25","source":"semif_authored144","public":true,"request":{"state":"The technician says terminal Cedar is online. Terminal Maple is explicitly offline.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: terminal Maple is online.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=b51b300da2936decc6b7","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":8,"line_sha256":"76b1450aa3220b19b8ce53fefccda80a38b3eb8d7cbc0ca8400101e847e8db2b","semif_id":"834f3b3e10cc61b33d25","family":"evidence_interpretation","variant":"original","group_id":"b51b300da2936decc6b7","label":2,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_76c7a79b1ff2363a60c5","source":"semif_authored144","public":true,"request":{"state":"The technician says terminal Cedar is online. Terminal Maple is explicitly offline.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: terminal Cedar is online.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=b51b300da2936decc6b7","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":9,"line_sha256":"9c62ce7a7f27588def2dbb1b9ff59f23ee91ce9eb28ea07dbc63fe353ac47628","semif_id":"76c7a79b1ff2363a60c5","family":"evidence_interpretation","variant":"criterion_reversal","group_id":"b51b300da2936decc6b7","label":0,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_36c71bb649b7d5eccb1f","source":"semif_authored144","public":true,"request":{"state":"The technician confirms terminal Maple is online and terminal Cedar is offline.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: terminal Maple is online.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=b51b300da2936decc6b7","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":10,"line_sha256":"ae4e718b2c02a7701141321eb3b593d0affa2a859c0c4743609abb3e870737eb","semif_id":"36c71bb649b7d5eccb1f","family":"evidence_interpretation","variant":"evidence_change","group_id":"b51b300da2936decc6b7","label":0,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_eac5121a4a1bf391a08c","source":"semif_authored144","public":true,"request":{"state":"The technician lists both terminals but omits their connection status.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: terminal Maple is online.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=b51b300da2936decc6b7","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":11,"line_sha256":"d1a5522627e5139c6006816eb3ba1c08ef1773d89f080fc5fa70fa4611f4ed34","semif_id":"eac5121a4a1bf391a08c","family":"evidence_interpretation","variant":"missing","group_id":"b51b300da2936decc6b7","label":1,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_44574c44e243ee1b2f52","source":"semif_authored144","public":true,"request":{"state":"The catalogue originally dated the letter to June. Its author issued a correction stating July was correct and June was mistaken.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the corrected date is in July.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=original group=e4cfd694c80e08d3e6a5","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":12,"line_sha256":"cdddec91b355222b28583f322688f4fa77b2e8ee81412acd5b6580ab6e58f908","semif_id":"44574c44e243ee1b2f52","family":"evidence_interpretation","variant":"original","group_id":"e4cfd694c80e08d3e6a5","label":2,"option_order":["insufficient","contradicted","supported"]}},{"id":"semif_2608370d01783735c3ea","source":"semif_authored144","public":true,"request":{"state":"The catalogue originally dated the letter to June. Its author issued a correction stating July was correct and June was mistaken.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the corrected date is in June.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=criterion_reversal group=e4cfd694c80e08d3e6a5","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":13,"line_sha256":"65202a7c669b4c8cfb51dfec194db6cc276eb53faf4d4d844d54c2172fd77e66","semif_id":"2608370d01783735c3ea","family":"evidence_interpretation","variant":"criterion_reversal","group_id":"e4cfd694c80e08d3e6a5","label":2,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_ec94ef909f7918695ab3","source":"semif_authored144","public":true,"request":{"state":"The catalogue author corrects the date to June and expressly rejects the earlier July date.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the corrected date is in July.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=evidence_change group=e4cfd694c80e08d3e6a5","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":14,"line_sha256":"7e868b0a7eb30ff0aee625f4a03ab859682dde6dcbd30266d8eadb5e927b112e","semif_id":"ec94ef909f7918695ab3","family":"evidence_interpretation","variant":"evidence_change","group_id":"e4cfd694c80e08d3e6a5","label":2,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_03d919fd0e9b3a7cd1e6","source":"semif_authored144","public":true,"request":{"state":"The catalogue mentions a date correction but gives no corrected date.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the corrected date is in July.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=e4cfd694c80e08d3e6a5","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":15,"line_sha256":"f71db05caba1ba82d5dae19a020f1d1e42da9ee9e92d740c2a8916ce5f69b11d","semif_id":"03d919fd0e9b3a7cd1e6","family":"evidence_interpretation","variant":"missing","group_id":"e4cfd694c80e08d3e6a5","label":0,"option_order":["insufficient","supported","contradicted"]}},{"id":"semif_86a15618ba4cb90a307e","source":"semif_authored144","public":true,"request":{"state":"The intern attached instructions for weighing the parcel and explicitly says no weighing has been performed.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the parcel has been weighed.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=20460d4ae0db46091e84","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":16,"line_sha256":"e961d35381dda6fbc265a61f30504a82a683402af31886df9587524c3a7617b2","semif_id":"86a15618ba4cb90a307e","family":"evidence_interpretation","variant":"original","group_id":"20460d4ae0db46091e84","label":0,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_3dd9c3ac98f8ddf7bdd3","source":"semif_authored144","public":true,"request":{"state":"The intern attached instructions for weighing the parcel and explicitly says no weighing has been performed.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the document contains weighing instructions.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=20460d4ae0db46091e84","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":17,"line_sha256":"9562948573ca3dca83a4f1fbf27c92833506d1e25101a05f65bb8ee0babfcdf4","semif_id":"3dd9c3ac98f8ddf7bdd3","family":"evidence_interpretation","variant":"criterion_reversal","group_id":"20460d4ae0db46091e84","label":1,"option_order":["contradicted","supported","insufficient"]}},{"id":"semif_36273dd4777876a26338","source":"semif_authored144","public":true,"request":{"state":"The intern performed the weighing and recorded the parcel's measured weight.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the parcel has been weighed.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=20460d4ae0db46091e84","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":18,"line_sha256":"a7d05364023ebd6b3f2345ee8a27858e9eb847ad13efd01abb622534d0c33747","semif_id":"36273dd4777876a26338","family":"evidence_interpretation","variant":"evidence_change","group_id":"20460d4ae0db46091e84","label":1,"option_order":["contradicted","supported","insufficient"]}},{"id":"semif_07ee70cfe4b01330ce89","source":"semif_authored144","public":true,"request":{"state":"The intern supplies a parcel identifier but no procedure or measurement information.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the parcel has been weighed.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=20460d4ae0db46091e84","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":19,"line_sha256":"533fb6800d779dd8e7dc311c2306a94d9e677cd44869c5ad7b82811b09d3bcf8","semif_id":"07ee70cfe4b01330ce89","family":"evidence_interpretation","variant":"missing","group_id":"20460d4ae0db46091e84","label":2,"option_order":["supported","contradicted","insufficient"]}},{"id":"semif_51ceab3ce622c7c759f1","source":"semif_authored144","public":true,"request":{"state":"During a puppet show, a character announces the discovery of a dragon egg. The production note states this is fictional and no real egg was discovered.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the egg discovery was fictional.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=original group=51b91ba31984c5939c1c","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":20,"line_sha256":"19782c41e501d7ebdbf94bdad95f8b87112cc5f7ade586d3fecb50ebb17d1534","semif_id":"51ceab3ce622c7c759f1","family":"evidence_interpretation","variant":"original","group_id":"51b91ba31984c5939c1c","label":2,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_b8091be929b1e489fcc4","source":"semif_authored144","public":true,"request":{"state":"During a puppet show, a character announces the discovery of a dragon egg. The production note states this is fictional and no real egg was discovered.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: a real egg was discovered.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"media":null,"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=criterion_reversal group=51b91ba31984c5939c1c","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":21,"line_sha256":"5d90512f35f0acf6cc6d77b69fb8afde56ad13a2760c3a3c617eb7a21c1d699a","semif_id":"b8091be929b1e489fcc4","family":"evidence_interpretation","variant":"criterion_reversal","group_id":"51b91ba31984c5939c1c","label":1,"option_order":["supported","contradicted","insufficient"]}},{"id":"semif_c89b86e3f55123bfa506","source":"semif_authored144","public":true,"request":{"state":"The note describes a real bird egg discovered outside a performance and expressly says the discovery was not fictional.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the egg discovery was fictional.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"media":null,"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=evidence_change group=51b91ba31984c5939c1c","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":22,"line_sha256":"c54b46a9d2ae82f50224c67196af5e61ad3dd1ea1dfd67ba7285a3782b4f1f97","semif_id":"c89b86e3f55123bfa506","family":"evidence_interpretation","variant":"evidence_change","group_id":"51b91ba31984c5939c1c","label":1,"option_order":["supported","contradicted","insufficient"]}},{"id":"semif_3b0958cabc339005cf66","source":"semif_authored144","public":true,"request":{"state":"A note mentions an egg without describing a discovery or whether the account is fictional.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the egg discovery was fictional.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=51b91ba31984c5939c1c","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":23,"line_sha256":"a70dec8b593bda8561766b8997d1c67b7b77bac3b68cbfe63cce2180800d461f","semif_id":"3b0958cabc339005cf66","family":"evidence_interpretation","variant":"missing","group_id":"51b91ba31984c5939c1c","label":0,"option_order":["insufficient","supported","contradicted"]}},{"id":"semif_f2b4ec4930322fe45118","source":"semif_authored144","public":true,"request":{"state":"The architect permits reproducing a plan in an internal training booklet and expressly forbids selling copies.","questions":{"decision":{"type":"choice","instructions":"Sell copies only with the architect's sales permission. May copies be sold?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"media":null,"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=770c1607f0a11f88f81e","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":24,"line_sha256":"6d8455b84b5dc54273b2b3339ff883c772369514195228f70588de34550b5149","semif_id":"f2b4ec4930322fe45118","family":"rule_application","variant":"original","group_id":"770c1607f0a11f88f81e","label":1,"option_order":["insufficient","prohibited","permitted"]}},{"id":"semif_61ac20b07a5189231e15","source":"semif_authored144","public":true,"request":{"state":"The architect permits reproducing a plan in an internal training booklet and expressly forbids selling copies.","questions":{"decision":{"type":"choice","instructions":"Internal training reproduction is allowed when the architect permits that use. May the plan appear in the training booklet?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=770c1607f0a11f88f81e","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":25,"line_sha256":"691bc05476ac4d932caea0932682f0b3f36fa71aec4943f4dd64b220f27a6598","semif_id":"61ac20b07a5189231e15","family":"rule_application","variant":"criterion_reversal","group_id":"770c1607f0a11f88f81e","label":1,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_4b3d794c004566610181","source":"semif_authored144","public":true,"request":{"state":"The architect expressly permits selling copies of the plan.","questions":{"decision":{"type":"choice","instructions":"Sell copies only with the architect's sales permission. May copies be sold?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=770c1607f0a11f88f81e","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":26,"line_sha256":"538c612d90b7bf9c593bbf4d6d800603851735619c737db79576f0dde0eddc5f","semif_id":"4b3d794c004566610181","family":"rule_application","variant":"evidence_change","group_id":"770c1607f0a11f88f81e","label":2,"option_order":["prohibited","insufficient","permitted"]}},{"id":"semif_b7c4e62d07986286fce0","source":"semif_authored144","public":true,"request":{"state":"An architect's permission letter exists, but the uses it allows are omitted.","questions":{"decision":{"type":"choice","instructions":"Sell copies only with the architect's sales permission. May copies be sold?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=770c1607f0a11f88f81e","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":27,"line_sha256":"4377841ac694a8560da15ba4a1806c65d7eadae2a049789cc592ad4dc4f1ecf0","semif_id":"b7c4e62d07986286fce0","family":"rule_application","variant":"missing","group_id":"770c1607f0a11f88f81e","label":0,"option_order":["insufficient","permitted","prohibited"]}},{"id":"semif_9473603b44af63beab79","source":"semif_authored144","public":true,"request":{"state":"The guide completed first-aid training but not mountain-navigation training.","questions":{"decision":{"type":"choice","instructions":"The first-aid badge requires only first-aid training. May this guide receive it?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=original group=ffd081ada3bd9be9207a","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":28,"line_sha256":"78bf3deaa931770bb9559d841f27b4bf9726a925240b3a1b419d76a8e3fc44da","semif_id":"9473603b44af63beab79","family":"rule_application","variant":"original","group_id":"ffd081ada3bd9be9207a","label":0,"option_order":["permitted","insufficient","prohibited"]}},{"id":"semif_d41072dc35d8cd14c056","source":"semif_authored144","public":true,"request":{"state":"The guide completed first-aid training but not mountain-navigation training.","questions":{"decision":{"type":"choice","instructions":"Under the alternative rule, the badge requires both first-aid and mountain-navigation training. May this guide receive it?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"prohibited"},"note":"family=rule_application variant=criterion_reversal group=ffd081ada3bd9be9207a","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":29,"line_sha256":"dc9d8d209c1446787c544c6eb8084619438d667bf170c61286828a982efca928","semif_id":"d41072dc35d8cd14c056","family":"rule_application","variant":"criterion_reversal","group_id":"ffd081ada3bd9be9207a","label":1,"option_order":["permitted","prohibited","insufficient"]}},{"id":"semif_33d6e5da58f0fee2490d","source":"semif_authored144","public":true,"request":{"state":"The guide completed mountain-navigation training but not first-aid training.","questions":{"decision":{"type":"choice","instructions":"The first-aid badge requires only first-aid training. May this guide receive it?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"media":null,"gold":{"decision":"prohibited"},"note":"family=rule_application variant=evidence_change group=ffd081ada3bd9be9207a","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":30,"line_sha256":"96bcadb9989a5c0102c65f37f6b6faa10f1baae9b1be69436a5ec7464777ca36","semif_id":"33d6e5da58f0fee2490d","family":"rule_application","variant":"evidence_change","group_id":"ffd081ada3bd9be9207a","label":0,"option_order":["prohibited","insufficient","permitted"]}},{"id":"semif_823bf736b92e9d8fa66d","source":"semif_authored144","public":true,"request":{"state":"The guide registered for training, but completion information is not supplied.","questions":{"decision":{"type":"choice","instructions":"The first-aid badge requires only first-aid training. May this guide receive it?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=ffd081ada3bd9be9207a","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":31,"line_sha256":"765beadff20a28e6d0841e473392c30fb786c7a0610017fcb5a7c66fe4f3d1f0","semif_id":"823bf736b92e9d8fa66d","family":"rule_application","variant":"missing","group_id":"ffd081ada3bd9be9207a","label":1,"option_order":["prohibited","insufficient","permitted"]}},{"id":"semif_8a4c3de28be95c105ca1","source":"semif_authored144","public":true,"request":{"state":"The designer approves making a sample badge but expressly withholds approval for mass production.","questions":{"decision":{"type":"choice","instructions":"Mass production is allowed only after mass-production approval. May it begin?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"media":null,"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=2c3a0cb61f7bae78739e","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":32,"line_sha256":"560f828590ab917169978b2220ad1f8706dd25fefef175d34b1e715a957492a2","semif_id":"8a4c3de28be95c105ca1","family":"rule_application","variant":"original","group_id":"2c3a0cb61f7bae78739e","label":2,"option_order":["permitted","insufficient","prohibited"]}},{"id":"semif_ea7ec66a5e8eac886548","source":"semif_authored144","public":true,"request":{"state":"The designer approves making a sample badge but expressly withholds approval for mass production.","questions":{"decision":{"type":"choice","instructions":"A sample may be made when sample production is approved. May the sample be made?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=2c3a0cb61f7bae78739e","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":33,"line_sha256":"3dddf198bace7e3ba583635c4c62832b623c417e823edfb6e123bc7f79d68449","semif_id":"ea7ec66a5e8eac886548","family":"rule_application","variant":"criterion_reversal","group_id":"2c3a0cb61f7bae78739e","label":0,"option_order":["permitted","insufficient","prohibited"]}},{"id":"semif_25172cd6ddc49ab0ca70","source":"semif_authored144","public":true,"request":{"state":"The designer explicitly approves mass production of the badge.","questions":{"decision":{"type":"choice","instructions":"Mass production is allowed only after mass-production approval. May it begin?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=2c3a0cb61f7bae78739e","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":34,"line_sha256":"36b77284c665a4d63e1e59d9374caad429406ff0cf87b2fc63d79ca6e543189b","semif_id":"25172cd6ddc49ab0ca70","family":"rule_application","variant":"evidence_change","group_id":"2c3a0cb61f7bae78739e","label":2,"option_order":["prohibited","insufficient","permitted"]}},{"id":"semif_fb73a85d386cf59b0c51","source":"semif_authored144","public":true,"request":{"state":"The designer issued a decision but its production scope is unavailable.","questions":{"decision":{"type":"choice","instructions":"Mass production is allowed only after mass-production approval. May it begin?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=2c3a0cb61f7bae78739e","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":35,"line_sha256":"f0515999b33f09466d28e49066220dbc124fe28abbc7070c088264edd06ca7c5","semif_id":"fb73a85d386cf59b0c51","family":"rule_application","variant":"missing","group_id":"2c3a0cb61f7bae78739e","label":0,"option_order":["insufficient","permitted","prohibited"]}},{"id":"semif_a02fd582d2881cc462b2","source":"semif_authored144","public":true,"request":{"state":"The visitor's wristband covers the sculpture garden and expressly excludes the rooftop terrace.","questions":{"decision":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the sculpture garden?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=original group=42ed4cbfde7f2f64274b","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":36,"line_sha256":"cbc3172ae1e36e1c1bc71774eaa79185254051a851fbc70c6d2a68d0dbaa237f","semif_id":"a02fd582d2881cc462b2","family":"rule_application","variant":"original","group_id":"42ed4cbfde7f2f64274b","label":2,"option_order":["prohibited","insufficient","permitted"]}},{"id":"semif_19f37d5a97034bd7d04f","source":"semif_authored144","public":true,"request":{"state":"The visitor's wristband covers the sculpture garden and expressly excludes the rooftop terrace.","questions":{"decision":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the rooftop terrace?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"media":null,"gold":{"decision":"prohibited"},"note":"family=rule_application variant=criterion_reversal group=42ed4cbfde7f2f64274b","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":37,"line_sha256":"f392adb5e2fd2be74666008c525e6015935c1f18e12e55f5f9baa9abfd3316a1","semif_id":"19f37d5a97034bd7d04f","family":"rule_application","variant":"criterion_reversal","group_id":"42ed4cbfde7f2f64274b","label":1,"option_order":["insufficient","prohibited","permitted"]}},{"id":"semif_fda0ca94652c6d739ecf","source":"semif_authored144","public":true,"request":{"state":"The wristband expressly excludes the sculpture garden and covers only the rooftop terrace.","questions":{"decision":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the sculpture garden?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"prohibited"},"note":"family=rule_application variant=evidence_change group=42ed4cbfde7f2f64274b","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":38,"line_sha256":"b2df3c2405ec4dff81cc7fb84c3062e3b233298e2ae9101c0401509b05b5f49c","semif_id":"fda0ca94652c6d739ecf","family":"rule_application","variant":"evidence_change","group_id":"42ed4cbfde7f2f64274b","label":0,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_d0e36c4cb10a29b3191c","source":"semif_authored144","public":true,"request":{"state":"The visitor has a wristband but its covered areas are not supplied.","questions":{"decision":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the sculpture garden?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=42ed4cbfde7f2f64274b","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":39,"line_sha256":"72cc14e30f353d2d774556d9ce81674ed3cfbc77dded3451f23e5115af6faae3","semif_id":"d0e36c4cb10a29b3191c","family":"rule_application","variant":"missing","group_id":"42ed4cbfde7f2f64274b","label":0,"option_order":["insufficient","permitted","prohibited"]}},{"id":"semif_a0a39ace7e5337d038a7","source":"semif_authored144","public":true,"request":{"state":"The account owner permits changing the display name but expressly forbids changing the recovery email.","questions":{"decision":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the recovery email be changed?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=6dd193172b9240a04bf9","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":40,"line_sha256":"656219530edfc4681c9aadc7f1a955a264b4d5a5ec885b66dce867002129716e","semif_id":"a0a39ace7e5337d038a7","family":"rule_application","variant":"original","group_id":"6dd193172b9240a04bf9","label":0,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_eeae283841ce75dd25bf","source":"semif_authored144","public":true,"request":{"state":"The account owner permits changing the display name but expressly forbids changing the recovery email.","questions":{"decision":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the display name be changed?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=6dd193172b9240a04bf9","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":41,"line_sha256":"dd3219239ebef5a1aeccb55b7e45527a66dca4a5fdd1584d526267221e7ff317","semif_id":"eeae283841ce75dd25bf","family":"rule_application","variant":"criterion_reversal","group_id":"6dd193172b9240a04bf9","label":1,"option_order":["insufficient","permitted","prohibited"]}},{"id":"semif_00bb4cc043f7d2b288d4","source":"semif_authored144","public":true,"request":{"state":"The owner expressly permits changing the recovery email.","questions":{"decision":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the recovery email be changed?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=6dd193172b9240a04bf9","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":42,"line_sha256":"6b3d818c9910289316980359d766a984aede09c2fb8340f43b616a7c42131903","semif_id":"00bb4cc043f7d2b288d4","family":"rule_application","variant":"evidence_change","group_id":"6dd193172b9240a04bf9","label":2,"option_order":["insufficient","prohibited","permitted"]}},{"id":"semif_1a81ead69e93e5d0c86a","source":"semif_authored144","public":true,"request":{"state":"The owner requested an account update, but the fields to be changed are omitted.","questions":{"decision":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the recovery email be changed?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=6dd193172b9240a04bf9","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":43,"line_sha256":"231636717ff1e5be74db376f1f853588cb0d45d16ecef89598310be521faf0e9","semif_id":"1a81ead69e93e5d0c86a","family":"rule_application","variant":"missing","group_id":"6dd193172b9240a04bf9","label":2,"option_order":["permitted","prohibited","insufficient"]}},{"id":"semif_4444dad25e438f128673","source":"semif_authored144","public":true,"request":{"state":"The applicant lives in the valley but works outside it.","questions":{"decision":{"type":"choice","instructions":"A resident's pass requires living in the valley and has no workplace restriction. May it be issued?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=original group=3200eb252d6737e03027","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":44,"line_sha256":"604bcae1c371242120a51a3c2d4669647190da2376be505243cdcf352e5eeba7","semif_id":"4444dad25e438f128673","family":"rule_application","variant":"original","group_id":"3200eb252d6737e03027","label":1,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_3226202ba1ada2570e9f","source":"semif_authored144","public":true,"request":{"state":"The applicant lives in the valley but works outside it.","questions":{"decision":{"type":"choice","instructions":"Under the alternative rule, a pass requires working in the valley; residence alone is insufficient. May it be issued?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"prohibited"},"note":"family=rule_application variant=criterion_reversal group=3200eb252d6737e03027","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":45,"line_sha256":"8448947ddecb0c2c87a7fd50a811d70585cbdc6df53e026bccd046fde96c41a2","semif_id":"3226202ba1ada2570e9f","family":"rule_application","variant":"criterion_reversal","group_id":"3200eb252d6737e03027","label":0,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_3f783eec5c90812479bc","source":"semif_authored144","public":true,"request":{"state":"The applicant lives outside the valley and works inside it.","questions":{"decision":{"type":"choice","instructions":"A resident's pass requires living in the valley and has no workplace restriction. May it be issued?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"prohibited"},"note":"family=rule_application variant=evidence_change group=3200eb252d6737e03027","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":46,"line_sha256":"2526d42b7f34c2f6e44909233ca4699ed4610ebaed96f20088d39efa29b7b4fd","semif_id":"3f783eec5c90812479bc","family":"rule_application","variant":"evidence_change","group_id":"3200eb252d6737e03027","label":1,"option_order":["permitted","prohibited","insufficient"]}},{"id":"semif_ffee861474a9665a093a","source":"semif_authored144","public":true,"request":{"state":"The applicant supplies a name but no home or work location.","questions":{"decision":{"type":"choice","instructions":"A resident's pass requires living in the valley and has no workplace restriction. May it be issued?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=3200eb252d6737e03027","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":47,"line_sha256":"61fec49012f547f2567bf5ef3d5608aa3fe75bfd7783e64ce8b402788b44c9f3","semif_id":"ffee861474a9665a093a","family":"rule_application","variant":"missing","group_id":"3200eb252d6737e03027","label":2,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_8c992e17e4fb54b044f5","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The florist confirms that the wreath was delivered. Candidate B: The florist offers to deliver a wreath tomorrow.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting completed delivery.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"media":null,"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=c0cbeb97315307af8111","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":48,"line_sha256":"89cf2043f353e2a8425b92e116152968bca6c14d724124bfe12e0d632d23c409","semif_id":"8c992e17e4fb54b044f5","family":"candidate_selection","variant":"original","group_id":"c0cbeb97315307af8111","label":1,"option_order":["B","A","insufficient"]}},{"id":"semif_695dd61195ca025c4f4c","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The florist confirms that the wreath was delivered. Candidate B: The florist offers to deliver a wreath tomorrow.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate offering a future delivery.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=c0cbeb97315307af8111","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":49,"line_sha256":"9346a5d170f0cf1d535dc73ba483ad49c2eace8e9ec58084f20dbd7411757ffe","semif_id":"695dd61195ca025c4f4c","family":"candidate_selection","variant":"criterion_reversal","group_id":"c0cbeb97315307af8111","label":0,"option_order":["B","insufficient","A"]}},{"id":"semif_e6155431ab7b0f8170d4","source":"semif_authored144","public":true,"request":{"state":"Candidate A: I can deliver a wreath tomorrow. Candidate B: The wreath was delivered.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting completed delivery.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=c0cbeb97315307af8111","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":50,"line_sha256":"038c2b872fd328c23e1db158bcd0903e5a399e96b1775c5feecf8a9d4f31c33c","semif_id":"e6155431ab7b0f8170d4","family":"candidate_selection","variant":"evidence_change","group_id":"c0cbeb97315307af8111","label":1,"option_order":["insufficient","B","A"]}},{"id":"semif_b49ecda75086d8c68d2a","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The florist gives an address. Candidate B: The wreath contains lilies.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting completed delivery.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=c0cbeb97315307af8111","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":51,"line_sha256":"ecf68f5df455aa43da7818fe2e7ad713f439c6eecdf7b799f02dfb4df301bbaf","semif_id":"b49ecda75086d8c68d2a","family":"candidate_selection","variant":"missing","group_id":"c0cbeb97315307af8111","label":2,"option_order":["B","A","insufficient"]}},{"id":"semif_587ced261c6092cd1325","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The customer asks how an engraving would look but explicitly declines having it engraved yet. Candidate B: The customer instructs the shop to engrave the ring.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate authorizing engraving.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=original group=428176cd8ad2c7a74eea","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":52,"line_sha256":"624cd699d50eecad754dd8457fcf13a651ee61cbdf61bf4ae96722ee00d40248","semif_id":"587ced261c6092cd1325","family":"candidate_selection","variant":"original","group_id":"428176cd8ad2c7a74eea","label":1,"option_order":["insufficient","B","A"]}},{"id":"semif_533d4423d311de82b27d","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The customer asks how an engraving would look but explicitly declines having it engraved yet. Candidate B: The customer instructs the shop to engrave the ring.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate asking only for information while withholding authorization.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"media":null,"gold":{"decision":"A"},"note":"family=candidate_selection variant=criterion_reversal group=428176cd8ad2c7a74eea","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":53,"line_sha256":"284139b9925f90df569c901d8f500e2cd7eccef7ed8960e094ee1f4457ffadbd","semif_id":"533d4423d311de82b27d","family":"candidate_selection","variant":"criterion_reversal","group_id":"428176cd8ad2c7a74eea","label":1,"option_order":["insufficient","A","B"]}},{"id":"semif_2bf540d515a51755b25c","source":"semif_authored144","public":true,"request":{"state":"Candidate A: Please engrave the ring. Candidate B: Show me how it would look, but do not engrave it yet.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate authorizing engraving.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"media":null,"gold":{"decision":"A"},"note":"family=candidate_selection variant=evidence_change group=428176cd8ad2c7a74eea","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":54,"line_sha256":"14c169b4741e827edb44afefd9bc380cbffb1a0c9fd006aad047633da4057d31","semif_id":"2bf540d515a51755b25c","family":"candidate_selection","variant":"evidence_change","group_id":"428176cd8ad2c7a74eea","label":2,"option_order":["insufficient","B","A"]}},{"id":"semif_fdb9fc214aea2710763a","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The customer gives a name. Candidate B: The ring is silver.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate authorizing engraving.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=428176cd8ad2c7a74eea","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":55,"line_sha256":"333f1885b1fcad000c7909789d7a5218ebad42cfe0f8a92b97a094aa29d294dd","semif_id":"fdb9fc214aea2710763a","family":"candidate_selection","variant":"missing","group_id":"428176cd8ad2c7a74eea","label":2,"option_order":["B","A","insufficient"]}},{"id":"semif_f6d68126b43f8c9ed183","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The note records that the completed insulation test passed. Candidate B: The note lists insulation-test steps but says no test was run.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an observed test result.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"media":null,"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=389e65bf1ba709b84b12","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":56,"line_sha256":"c1f17ff7f97d4730b49ea049d68795296e425330b4cc28714606ab0259987312","semif_id":"f6d68126b43f8c9ed183","family":"candidate_selection","variant":"original","group_id":"389e65bf1ba709b84b12","label":1,"option_order":["insufficient","A","B"]}},{"id":"semif_44bae0c8a1615a096c43","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The note records that the completed insulation test passed. Candidate B: The note lists insulation-test steps but says no test was run.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate containing only an unexecuted test procedure.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=389e65bf1ba709b84b12","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":57,"line_sha256":"73ecb16d7d1766027826cdef506368c3a0fdebacc5a24208389586b776a22252","semif_id":"44bae0c8a1615a096c43","family":"candidate_selection","variant":"criterion_reversal","group_id":"389e65bf1ba709b84b12","label":0,"option_order":["B","A","insufficient"]}},{"id":"semif_2175379f4c69e6207626","source":"semif_authored144","public":true,"request":{"state":"Candidate A: Here are insulation-test steps; no test was run. Candidate B: The completed insulation test passed.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an observed test result.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=389e65bf1ba709b84b12","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":58,"line_sha256":"a66d7270af284da26d40590eeeddf810d1611af432d556a9645076a9f6013841","semif_id":"2175379f4c69e6207626","family":"candidate_selection","variant":"evidence_change","group_id":"389e65bf1ba709b84b12","label":1,"option_order":["insufficient","B","A"]}},{"id":"semif_def9fe59d262357a673a","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The cable is red. Candidate B: The tester is in a drawer.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an observed test result.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=389e65bf1ba709b84b12","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":59,"line_sha256":"6b011ff841294db3944fc693427f048aea9605aa0d0c050bb2d41c0fe518a597","semif_id":"def9fe59d262357a673a","family":"candidate_selection","variant":"missing","group_id":"389e65bf1ba709b84b12","label":0,"option_order":["insufficient","A","B"]}},{"id":"semif_712f6ffb763eef7c9a12","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The editor quotes a proposal for a paywall only to argue against it. Candidate B: The editor endorses introducing a paywall.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate supporting a paywall.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=original group=45f8ead2522a14384cdc","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":60,"line_sha256":"b1b66ff0d760812af5cf48a4e5966e5c231a54ef441925fc9a49d1f01b0d720f","semif_id":"712f6ffb763eef7c9a12","family":"candidate_selection","variant":"original","group_id":"45f8ead2522a14384cdc","label":0,"option_order":["B","A","insufficient"]}},{"id":"semif_e1d610bd14e3d16c09b9","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The editor quotes a proposal for a paywall only to argue against it. Candidate B: The editor endorses introducing a paywall.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate opposing a paywall.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"media":null,"gold":{"decision":"A"},"note":"family=candidate_selection variant=criterion_reversal group=45f8ead2522a14384cdc","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":61,"line_sha256":"0b5013b84e46f56ca742a2903fd935c164d33c469dabc04dee5dff2316e7f530","semif_id":"e1d610bd14e3d16c09b9","family":"candidate_selection","variant":"criterion_reversal","group_id":"45f8ead2522a14384cdc","label":0,"option_order":["A","insufficient","B"]}},{"id":"semif_22ec57822db50ec121cf","source":"semif_authored144","public":true,"request":{"state":"Candidate A: I endorse introducing a paywall. Candidate B: I reject introducing a paywall.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate supporting a paywall.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"media":null,"gold":{"decision":"A"},"note":"family=candidate_selection variant=evidence_change group=45f8ead2522a14384cdc","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":62,"line_sha256":"e68cc16bb90711db2a49e6cf906b29d3e9034a46087857aa800061034a5c1f75","semif_id":"22ec57822db50ec121cf","family":"candidate_selection","variant":"evidence_change","group_id":"45f8ead2522a14384cdc","label":1,"option_order":["insufficient","A","B"]}},{"id":"semif_3e75f2b84e8e08e3d3f2","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The editor names the magazine. Candidate B: The website has a blue header.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate supporting a paywall.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=45f8ead2522a14384cdc","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":63,"line_sha256":"c902eff9d1fd2f98073510179439fee5c04d0f1d6d4eb995792ad6bd1f68cd22","semif_id":"3e75f2b84e8e08e3d3f2","family":"candidate_selection","variant":"missing","group_id":"45f8ead2522a14384cdc","label":2,"option_order":["B","A","insufficient"]}},{"id":"semif_ebd74aa05324a1556288","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The owner asks to reseal the existing window while keeping its frame. Candidate B: The owner asks to remove the window and fit a complete replacement.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate asking to repair the existing window.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"media":null,"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=3ecec8e2c166e397ad07","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":64,"line_sha256":"29373afd1f31e46bd9131be7bff97948583ae454e2800e284d92c7e095445ae7","semif_id":"ebd74aa05324a1556288","family":"candidate_selection","variant":"original","group_id":"3ecec8e2c166e397ad07","label":1,"option_order":["insufficient","A","B"]}},{"id":"semif_d7e84e4245a29817da1f","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The owner asks to reseal the existing window while keeping its frame. Candidate B: The owner asks to remove the window and fit a complete replacement.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate asking for complete replacement.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=3ecec8e2c166e397ad07","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":65,"line_sha256":"e1333a822e60c3e20bbbaba0a8a13128ad420437e533f2ee6560ae397c9af6af","semif_id":"d7e84e4245a29817da1f","family":"candidate_selection","variant":"criterion_reversal","group_id":"3ecec8e2c166e397ad07","label":2,"option_order":["insufficient","A","B"]}},{"id":"semif_f41ae66b3c09630c7a42","source":"semif_authored144","public":true,"request":{"state":"Candidate A: Remove the window and install a replacement. Candidate B: Reseal the existing window and keep its frame.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate asking to repair the existing window.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=3ecec8e2c166e397ad07","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":66,"line_sha256":"12e922b40631c0eabc53fd463a9a122a76b793eb1aff308ff547b4ddd2597061","semif_id":"f41ae66b3c09630c7a42","family":"candidate_selection","variant":"evidence_change","group_id":"3ecec8e2c166e397ad07","label":2,"option_order":["insufficient","A","B"]}},{"id":"semif_82c2cd97c160ad27951b","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The window faces a courtyard. Candidate B: The owner gives a postcode.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate asking to repair the existing window.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=3ecec8e2c166e397ad07","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":67,"line_sha256":"5e827dfcaef587f3b69c77190c027364342d2447960deba7094c78ad6f374f90","semif_id":"82c2cd97c160ad27951b","family":"candidate_selection","variant":"missing","group_id":"3ecec8e2c166e397ad07","label":0,"option_order":["insufficient","B","A"]}},{"id":"semif_7ada97b6f99f4788e5c7","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The sponsor confirms receipt of a grant application but explicitly says no award decision has been made. Candidate B: The sponsor confirms that the grant was awarded.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an awarded grant.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=original group=a71a17a287f3d30db2aa","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":68,"line_sha256":"062d86d202a44054d2ca7fd9d5df1cba61b83dd6e66b9674f6759072ba0d3574","semif_id":"7ada97b6f99f4788e5c7","family":"candidate_selection","variant":"original","group_id":"a71a17a287f3d30db2aa","label":0,"option_order":["B","A","insufficient"]}},{"id":"semif_4d7a003342db5b0a17e3","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The sponsor confirms receipt of a grant application but explicitly says no award decision has been made. Candidate B: The sponsor confirms that the grant was awarded.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting receipt with the award still undecided.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"media":null,"gold":{"decision":"A"},"note":"family=candidate_selection variant=criterion_reversal group=a71a17a287f3d30db2aa","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":69,"line_sha256":"3126261538009017470f7ac26ee182a3da3f78109e70f36c838825289a31b8d8","semif_id":"4d7a003342db5b0a17e3","family":"candidate_selection","variant":"criterion_reversal","group_id":"a71a17a287f3d30db2aa","label":1,"option_order":["insufficient","A","B"]}},{"id":"semif_19d3d8d0bf773e88be7b","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The grant was awarded. Candidate B: The application was received; the award remains undecided.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an awarded grant.","criteria":{"A":"Candidate A","B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence"}}}},"media":null,"gold":{"decision":"A"},"note":"family=candidate_selection variant=evidence_change group=a71a17a287f3d30db2aa","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":70,"line_sha256":"f4f1112105587e7d1c76b01cdef209a4f8368a56ad9b98e6e40634bd17be02c2","semif_id":"19d3d8d0bf773e88be7b","family":"candidate_selection","variant":"evidence_change","group_id":"a71a17a287f3d30db2aa","label":0,"option_order":["A","B","insufficient"]}},{"id":"semif_d6de731cca52359b1c87","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The sponsor gives its name. Candidate B: The application has a reference number.","questions":{"decision":{"type":"choice","instructions":"Choose the candidate reporting an awarded grant.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=a71a17a287f3d30db2aa","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":71,"line_sha256":"fe82e361e4b6f004e986e61119470b7af9b0c1e5e8371efa7611d7a17b87fb7f","semif_id":"d6de731cca52359b1c87","family":"candidate_selection","variant":"missing","group_id":"a71a17a287f3d30db2aa","label":0,"option_order":["insufficient","A","B"]}},{"id":"semif_b45c63fb92729af27f05","source":"semif_authored144","public":true,"request":{"state":"On Monday the owner authorized scanning her notebook. On Tuesday she explicitly withdrew that authorization. The record says withdrawals cancel earlier permission, and no later permission was given.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: permission to scan the notebook is currently valid.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}}}},"media":null,"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=986d7eaba0c7e9738e79","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":72,"line_sha256":"3a9217aa64137c66dd3201087f0a5caa53c2f155b5ee7d3af92a1768fa0f7477","semif_id":"b45c63fb92729af27f05","family":"evidence_interpretation","variant":"original","group_id":"986d7eaba0c7e9738e79","label":0,"option_order":["contradicted","supported","insufficient"]}},{"id":"semif_46b7029b9a704138b77a","source":"semif_authored144","public":true,"request":{"state":"On Monday the owner authorized scanning her notebook. On Tuesday she explicitly withdrew that authorization. The record says withdrawals cancel earlier permission, and no later permission was given.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the owner gave permission on Monday.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=986d7eaba0c7e9738e79","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":73,"line_sha256":"619b2ef40bdcdac22a59d18d980f7175c8ad4284075897d1f6fc4d99387f1721","semif_id":"46b7029b9a704138b77a","family":"evidence_interpretation","variant":"criterion_reversal","group_id":"986d7eaba0c7e9738e79","label":2,"option_order":["insufficient","contradicted","supported"]}},{"id":"semif_caf773b23934fa7362a1","source":"semif_authored144","public":true,"request":{"state":"The owner authorized scanning the notebook on Monday. The complete permission record confirms that she has not withdrawn it and it remains valid.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: permission to scan the notebook is currently valid.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=986d7eaba0c7e9738e79","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":74,"line_sha256":"a8382e8657752bcf95d155ca6412344348fec7c4969f90d53b823b904eb2d14a","semif_id":"caf773b23934fa7362a1","family":"evidence_interpretation","variant":"evidence_change","group_id":"986d7eaba0c7e9738e79","label":0,"option_order":["supported","contradicted","insufficient"]}},{"id":"semif_f46f392ef9e9e9df564b","source":"semif_authored144","public":true,"request":{"state":"The owner authorized scanning on Monday. The supplied record omits all later permission changes and does not establish current validity.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: permission to scan the notebook is currently valid.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=986d7eaba0c7e9738e79","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":75,"line_sha256":"02184e34650daf91d0424f9bbd739aab20ca996fdaef6777e3cae78204f2bc0d","semif_id":"f46f392ef9e9e9df564b","family":"evidence_interpretation","variant":"missing","group_id":"986d7eaba0c7e9738e79","label":1,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_005dcf6d2c0f21d64796","source":"semif_authored144","public":true,"request":{"state":"The homeowner first allowed a surveyor to enter the attic. Before the visit she explicitly withdrew that permission. No later decision exists.","questions":{"decision":{"type":"choice","instructions":"Use the homeowner's latest permission decision; a withdrawal cancels the earlier grant. May the surveyor enter the attic?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=ca2e0b043985fa49bcd2","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":76,"line_sha256":"9440dc64a7f53830326cb2d7def27686e51501f48458adea5983b3761d658a95","semif_id":"005dcf6d2c0f21d64796","family":"rule_application","variant":"original","group_id":"ca2e0b043985fa49bcd2","label":1,"option_order":["permitted","prohibited","insufficient"]}},{"id":"semif_edb708bd0fe5fd67d6ee","source":"semif_authored144","public":true,"request":{"state":"The homeowner first allowed a surveyor to enter the attic. Before the visit she explicitly withdrew that permission. No later decision exists.","questions":{"decision":{"type":"choice","instructions":"For this fictional protocol, the first recorded decision is binding and later withdrawals are ignored. May the surveyor enter the attic?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=ca2e0b043985fa49bcd2","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":77,"line_sha256":"6a2e4901524287679f7aad9cd65a4974a4ac68bbaa22a775ec76f8646d2fe69a","semif_id":"edb708bd0fe5fd67d6ee","family":"rule_application","variant":"criterion_reversal","group_id":"ca2e0b043985fa49bcd2","label":0,"option_order":["permitted","prohibited","insufficient"]}},{"id":"semif_97a35499bd8717f3e786","source":"semif_authored144","public":true,"request":{"state":"The homeowner first refused attic entry, then explicitly granted it before the visit. No later decision exists.","questions":{"decision":{"type":"choice","instructions":"Use the homeowner's latest permission decision; a withdrawal cancels the earlier grant. May the surveyor enter the attic?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=ca2e0b043985fa49bcd2","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":78,"line_sha256":"6322c2cbf3063dc0dbd11c78d1e2242b014cdc3455ebabbc00861f2cfe53c71c","semif_id":"97a35499bd8717f3e786","family":"rule_application","variant":"evidence_change","group_id":"ca2e0b043985fa49bcd2","label":2,"option_order":["prohibited","insufficient","permitted"]}},{"id":"semif_31eb4533d2c89959ec46","source":"semif_authored144","public":true,"request":{"state":"The homeowner gave an early permission decision, but neither its content nor any later decisions are available.","questions":{"decision":{"type":"choice","instructions":"Use the homeowner's latest permission decision; a withdrawal cancels the earlier grant. May the surveyor enter the attic?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=ca2e0b043985fa49bcd2","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":79,"line_sha256":"24b44df64d8382c2ff23b0df84aee7ce8c5d61b09c20f63c49f961cad232f3a3","semif_id":"31eb4533d2c89959ec46","family":"rule_application","variant":"missing","group_id":"ca2e0b043985fa49bcd2","label":0,"option_order":["insufficient","prohibited","permitted"]}},{"id":"semif_0aa24a6701ac9c56c4a0","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The speaker authorized streaming, then explicitly withdrew the authorization before the event. Candidate B: The speaker authorized streaming and the complete record confirms it remains in force.","questions":{"decision":{"type":"choice","instructions":"Select the candidate with currently valid streaming permission; a withdrawal cancels an earlier grant.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=original group=85051ba3c30277824ebe","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":80,"line_sha256":"4017dd15d93c91ab811e456dae7c7912599096e1b69b0f484093e6013e841a0c","semif_id":"0aa24a6701ac9c56c4a0","family":"candidate_selection","variant":"original","group_id":"85051ba3c30277824ebe","label":1,"option_order":["insufficient","B","A"]}},{"id":"semif_796d5c0da6eaadae6a3f","source":"semif_authored144","public":true,"request":{"state":"Candidate A: The speaker authorized streaming, then explicitly withdrew the authorization before the event. Candidate B: The speaker authorized streaming and the complete record confirms it remains in force.","questions":{"decision":{"type":"choice","instructions":"Select the candidate documenting an authorization that was later withdrawn.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"media":null,"gold":{"decision":"A"},"note":"family=candidate_selection variant=criterion_reversal group=85051ba3c30277824ebe","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":81,"line_sha256":"4487fc039fd3b2e9fc1bf18a7936cea404cb6a4cce2ad0f2df4737a0450ed833","semif_id":"796d5c0da6eaadae6a3f","family":"candidate_selection","variant":"criterion_reversal","group_id":"85051ba3c30277824ebe","label":1,"option_order":["B","A","insufficient"]}},{"id":"semif_599b3092b6fd18312d23","source":"semif_authored144","public":true,"request":{"state":"Candidate A: Streaming was authorized and the complete record says permission remains in force. Candidate B: Streaming was authorized, then explicitly withdrawn.","questions":{"decision":{"type":"choice","instructions":"Select the candidate with currently valid streaming permission; a withdrawal cancels an earlier grant.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}}}},"media":null,"gold":{"decision":"A"},"note":"family=candidate_selection variant=evidence_change group=85051ba3c30277824ebe","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":82,"line_sha256":"0b1fc7b68e5d20226c4fd87d7cf2f0af2c21ce1cac1e2bb755068f34550e9d4a","semif_id":"599b3092b6fd18312d23","family":"candidate_selection","variant":"evidence_change","group_id":"85051ba3c30277824ebe","label":2,"option_order":["B","insufficient","A"]}},{"id":"semif_960a4bcd27363e6db4b9","source":"semif_authored144","public":true,"request":{"state":"Candidate A: A permission form is mentioned but its terms and later changes are missing. Candidate B: The event name is supplied without permission history.","questions":{"decision":{"type":"choice","instructions":"Select the candidate with currently valid streaming permission; a withdrawal cancels an earlier grant.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=85051ba3c30277824ebe","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":83,"line_sha256":"341b018144f4d27060f78e9d9cb5adc9744bab1c184cf172b1afeca20f6813ed","semif_id":"960a4bcd27363e6db4b9","family":"candidate_selection","variant":"missing","group_id":"85051ba3c30277824ebe","label":2,"option_order":["B","A","insufficient"]}},{"id":"semif_c6a411e7f2bbdf377173","source":"semif_authored144","public":true,"request":{"state":"The director gave Lina authority to approve exhibit labels and explicitly forbade further delegation. Lina then told Mo to approve them. The rule states an attempted forbidden delegation grants no authority; Mo has no other grant.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: Mo has authority to approve exhibit labels.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=e7798dab5f4f121cab77","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":84,"line_sha256":"4d437323bdea93fd769fb2f2ab69d14aefa976c7cdd814e1217d17209937f25a","semif_id":"c6a411e7f2bbdf377173","family":"evidence_interpretation","variant":"original","group_id":"e7798dab5f4f121cab77","label":0,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_e332aa927d0c2ed044aa","source":"semif_authored144","public":true,"request":{"state":"The director gave Lina authority to approve exhibit labels and explicitly forbade further delegation. Lina then told Mo to approve them. The rule states an attempted forbidden delegation grants no authority; Mo has no other grant.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: Lina has authority to approve exhibit labels.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=e7798dab5f4f121cab77","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":85,"line_sha256":"95318b27890681c64b5fcd1446106c7127f9cb3ec2f90d89dfeba0d33d984298","semif_id":"e332aa927d0c2ed044aa","family":"evidence_interpretation","variant":"criterion_reversal","group_id":"e7798dab5f4f121cab77","label":2,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_a0128e551d095f3449f8","source":"semif_authored144","public":true,"request":{"state":"The director gives Mo direct authority to approve exhibit labels; the record confirms this grant is valid.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: Mo has authority to approve exhibit labels.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=e7798dab5f4f121cab77","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":86,"line_sha256":"c561f9f9e2a0289e139b78c42155a5e356c368effcf01d8b07096760c52428bc","semif_id":"a0128e551d095f3449f8","family":"evidence_interpretation","variant":"evidence_change","group_id":"e7798dab5f4f121cab77","label":0,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_46d6a0731bd288c370f0","source":"semif_authored144","public":true,"request":{"state":"Mo is named in correspondence about labels, but no authority grants or delegation terms are supplied.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: Mo has authority to approve exhibit labels.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=e7798dab5f4f121cab77","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":87,"line_sha256":"1c46f276c92e30a59a18eb317ef01a3370f18840f7f945f3f77d42310f04dffe","semif_id":"46d6a0731bd288c370f0","family":"evidence_interpretation","variant":"missing","group_id":"e7798dab5f4f121cab77","label":0,"option_order":["insufficient","contradicted","supported"]}},{"id":"semif_21ca7a8b970e36b8e20f","source":"semif_authored144","public":true,"request":{"state":"The producer authorizes the stage manager to approve props. The stage manager passes the task to an assistant, who has no other approval authority.","questions":{"decision":{"type":"choice","instructions":"The stage manager may approve props but may not delegate that authority. Only authorized approvers may approve. May the assistant approve a prop?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"media":null,"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=928ebc7817a8f1cac3e4","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":88,"line_sha256":"cb107f8ae09d2978fe7d4edf4b4553479e26fad548859ed32a1819b9c51dd144","semif_id":"21ca7a8b970e36b8e20f","family":"rule_application","variant":"original","group_id":"928ebc7817a8f1cac3e4","label":0,"option_order":["prohibited","insufficient","permitted"]}},{"id":"semif_4aa83af95bb69e90bc2d","source":"semif_authored144","public":true,"request":{"state":"The producer authorizes the stage manager to approve props. The stage manager passes the task to an assistant, who has no other approval authority.","questions":{"decision":{"type":"choice","instructions":"The stage manager may delegate prop approval to an assistant, and doing so grants the assistant approval authority. May this assistant approve?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=928ebc7817a8f1cac3e4","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":89,"line_sha256":"67d4bfc2d38d76f11ee3e41c1171ef5598772d2c0d7b0de1c9e49017c81dc2de","semif_id":"4aa83af95bb69e90bc2d","family":"rule_application","variant":"criterion_reversal","group_id":"928ebc7817a8f1cac3e4","label":0,"option_order":["permitted","prohibited","insufficient"]}},{"id":"semif_cdfa9eed51fafcf09b72","source":"semif_authored144","public":true,"request":{"state":"The producer directly authorizes the assistant to approve props.","questions":{"decision":{"type":"choice","instructions":"The stage manager may approve props but may not delegate that authority. Only authorized approvers may approve. May the assistant approve a prop?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=928ebc7817a8f1cac3e4","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":90,"line_sha256":"0a2fc850c6525063e8e7f27c6442e1316283526b31e36295fc66530eaa71435b","semif_id":"cdfa9eed51fafcf09b72","family":"rule_application","variant":"evidence_change","group_id":"928ebc7817a8f1cac3e4","label":2,"option_order":["insufficient","prohibited","permitted"]}},{"id":"semif_d51b05bfb9d6d7f09331","source":"semif_authored144","public":true,"request":{"state":"An assistant works on the production, but no approval grants or delegation history are provided.","questions":{"decision":{"type":"choice","instructions":"The stage manager may approve props but may not delegate that authority. Only authorized approvers may approve. May the assistant approve a prop?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=928ebc7817a8f1cac3e4","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":91,"line_sha256":"a33d0facd74007f56a160a7e12d893d9ecab43b60a5ec0ce3b05523c820bd036","semif_id":"d51b05bfb9d6d7f09331","family":"rule_application","variant":"missing","group_id":"928ebc7817a8f1cac3e4","label":1,"option_order":["prohibited","insufficient","permitted"]}},{"id":"semif_fdd902cc6e65fad7cb4f","source":"semif_authored144","public":true,"request":{"state":"The rule permits direct delegation by the director but forbids any further delegation. Candidate A: I received prop approval authority directly from the director. Candidate B: I received it only from A, and have no other grant.","questions":{"decision":{"type":"choice","instructions":"Select the candidate holding valid prop approval authority under the stated delegation rule.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"media":null,"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=53b0770ec36835b81d3a","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":92,"line_sha256":"01ceabcdf17606074dcc6242e11c9ad7d76b21e95ae96653c2ef513e1bb11fb2","semif_id":"fdd902cc6e65fad7cb4f","family":"candidate_selection","variant":"original","group_id":"53b0770ec36835b81d3a","label":1,"option_order":["B","A","insufficient"]}},{"id":"semif_42384af9ae56beb37ff5","source":"semif_authored144","public":true,"request":{"state":"The rule permits direct delegation by the director but forbids any further delegation. Candidate A: I received prop approval authority directly from the director. Candidate B: I received it only from A, and have no other grant.","questions":{"decision":{"type":"choice","instructions":"Select the candidate whose claimed authority comes only through a prohibited second delegation.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=53b0770ec36835b81d3a","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":93,"line_sha256":"2813af05a60a0fb7d63dacf7d4cdadd5144d59573a06e5f02d9fa86ee214d7b4","semif_id":"42384af9ae56beb37ff5","family":"candidate_selection","variant":"criterion_reversal","group_id":"53b0770ec36835b81d3a","label":2,"option_order":["insufficient","A","B"]}},{"id":"semif_cdf7f428857e1b841cb3","source":"semif_authored144","public":true,"request":{"state":"The rule permits direct delegation by the director but forbids further delegation. Candidate A: My only claimed authority came from B. Candidate B: The director directly granted me prop approval authority.","questions":{"decision":{"type":"choice","instructions":"Select the candidate holding valid prop approval authority under the stated delegation rule.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=53b0770ec36835b81d3a","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":94,"line_sha256":"03bc0db267eff459bb736b3c3b40dad41880dc42af3ccdb8b0a4ced9b900989e","semif_id":"cdf7f428857e1b841cb3","family":"candidate_selection","variant":"evidence_change","group_id":"53b0770ec36835b81d3a","label":2,"option_order":["A","insufficient","B"]}},{"id":"semif_774f7f9245f83c796d44","source":"semif_authored144","public":true,"request":{"state":"The director's delegation rule is supplied, but Candidate A and Candidate B provide only their names and no authority histories.","questions":{"decision":{"type":"choice","instructions":"Select the candidate holding valid prop approval authority under the stated delegation rule.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=53b0770ec36835b81d3a","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":95,"line_sha256":"62b3518676733167a27524fa094c74cb9467666a6440cb3c3d771d18831ae001","semif_id":"774f7f9245f83c796d44","family":"candidate_selection","variant":"missing","group_id":"53b0770ec36835b81d3a","label":2,"option_order":["B","A","insufficient"]}},{"id":"semif_92ec2ffa3dd46ada1335","source":"semif_authored144","public":true,"request":{"state":"The return policy accepts faulty lamps despite opened packaging, but explicitly excludes water damage even when the lamp is faulty. This opened lamp is faulty because water entered it. No other cause is present.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: this lamp qualifies for return under the stated policy.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=056cc5b726d6e8300a77","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":96,"line_sha256":"f4e473cd8feb10ea7874d3eae00efa068c094e2d17551c15dc7a0dffbf286d29","semif_id":"92ec2ffa3dd46ada1335","family":"evidence_interpretation","variant":"original","group_id":"056cc5b726d6e8300a77","label":0,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_6149a17bc154f9c5b4a0","source":"semif_authored144","public":true,"request":{"state":"The return policy accepts faulty lamps despite opened packaging, but explicitly excludes water damage even when the lamp is faulty. This opened lamp is faulty because water entered it. No other cause is present.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the lamp has a fault.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=056cc5b726d6e8300a77","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":97,"line_sha256":"c8abb18dd997b9c8e790fc26c1e2d63aba6d7abe2dab8fd4545323f7ef2e1998","semif_id":"6149a17bc154f9c5b4a0","family":"evidence_interpretation","variant":"criterion_reversal","group_id":"056cc5b726d6e8300a77","label":0,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_be866e1dbec5d42d9423","source":"semif_authored144","public":true,"request":{"state":"The return policy accepts faulty lamps despite opened packaging, but excludes water damage even when faulty. This opened lamp has a manufacturing wiring fault, with no water damage or other exclusion.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: this lamp qualifies for return under the stated policy.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=056cc5b726d6e8300a77","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":98,"line_sha256":"da3acbab2ca934c780a3e0c349d61d6bc6f34f93ade5e3c39d1a03f0784f785e","semif_id":"be866e1dbec5d42d9423","family":"evidence_interpretation","variant":"evidence_change","group_id":"056cc5b726d6e8300a77","label":2,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_97d2ed2591fc7638806c","source":"semif_authored144","public":true,"request":{"state":"The return policy accepts faulty lamps despite opened packaging but excludes water damage even when faulty. The lamp is faulty, but the cause and presence of water damage are omitted.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: this lamp qualifies for return under the stated policy.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=056cc5b726d6e8300a77","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":99,"line_sha256":"58b1bebbb4f24da0739f2f867e9b48cc84a0c3f8a15d9638d95145db5ddff591","semif_id":"97d2ed2591fc7638806c","family":"evidence_interpretation","variant":"missing","group_id":"056cc5b726d6e8300a77","label":0,"option_order":["insufficient","supported","contradicted"]}},{"id":"semif_4533dd91267b4d815690","source":"semif_authored144","public":true,"request":{"state":"The opened paint tin has a manufacturing defect and also contains added solvent. Both facts are confirmed.","questions":{"decision":{"type":"choice","instructions":"Normally reject opened paint. Accept it for a manufacturing defect, except that added solvent always makes it ineligible. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"media":null,"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=e3ce8ec496dbc9974f94","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":100,"line_sha256":"b2c8e57b970d4a9a6e9f986d47e8ff0645cfc0f0f3c3b693ace5427831314e40","semif_id":"4533dd91267b4d815690","family":"rule_application","variant":"original","group_id":"e3ce8ec496dbc9974f94","label":2,"option_order":["permitted","insufficient","prohibited"]}},{"id":"semif_d880e65fef5e2993fdd0","source":"semif_authored144","public":true,"request":{"state":"The opened paint tin has a manufacturing defect and also contains added solvent. Both facts are confirmed.","questions":{"decision":{"type":"choice","instructions":"Accept defective paint even if opened or mixed with solvent; neither condition overrides the defect exception. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=e3ce8ec496dbc9974f94","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":101,"line_sha256":"eebb6da982496d47bb110182f50520ab215bd53576b9c9a14cd646d36c6e83c0","semif_id":"d880e65fef5e2993fdd0","family":"rule_application","variant":"criterion_reversal","group_id":"e3ce8ec496dbc9974f94","label":0,"option_order":["permitted","insufficient","prohibited"]}},{"id":"semif_90494c76078ffc3c717a","source":"semif_authored144","public":true,"request":{"state":"The opened tin has a manufacturing defect and the inspection confirms no solvent was added.","questions":{"decision":{"type":"choice","instructions":"Normally reject opened paint. Accept it for a manufacturing defect, except that added solvent always makes it ineligible. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=e3ce8ec496dbc9974f94","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":102,"line_sha256":"3b79d2949f80573b5048e271419c1654d57a5e5ee94f59449d009714b0ba92d0","semif_id":"90494c76078ffc3c717a","family":"rule_application","variant":"evidence_change","group_id":"e3ce8ec496dbc9974f94","label":0,"option_order":["permitted","insufficient","prohibited"]}},{"id":"semif_9ac5cc82c4126c91174a","source":"semif_authored144","public":true,"request":{"state":"The opened tin is defective, but whether solvent was added is not known.","questions":{"decision":{"type":"choice","instructions":"Normally reject opened paint. Accept it for a manufacturing defect, except that added solvent always makes it ineligible. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=e3ce8ec496dbc9974f94","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":103,"line_sha256":"2efefa757e37f35491fdb1760e1a33670eec1133282603d095dcac6fa5c6025d","semif_id":"9ac5cc82c4126c91174a","family":"rule_application","variant":"missing","group_id":"e3ce8ec496dbc9974f94","label":1,"option_order":["permitted","insufficient","prohibited"]}},{"id":"semif_386e10b1c1ce906473a4","source":"semif_authored144","public":true,"request":{"state":"Accept late submissions for documented outages, except that missing signatures always disqualify. Candidate A: My submission was late because of a documented outage and includes the required signature. Candidate B: Mine was late because of a documented outage but lacks the signature.","questions":{"decision":{"type":"choice","instructions":"Select the submission qualifying under the stated exception and exclusion.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"media":null,"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=a5b234c2e4fb304f1071","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":104,"line_sha256":"f850047e7afe4504ef239b613b8bbf29a52335e69db396d9d2c33ab1b8b2f4e7","semif_id":"386e10b1c1ce906473a4","family":"candidate_selection","variant":"original","group_id":"a5b234c2e4fb304f1071","label":1,"option_order":["B","A","insufficient"]}},{"id":"semif_4da6aedc1139224eeff2","source":"semif_authored144","public":true,"request":{"state":"Accept late submissions for documented outages, except that missing signatures always disqualify. Candidate A: My submission was late because of a documented outage and includes the required signature. Candidate B: Mine was late because of a documented outage but lacks the signature.","questions":{"decision":{"type":"choice","instructions":"Select the submission disqualified specifically by the signature exclusion despite the outage exception.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=a5b234c2e4fb304f1071","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":105,"line_sha256":"dd3378780bc7ad688a83f47c1dbf5e86c213ca3cfc7550e79e6b938922f21b97","semif_id":"4da6aedc1139224eeff2","family":"candidate_selection","variant":"criterion_reversal","group_id":"a5b234c2e4fb304f1071","label":1,"option_order":["insufficient","B","A"]}},{"id":"semif_a22dab8ba073ff6cdb8d","source":"semif_authored144","public":true,"request":{"state":"Accept late submissions for documented outages, except that missing signatures always disqualify. Candidate A: My late outage-related submission lacks the signature. Candidate B: My late outage-related submission has the signature. Both outages are documented.","questions":{"decision":{"type":"choice","instructions":"Select the submission qualifying under the stated exception and exclusion.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=a5b234c2e4fb304f1071","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":106,"line_sha256":"786e99c38e1c8fa4ad1ac4fa772f7fa79e85c3ce07e190fbedb69fa98153db19","semif_id":"a22dab8ba073ff6cdb8d","family":"candidate_selection","variant":"evidence_change","group_id":"a5b234c2e4fb304f1071","label":1,"option_order":["insufficient","B","A"]}},{"id":"semif_42c54fa2ad45743e24d8","source":"semif_authored144","public":true,"request":{"state":"Accept late submissions for documented outages, except that missing signatures always disqualify. Candidate A and Candidate B report late submissions but provide neither outage documentation nor signature information.","questions":{"decision":{"type":"choice","instructions":"Select the submission qualifying under the stated exception and exclusion.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=a5b234c2e4fb304f1071","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":107,"line_sha256":"4e686a729a1cd636c46b8915083c9e773501c55305d2107f455a4a2c0e16f842","semif_id":"42c54fa2ad45743e24d8","family":"candidate_selection","variant":"missing","group_id":"a5b234c2e4fb304f1071","label":1,"option_order":["A","insufficient","B"]}},{"id":"semif_f3d13cf184f916677def","source":"semif_authored144","public":true,"request":{"state":"The desk log says the crate is in storage. The warehouse inventory says it has left storage. Both reports are current. The supplied protocol says the warehouse inventory controls location decisions whenever these two sources conflict.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: under the supplied protocol, the crate is currently in storage.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=d142f7fd0efa0eddc788","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":108,"line_sha256":"fe1bd700290ca4326b420a6bcc56d29ff68b13059139ad0f0d4f3f852cf80583","semif_id":"f3d13cf184f916677def","family":"evidence_interpretation","variant":"original","group_id":"d142f7fd0efa0eddc788","label":0,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_1105577da4c8dad4609d","source":"semif_authored144","public":true,"request":{"state":"The desk log says the crate is in storage. The warehouse inventory says it has left storage. Both reports are current. The supplied protocol says the warehouse inventory controls location decisions whenever these two sources conflict.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the desk log reports the crate in storage.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=d142f7fd0efa0eddc788","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":109,"line_sha256":"8cbd882135506ead376a5a0c54728802f4dbefc47e8c031dc0c191470c88585c","semif_id":"1105577da4c8dad4609d","family":"evidence_interpretation","variant":"criterion_reversal","group_id":"d142f7fd0efa0eddc788","label":2,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_0ebd299739302d8b057b","source":"semif_authored144","public":true,"request":{"state":"The desk log says the crate has left storage; the warehouse inventory says it is in storage. Both reports are current, and the inventory controls conflicts under the protocol.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: under the supplied protocol, the crate is currently in storage.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=d142f7fd0efa0eddc788","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":110,"line_sha256":"a9d78891e36ebb920e80ede8c7fb8e78dbffa16049ddb4298545dff4041ba15c","semif_id":"0ebd299739302d8b057b","family":"evidence_interpretation","variant":"evidence_change","group_id":"d142f7fd0efa0eddc788","label":2,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_eafc22c8c40df3932a8e","source":"semif_authored144","public":true,"request":{"state":"The protocol gives the inventory priority, but the current inventory and desk-log entries for the crate are omitted.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: under the supplied protocol, the crate is currently in storage.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=d142f7fd0efa0eddc788","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":111,"line_sha256":"75e8f28b7a1b0ae9f2b4347d08e6c3c9a68376450ec7dac560e80004ba6bd2a8","semif_id":"eafc22c8c40df3932a8e","family":"evidence_interpretation","variant":"missing","group_id":"d142f7fd0efa0eddc788","label":2,"option_order":["supported","contradicted","insufficient"]}},{"id":"semif_0aa1447680e4f25862bb","source":"semif_authored144","public":true,"request":{"state":"The desk clerk reports the equipment deposit as paid. The signed ledger reports it as unpaid. Both reports are current.","questions":{"decision":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; in conflicts, the signed ledger overrides the desk clerk. May it be released?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"media":null,"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=3476d3f4f32290fe57c3","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":112,"line_sha256":"663d344f68cc3d1b1d93b2049cf605521821227f0f4e69f84ed910f38d6ffc2d","semif_id":"0aa1447680e4f25862bb","family":"rule_application","variant":"original","group_id":"3476d3f4f32290fe57c3","label":1,"option_order":["insufficient","prohibited","permitted"]}},{"id":"semif_980e76145c925747220c","source":"semif_authored144","public":true,"request":{"state":"The desk clerk reports the equipment deposit as paid. The signed ledger reports it as unpaid. Both reports are current.","questions":{"decision":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; under this alternative protocol, the desk clerk overrides the ledger. May it be released?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=3476d3f4f32290fe57c3","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":113,"line_sha256":"4af81ff328aa5061c5a8ce8fbe728a292965828a2b94e13f1d220db3d244c8c9","semif_id":"980e76145c925747220c","family":"rule_application","variant":"criterion_reversal","group_id":"3476d3f4f32290fe57c3","label":2,"option_order":["insufficient","prohibited","permitted"]}},{"id":"semif_4aee37a6a155747766aa","source":"semif_authored144","public":true,"request":{"state":"The desk clerk reports the deposit unpaid, while the signed ledger reports it paid. Both reports are current.","questions":{"decision":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; in conflicts, the signed ledger overrides the desk clerk. May it be released?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=3476d3f4f32290fe57c3","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":114,"line_sha256":"1df9d665b797e8cc467e4860bc57e75e7db135844396f0d930021c218b6c122e","semif_id":"4aee37a6a155747766aa","family":"rule_application","variant":"evidence_change","group_id":"3476d3f4f32290fe57c3","label":0,"option_order":["permitted","prohibited","insufficient"]}},{"id":"semif_6db17cc22bd656d78558","source":"semif_authored144","public":true,"request":{"state":"The record mentions the desk clerk and ledger but supplies neither deposit status.","questions":{"decision":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; in conflicts, the signed ledger overrides the desk clerk. May it be released?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=3476d3f4f32290fe57c3","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":115,"line_sha256":"4eed9d2417217f0e18e4a9fdb0ece6f6b28c85cdad8fbf6c8bbc717fe6812208","semif_id":"6db17cc22bd656d78558","family":"rule_application","variant":"missing","group_id":"3476d3f4f32290fe57c3","label":2,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_ef3e0f8e657bf59f8a29","source":"semif_authored144","public":true,"request":{"state":"For this task, the current signed checklist controls readiness when it conflicts with chat. Candidate A: Chat says ready, but the signed checklist says not ready. Candidate B: Chat says not ready, but the signed checklist says ready.","questions":{"decision":{"type":"choice","instructions":"Select the candidate ready according to the controlling signed checklist.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=original group=c979db7416b206c1eb09","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":116,"line_sha256":"e6e3d005323ea445c880706c2b7c483f8103e545dd7bd5863d542f151aabb700","semif_id":"ef3e0f8e657bf59f8a29","family":"candidate_selection","variant":"original","group_id":"c979db7416b206c1eb09","label":2,"option_order":["A","insufficient","B"]}},{"id":"semif_f756e1cb04e11766341e","source":"semif_authored144","public":true,"request":{"state":"For this task, the current signed checklist controls readiness when it conflicts with chat. Candidate A: Chat says ready, but the signed checklist says not ready. Candidate B: Chat says not ready, but the signed checklist says ready.","questions":{"decision":{"type":"choice","instructions":"Under the alternative protocol, chat controls conflicts. Select the candidate ready according to chat.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}}}},"media":null,"gold":{"decision":"A"},"note":"family=candidate_selection variant=criterion_reversal group=c979db7416b206c1eb09","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":117,"line_sha256":"f0a758f87a1afcca4202f14b9184e7071208f6ff36e3e527f42896fffddbd8b2","semif_id":"f756e1cb04e11766341e","family":"candidate_selection","variant":"criterion_reversal","group_id":"c979db7416b206c1eb09","label":1,"option_order":["insufficient","A","B"]}},{"id":"semif_d167e51bb1f499099645","source":"semif_authored144","public":true,"request":{"state":"The signed checklist controls conflicts. Candidate A: The checklist says ready and chat says not ready. Candidate B: The checklist says not ready and chat says ready.","questions":{"decision":{"type":"choice","instructions":"Select the candidate ready according to the controlling signed checklist.","criteria":{"A":"Candidate A","B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence"}}}},"media":null,"gold":{"decision":"A"},"note":"family=candidate_selection variant=evidence_change group=c979db7416b206c1eb09","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":118,"line_sha256":"49a29277e3e839829dc1ecb8418663edbb291ac8002d1c0780e226d41bba02c5","semif_id":"d167e51bb1f499099645","family":"candidate_selection","variant":"evidence_change","group_id":"c979db7416b206c1eb09","label":0,"option_order":["A","B","insufficient"]}},{"id":"semif_674ca4e9346216255466","source":"semif_authored144","public":true,"request":{"state":"The signed checklist controls conflicts, but Candidate A and Candidate B provide only project names without checklist or chat status.","questions":{"decision":{"type":"choice","instructions":"Select the candidate ready according to the controlling signed checklist.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=c979db7416b206c1eb09","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":119,"line_sha256":"1877cfb79eb571e504d769e4959804ebd0113011382ae0d3b27e477bf730ee1e","semif_id":"674ca4e9346216255466","family":"candidate_selection","variant":"missing","group_id":"c979db7416b206c1eb09","label":0,"option_order":["insufficient","B","A"]}},{"id":"semif_1df851a202994a07f123","source":"semif_authored144","public":true,"request":{"state":"The agreed handover requires both the source files and the user guide. The delivery record confirms the source files arrived and explicitly says the guide was not delivered.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the agreed handover is complete.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=f7403e39d6796072491c","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":120,"line_sha256":"5ae9cfe52b97a1b316faa13cf482bc8aae9db46079822741de886e04157b3a98","semif_id":"1df851a202994a07f123","family":"evidence_interpretation","variant":"original","group_id":"f7403e39d6796072491c","label":0,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_b5e6788d95b8395d2888","source":"semif_authored144","public":true,"request":{"state":"The agreed handover requires both the source files and the user guide. The delivery record confirms the source files arrived and explicitly says the guide was not delivered.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the source files were delivered.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=f7403e39d6796072491c","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":121,"line_sha256":"e9be942d75be4631aa06905bf1326d2b4de5c8ccf9d81587fbe67169144c74f5","semif_id":"b5e6788d95b8395d2888","family":"evidence_interpretation","variant":"criterion_reversal","group_id":"f7403e39d6796072491c","label":2,"option_order":["insufficient","contradicted","supported"]}},{"id":"semif_32f36cd0fbbfb3604586","source":"semif_authored144","public":true,"request":{"state":"The agreed handover requires the source files and user guide. The delivery record confirms that both arrived.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the agreed handover is complete.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=f7403e39d6796072491c","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":122,"line_sha256":"41f9cc53a28c7a1bdcb503b448132898c3d541639442deba31e10980d021d985","semif_id":"32f36cd0fbbfb3604586","family":"evidence_interpretation","variant":"evidence_change","group_id":"f7403e39d6796072491c","label":0,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_843185706adff0c60f8b","source":"semif_authored144","public":true,"request":{"state":"The agreement requires source files and a user guide, but the delivery record gives no information about either item.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the agreed handover is complete.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=f7403e39d6796072491c","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":123,"line_sha256":"e5b92d781979799529c52a3a0207ce35c44e9b5da47ca44ac01a6baf87533ce1","semif_id":"843185706adff0c60f8b","family":"evidence_interpretation","variant":"missing","group_id":"f7403e39d6796072491c","label":2,"option_order":["contradicted","supported","insufficient"]}},{"id":"semif_b531dd603bf080dab820","source":"semif_authored144","public":true,"request":{"state":"The contractor supplied the translated manual but explicitly did not supply the glossary.","questions":{"decision":{"type":"choice","instructions":"Approve final handover exactly when both the translated manual and glossary have been supplied. May final handover be approved?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=a060fd4621be415e9f1a","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":124,"line_sha256":"502a957fdd6df3e2a31de0c66c4d9809bb0d3a2ee4cdd3f70b63a19d1c8fd872","semif_id":"b531dd603bf080dab820","family":"rule_application","variant":"original","group_id":"a060fd4621be415e9f1a","label":0,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_be2dd67bc733c1213bef","source":"semif_authored144","public":true,"request":{"state":"The contractor supplied the translated manual but explicitly did not supply the glossary.","questions":{"decision":{"type":"choice","instructions":"Approve the translation milestone when the manual has been supplied; the glossary is not required for this milestone. May this milestone be approved?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=a060fd4621be415e9f1a","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":125,"line_sha256":"bd71f0a8e940463d081b4ad6b063dba70101621185e734e3c1d9170980f26f74","semif_id":"be2dd67bc733c1213bef","family":"rule_application","variant":"criterion_reversal","group_id":"a060fd4621be415e9f1a","label":1,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_b8e2750307df9761e716","source":"semif_authored144","public":true,"request":{"state":"The contractor supplied both the translated manual and the glossary.","questions":{"decision":{"type":"choice","instructions":"Approve final handover exactly when both the translated manual and glossary have been supplied. May final handover be approved?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=a060fd4621be415e9f1a","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":126,"line_sha256":"54cb789b7b1e1577718302be552b22db31fd71e109eeaddd007bd758cc046ccf","semif_id":"b8e2750307df9761e716","family":"rule_application","variant":"evidence_change","group_id":"a060fd4621be415e9f1a","label":0,"option_order":["permitted","prohibited","insufficient"]}},{"id":"semif_5d31b4bf2d0fc193f53b","source":"semif_authored144","public":true,"request":{"state":"The contractor reports working on the handover but supplies no delivery record for either item.","questions":{"decision":{"type":"choice","instructions":"Approve final handover exactly when both the translated manual and glossary have been supplied. May final handover be approved?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=a060fd4621be415e9f1a","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":127,"line_sha256":"b8c0bfba132eefb32bc616bbf2cef79af3286d7ff4d49d6c40bd1dfc5e831a01","semif_id":"5d31b4bf2d0fc193f53b","family":"rule_application","variant":"missing","group_id":"a060fd4621be415e9f1a","label":1,"option_order":["permitted","insufficient","prohibited"]}},{"id":"semif_a9230edfc6ec9e524af0","source":"semif_authored144","public":true,"request":{"state":"A complete incident package requires both a timeline and a remedy report. Candidate A: Both documents are attached. Candidate B: Only the timeline is attached; the remedy report is absent.","questions":{"decision":{"type":"choice","instructions":"Select the complete incident package.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}}}},"media":null,"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=2b0f8eda8903e38a47de","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":128,"line_sha256":"cfb4f0a4ebc960e89a40dab1b5fe85ec1a5c7d1658eb6d6327a5adfc8a86d15c","semif_id":"a9230edfc6ec9e524af0","family":"candidate_selection","variant":"original","group_id":"2b0f8eda8903e38a47de","label":2,"option_order":["B","insufficient","A"]}},{"id":"semif_c100e84585149a04be49","source":"semif_authored144","public":true,"request":{"state":"A complete incident package requires both a timeline and a remedy report. Candidate A: Both documents are attached. Candidate B: Only the timeline is attached; the remedy report is absent.","questions":{"decision":{"type":"choice","instructions":"Select the package containing the timeline but missing the remedy report.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=2b0f8eda8903e38a47de","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":129,"line_sha256":"bb70f424172e0290c15b8a9d69ac3e01045e3979dcb222c597afefd5791b79d5","semif_id":"c100e84585149a04be49","family":"candidate_selection","variant":"criterion_reversal","group_id":"2b0f8eda8903e38a47de","label":0,"option_order":["B","A","insufficient"]}},{"id":"semif_579beed184ffc1104aa9","source":"semif_authored144","public":true,"request":{"state":"A complete package requires both documents. Candidate A: The timeline is attached but the remedy report is absent. Candidate B: Both the timeline and remedy report are attached.","questions":{"decision":{"type":"choice","instructions":"Select the complete incident package.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=2b0f8eda8903e38a47de","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":130,"line_sha256":"214117aff4ea34fdb07a68340eee580b04b114d74590c6ced3eb9a1065aec442","semif_id":"579beed184ffc1104aa9","family":"candidate_selection","variant":"evidence_change","group_id":"2b0f8eda8903e38a47de","label":0,"option_order":["B","A","insufficient"]}},{"id":"semif_c4e9bd463b6fde5be743","source":"semif_authored144","public":true,"request":{"state":"A complete package requires both documents. Candidate A and Candidate B give only package titles and no attachment information.","questions":{"decision":{"type":"choice","instructions":"Select the complete incident package.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=2b0f8eda8903e38a47de","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":131,"line_sha256":"6ca1b841866f92dc39e924111bf359b9d3c6974ddd92be25b2d0a37da7b857c2","semif_id":"c4e9bd463b6fde5be743","family":"candidate_selection","variant":"missing","group_id":"2b0f8eda8903e38a47de","label":1,"option_order":["B","insufficient","A"]}},{"id":"semif_b863998d4be2721a5dfd","source":"semif_authored144","public":true,"request":{"state":"The routing rule assigns repairs to the in-house workshop when it can perform the work; use the outside workshop only when the in-house workshop cannot. Both workshops confirm they can perform this repair.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the routing rule assigns this repair to the outside workshop.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"contradicted"},"note":"family=evidence_interpretation variant=original group=01020a1eaf46ec2fba5d","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":132,"line_sha256":"bae89fc634702f230c942773ef3b0d1f8ed6ae8940147ad73d34cb55938e0079","semif_id":"b863998d4be2721a5dfd","family":"evidence_interpretation","variant":"original","group_id":"01020a1eaf46ec2fba5d","label":2,"option_order":["supported","insufficient","contradicted"]}},{"id":"semif_37cebc3792e98049439f","source":"semif_authored144","public":true,"request":{"state":"The routing rule assigns repairs to the in-house workshop when it can perform the work; use the outside workshop only when the in-house workshop cannot. Both workshops confirm they can perform this repair.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the outside workshop can perform this repair.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=criterion_reversal group=01020a1eaf46ec2fba5d","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":133,"line_sha256":"7c2295b4f5f42981dc26faa73d1b81c448f5fd35781fc9653e6743f0fa486fc8","semif_id":"37cebc3792e98049439f","family":"evidence_interpretation","variant":"criterion_reversal","group_id":"01020a1eaf46ec2fba5d","label":1,"option_order":["insufficient","supported","contradicted"]}},{"id":"semif_64d63f86e328aff1481c","source":"semif_authored144","public":true,"request":{"state":"The routing rule assigns repairs to the in-house workshop when it can perform the work; use the outside workshop only when the in-house workshop cannot. The in-house workshop cannot perform this repair; the outside workshop can.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the routing rule assigns this repair to the outside workshop.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"supported"},"note":"family=evidence_interpretation variant=evidence_change group=01020a1eaf46ec2fba5d","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":134,"line_sha256":"d05f8d4c70bfccc253e82b18d18196370a1116b551a9494130a6c2c67fb20d91","semif_id":"64d63f86e328aff1481c","family":"evidence_interpretation","variant":"evidence_change","group_id":"01020a1eaf46ec2fba5d","label":2,"option_order":["contradicted","insufficient","supported"]}},{"id":"semif_6d64a8459f64eac7f4cc","source":"semif_authored144","public":true,"request":{"state":"The routing rule assigns repairs to the in-house workshop when it can perform the work; use the outside workshop only when the in-house workshop cannot. Neither workshop's ability to perform this repair is supplied.","questions":{"decision":{"type":"choice","instructions":"Assess the claim: the routing rule assigns this repair to the outside workshop.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=evidence_interpretation variant=missing group=01020a1eaf46ec2fba5d","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":135,"line_sha256":"39c08980690a5cd793c933f91c208b7151b2184a0ad54ceea9d92e41a3973188","semif_id":"6d64a8459f64eac7f4cc","family":"evidence_interpretation","variant":"missing","group_id":"01020a1eaf46ec2fba5d","label":0,"option_order":["insufficient","contradicted","supported"]}},{"id":"semif_2708193212d8a4a523c7","source":"semif_authored144","public":true,"request":{"state":"The town hall and school both confirm they are available and suitable for the meeting.","questions":{"decision":{"type":"choice","instructions":"Book the school only as a fallback when the town hall is unavailable or unsuitable; otherwise book the town hall. May the school be booked under this policy?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"prohibited"},"note":"family=rule_application variant=original group=6a06a2f278301c1412ee","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":136,"line_sha256":"ad0ef7c8d3399bcd03f5dff1f65e2dbe800608405af114698003bb1779e3b594","semif_id":"2708193212d8a4a523c7","family":"rule_application","variant":"original","group_id":"6a06a2f278301c1412ee","label":0,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_5a37d1dfcad20810a76a","source":"semif_authored144","public":true,"request":{"state":"The town hall and school both confirm they are available and suitable for the meeting.","questions":{"decision":{"type":"choice","instructions":"Under the alternative policy, either suitable available venue may be booked without priority. May the school be booked?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=criterion_reversal group=6a06a2f278301c1412ee","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":137,"line_sha256":"93feae55494ec8b724b53fea0f53ae766f729a277a2d981aa70b2d50aede3511","semif_id":"5a37d1dfcad20810a76a","family":"rule_application","variant":"criterion_reversal","group_id":"6a06a2f278301c1412ee","label":1,"option_order":["prohibited","permitted","insufficient"]}},{"id":"semif_acacd8b46acd25cb954b","source":"semif_authored144","public":true,"request":{"state":"The town hall is confirmed unavailable, while the school is available and suitable.","questions":{"decision":{"type":"choice","instructions":"Book the school only as a fallback when the town hall is unavailable or unsuitable; otherwise book the town hall. May the school be booked under this policy?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}}}},"media":null,"gold":{"decision":"permitted"},"note":"family=rule_application variant=evidence_change group=6a06a2f278301c1412ee","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":138,"line_sha256":"2b02d4570f70e00cb67db4f8a6ae1fef2024fef2b22a46f0df44b9889685c033","semif_id":"acacd8b46acd25cb954b","family":"rule_application","variant":"evidence_change","group_id":"6a06a2f278301c1412ee","label":1,"option_order":["insufficient","permitted","prohibited"]}},{"id":"semif_4089c199b8b796ec3ed8","source":"semif_authored144","public":true,"request":{"state":"The two venues are named, but neither availability nor suitability is provided.","questions":{"decision":{"type":"choice","instructions":"Book the school only as a fallback when the town hall is unavailable or unsuitable; otherwise book the town hall. May the school be booked under this policy?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=rule_application variant=missing group=6a06a2f278301c1412ee","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":139,"line_sha256":"24f452426c2249ee67e1791b97b0b4330d2f3b94cdfbc91447a9a04c9289e914","semif_id":"4089c199b8b796ec3ed8","family":"rule_application","variant":"missing","group_id":"6a06a2f278301c1412ee","label":0,"option_order":["insufficient","prohibited","permitted"]}},{"id":"semif_02f2b95a5c5a4777a45c","source":"semif_authored144","public":true,"request":{"state":"Candidate A is the preferred translator and is confirmed able to handle this language. Candidate B is the fallback translator and is also able to handle it.","questions":{"decision":{"type":"choice","instructions":"Choose A whenever A is able; choose B only when A is unable and B is able.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}}}},"media":null,"gold":{"decision":"A"},"note":"family=candidate_selection variant=original group=2797f8c23d05c0ee06b3","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":140,"line_sha256":"c0d13120fc1e3b4366104c2b23058225cbc636563e4e65ed9b52a5c358379e96","semif_id":"02f2b95a5c5a4777a45c","family":"candidate_selection","variant":"original","group_id":"2797f8c23d05c0ee06b3","label":1,"option_order":["B","A","insufficient"]}},{"id":"semif_ebea4681e3dcc7d92ca2","source":"semif_authored144","public":true,"request":{"state":"Candidate A is the preferred translator and is confirmed able to handle this language. Candidate B is the fallback translator and is also able to handle it.","questions":{"decision":{"type":"choice","instructions":"Under the alternative priority, choose B whenever B is able; choose A only when B is unable and A is able.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=criterion_reversal group=2797f8c23d05c0ee06b3","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":141,"line_sha256":"f72dabc70f174d755bfc975fe608bb7daa17d7ba8bd97177a3e6ee81c8d0dbe0","semif_id":"ebea4681e3dcc7d92ca2","family":"candidate_selection","variant":"criterion_reversal","group_id":"2797f8c23d05c0ee06b3","label":1,"option_order":["insufficient","B","A"]}},{"id":"semif_b8c5b9b285cbdddc9f8d","source":"semif_authored144","public":true,"request":{"state":"Candidate A is the preferred translator but is confirmed unable to handle this language. Candidate B is the fallback and is confirmed able.","questions":{"decision":{"type":"choice","instructions":"Choose A whenever A is able; choose B only when A is unable and B is able.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}}}},"media":null,"gold":{"decision":"B"},"note":"family=candidate_selection variant=evidence_change group=2797f8c23d05c0ee06b3","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":142,"line_sha256":"2efe29ffb7db399f3bba4bbd7fd8ae18c674619aa9dbb29a38bc7cf349eb843e","semif_id":"b8c5b9b285cbdddc9f8d","family":"candidate_selection","variant":"evidence_change","group_id":"2797f8c23d05c0ee06b3","label":2,"option_order":["A","insufficient","B"]}},{"id":"semif_7ed75d854d99b3a81916","source":"semif_authored144","public":true,"request":{"state":"Candidate A is preferred and Candidate B is the fallback, but neither candidate's ability to handle the language is supplied.","questions":{"decision":{"type":"choice","instructions":"Choose A whenever A is able; choose B only when A is unable and B is able.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}}}},"media":null,"gold":{"decision":"insufficient"},"note":"family=candidate_selection variant=missing group=2797f8c23d05c0ee06b3","provenance":{"file":"benchmarks/data/authored144.jsonl","repo":"github.com/TheoLeeCJ/SemIf","commit":"ca3ba65f142967030ecb453346e94d6f476a69df","file_sha256":"8162d1c73f925af64453f1ec05ef36d583b3815bf698e60f0d454bd11537e079","license":"MIT","line":143,"line_sha256":"7e26c499b875039865b9a22e2c91b69239dee58b604b45d436ecda9f483b57d2","semif_id":"7ed75d854d99b3a81916","family":"candidate_selection","variant":"missing","group_id":"2797f8c23d05c0ee06b3","label":0,"option_order":["insufficient","B","A"]}},{"id":"own_t01","source":"own_text","public":true,"request":{"state":"Ticket #4471 - opened 2026-03-14 09:12\nFrom: Dana (customer since 2023)\nSubject: Charged twice for March\n\nHi, I'm on the Pennimere Meal Kits weekly plan. My card statement shows two charges of 64.90 on March 1, but I only received one box. I already contacted my bank and they told me to ask you first. Please refund the duplicate charge. I'd also like to pause deliveries for two weeks while I'm travelling, starting March 20. Thanks, Dana","questions":{"team":{"type":"choice","instructions":"Which team should handle this ticket first?","criteria":{"billing":"Charges, invoices and refunds","deliveries":"Boxes, couriers and delivery dates","account":"Sign-in, plan changes and pauses","technical":"App or website faults","other":null}},"several_actions":{"type":"noul","instructions":"Does the ticket ask for more than one action?"},"upset":{"type":"score","instructions":"How upset is the customer?","criteria":["calm","mildly annoyed","frustrated","angry"]},"refund":{"type":"noul","instructions":"Refund request?","criteria":{"true":"The customer explicitly asks for money back","false":"No refund is requested"}}}},"media":null,"gold":{"team":"billing","several_actions":"true","upset":"1","refund":"true"},"note":"support ticket","provenance":{"from":"the Kev port's own records","status":"published in conversion/kev/own_records_public.json","request_sha256":"bdcf066020a5ddd181a2c35436bc71d984771dfe00a3985d9716afd44a05ef6f"}},{"id":"own_t04","source":"own_text","public":true,"request":{"state":"Quennick Labs - mobile team stand-up, Wednesday\nPresent: Aiko, Ben, Chloé, Femi\nAiko: finished the offline sync screen; waiting on design review before merging.\nBen: the crash on login for Android 12 users is still open; he can reproduce it only on one test phone. Blocked until the device lab returns the second phone on Friday.\nChloé: release notes for 4.2 drafted; needs numbers from Femi.\nFemi: analytics export fixed; will send Chloé the weekly active user numbers today.\nDecisions: release 4.2 moves from Thursday to Monday because of the login crash. Ben owns the crash; Aiko will pair with him tomorrow.","questions":{"crash_owner":{"type":"choice","instructions":"Who owns the login crash?","criteria":{"aiko":"Aiko","ben":"Ben","chloe":"Chloé","femi":"Femi"}},"blocked":{"type":"noul","instructions":"Is any team member blocked?"},"release_day":{"type":"choice","instructions":"When is release 4.2 now planned?","criteria":{"thursday":"Thursday","friday":"Friday","monday":"Monday","unknown":"Not decided"}},"blockers":{"type":"score","instructions":"How many people report a blocker?","criteria":["none","one","two or more"]}}},"media":null,"gold":{"crash_owner":"ben","blocked":"true","release_day":"monday","blockers":"1"},"note":"meeting notes","provenance":{"from":"the Kev port's own records","status":"published in conversion/kev/own_records_public.json","request_sha256":"8438ab4a1d858ad47f2297849e2cdf1f74d82cfae449746e1fd714191f1ba81a"}},{"id":"own_t06","source":"own_text","public":true,"request":{"state":"[19:02] Guest (room 314): hi, the heating in my room isn't working and it's freezing\n[19:03] Front desk (Wendlecote Hotel): I'm sorry to hear that! I'll send someone from maintenance right away.\n[19:04] Guest: thanks. also can I keep the room until 2pm tomorrow? my train is at 3\n[19:06] Front desk: Late checkout until 14:00 is possible for 20 €, shall I add it?\n[19:06] Guest: yes please\n[19:25] Guest: still nobody came and it's getting colder\n[19:26] Front desk: Apologies, our technician is on another call. He will be with you in 10 minutes, and we can move you to room 402 if you prefer.","questions":{"first_request":{"type":"choice","instructions":"What did the guest ask for first?","criteria":{"room_service":"Food or drinks to the room","late_checkout":"Leave the room later than usual","towels":"Extra towels or linen","maintenance":"Something in the room is broken","billing":"A question about the bill","booking_change":"Change the dates or room type","lost_item":"Something was lost","noise":"A noise complaint","transport":"Help with a taxi or train","other":null}},"fixed":{"type":"noul","instructions":"Has the heating problem been fixed by the end of the chat?"},"patience":{"type":"score","instructions":"How patient is the guest by the end?","criteria":["patient","impatient","angry"]}}},"media":null,"gold":{"first_request":"maintenance","fixed":"false","patience":"1"},"note":"chat log, 10-option choice","provenance":{"from":"the Kev port's own records","status":"published in conversion/kev/own_records_public.json","request_sha256":"b623e06749da1fb759facad5993c0abcf16f38a265841eec4b83ffa49e5c5150"}},{"id":"own_t07","source":"own_text","public":true,"request":{"state":"Ostlebury Dental - voicemail transcript, Friday 16:48\n'Hello, this is Jonas. I have a cleaning appointment on Tuesday at 9:30 with Dr Amal. Something came up at work, so I need to move it. I can do Wednesday after 15:00 or any time on Thursday morning. Please call me back to confirm. I can't check messages during the day, so after 18:00 is best. Thank you.'\nFree slots next week: Tue 11:00, Wed 15:30, Wed 17:00, Thu 08:30, Thu 12:30.","questions":{"slot":{"type":"choice","instructions":"Which of these free slots fits the caller?","criteria":{"tue_1100":"Tuesday 11:00","wed_1530":"Wednesday 15:30","thu_1230":"Thursday 12:30","none":"None of them fits"}}}},"media":null,"gold":{"slot":"wed_1530"},"note":"voicemail + scheduling","provenance":{"from":"the Kev port's own records","status":"published in conversion/kev/own_records_public.json","request_sha256":"32dd7b8e619f0d0d3b514ab4fb2fa13b25a9362e2783353ab865c9f632629576"}},{"id":"own_t10","source":"own_text","public":true,"request":{"state":"To the managing agent, Pennimere Court\n\nI live in flat 9 and I am writing, for the third time this month, about the noise from the gym equipment in the basement. The machines are used until after midnight, and the thumping comes up through the floor of my bedroom. I have asked for the opening hours to be limited to 07:00-22:00, as the house rules say. If nothing changes by the end of next week, I will raise it with the residents' committee.\n\nYours, Teodora (flat 9)","questions":{"topic":{"type":"choice","instructions":"What is the letter about?","criteria":{"noise":"Noise","repairs":"Repairs","rent":"Rent or charges","parking":"Parking","pets":"Pets"}},"first_time":{"type":"noul","instructions":"Is this the first time the writer complains?"},"escalation":{"type":"score","instructions":"How far is the writer prepared to take this?","criteria":["no further step","a reminder","the residents' committee","a lawyer or court"]}}},"media":null,"gold":{"topic":"noise","first_time":"false","escalation":"2"},"note":"complaint letter","provenance":{"from":"the Kev port's own records","status":"published in conversion/kev/own_records_public.json","request_sha256":"d59f5cd3879740ed999247f8c4dde43654a853cee13b612397c96c3a8bb3524e"}},{"id":"own_j01","source":"own_json","public":true,"request":{"state":{"invoice":"FPW-2026-0412","issuer":"Fennistone Print Works","customer":"Ostlebury Community Choir","issued":"2026-04-02","due":"2026-05-02","status":"unpaid","reminders_sent":2,"lines":[{"item":"Concert posters A2","qty":40,"unit_price":1.85},{"item":"Programme booklets","qty":250,"unit_price":0.62},{"item":"Delivery","qty":1,"unit_price":12.0}],"subtotal":241.0,"vat_rate":0.2,"total":289.2,"payment_terms":"bank transfer within 30 days","today":"2026-05-19"},"questions":{"overdue":{"type":"noul","instructions":"Is the invoice overdue?"},"method":{"type":"choice","instructions":"How is the customer expected to pay?","criteria":{"card":null,"bank_transfer":null,"cash":null,"cheque":null}},"size":{"type":"score","instructions":"How large is the invoice total?","criteria":["under 100","100 to 250","250 to 500","over 500"]}}},"media":null,"gold":{"overdue":"true","method":"bank_transfer","size":"2"},"note":"invoice (object state)","provenance":{"from":"the Kev port's own records","status":"published in conversion/kev/own_records_public.json","request_sha256":"58806342f4cb3636227489177db86fef2b32d220ec15a1d0ddfda413a8661b57"}},{"id":"own_j02","source":"own_json","public":true,"request":{"state":{"order_id":"PF-88213","shop":"Pennimere Foods","placed":"2026-06-03 18:22","items":[{"sku":"OAT-1KG","name":"Rolled oats 1 kg","qty":2,"status":"shipped"},{"sku":"HNY-450","name":"Wildflower honey 450 g","qty":1,"status":"out of stock"},{"sku":"TEA-GRN","name":"Green tea, 50 bags","qty":3,"status":"shipped"}],"shipping":{"carrier":"regional courier","tracking_status":"delivered","delivered_at":"2026-06-05 11:40"},"payment":{"method":"card","captured":27.4,"refunded":0.0},"customer_note":"Where is my honey? I was charged for it."},"questions":{"next_action":{"type":"choice","instructions":"What should the shop do next?","criteria":{"refund_missing_item":"Refund the item that was not sent","resend_order":"Send the whole order again","track_parcel":"Ask the courier where the parcel is","cancel_order":"Cancel the order","change_address":"Update the delivery address","discount_code":"Give a discount code","replace_damaged":"Replace damaged goods","confirm_delivery":"Tell the customer the parcel was delivered","fraud_review":"Flag the order for fraud review","no_action":"Nothing needs to be done"}},"partial":{"type":"noul","instructions":"Was only part of the order shipped?"},"charged_claim":{"type":"noul","instructions":"Does the customer say they paid for something they did not receive?"}}},"media":null,"gold":{"next_action":"refund_missing_item","partial":"true","charged_claim":"true"},"note":"order (object state), 10-option choice","provenance":{"from":"the Kev port's own records","status":"published in conversion/kev/own_records_public.json","request_sha256":"7edbced3281625cc811a6e12bbfb343102d0d0d9ba4eedff00f847492a7de3e6"}},{"id":"own_a01","source":"own_json","public":true,"request":{"state":[{"ticket":"HD-101","summary":"Printer on floor 2 jams on every job","reported":"09:05","affects":"1 team"},{"ticket":"HD-102","summary":"Card readers at the main entrance reject all badges","reported":"09:12","affects":"whole building"},{"ticket":"HD-103","summary":"Request for a second monitor","reported":"09:30","affects":"1 person"},{"ticket":"HD-104","summary":"Smell of burning plastic near the server room","reported":"09:41","affects":"unknown"}],"questions":{"first":{"type":"choice","instructions":"Which ticket should be handled first?","criteria":{"HD-101":"Printer jams","HD-102":"Entrance card readers","HD-103":"Second monitor","HD-104":"Burning smell"}},"hazard":{"type":"noul","instructions":"Does any ticket describe a possible safety hazard?"},"load":{"type":"score","instructions":"Is the queue overloaded?","criteria":["normal","overloaded"]}}},"media":null,"gold":{"first":"HD-104","hazard":"true","load":"0"},"note":"helpdesk queue (top-level array state), 2 levels","provenance":{"from":"the Kev port's own records","status":"published in conversion/kev/own_records_public.json","request_sha256":"5bb78aa791604b60ae968d5d71299203031e5a7a76587aed990af25ebfc33b3d"}},{"id":"own_m01","source":"own_text","public":true,"request":{"state":"Flat to let - Brindlecourt Road, Ostlebury\nTwo-bedroom flat on the third floor of a building with a lift. Rent 1,150 per month, deposit equal to five weeks' rent, minimum tenancy 12 months. Available from 1 September. Bills are not included except water. Pets considered by agreement (small dogs and cats only, no other animals). The flat is unfurnished apart from a fridge and a washing machine. Bike storage in the basement; no parking space. Viewings on Saturdays between 10:00 and 13:00. Contact the agent, Pilar, to book a viewing.","questions":{"bedrooms":{"type":"choice","instructions":"How many bedrooms does the flat have?","criteria":{"one":null,"two":null,"three":null}},"lift":{"type":"noul","instructions":"Is there a lift?"},"furnished":{"type":"score","instructions":"How furnished is the flat?","criteria":["unfurnished","partly furnished","fully furnished"]},"rabbit":{"type":"choice","instructions":"Could a tenant keep a rabbit?","criteria":{"yes":"Allowed","by_agreement":"Only with the landlord's agreement","no":"Not allowed"}},"parking":{"type":"noul","instructions":"Does the flat come with a parking space?"},"bill":{"type":"choice","instructions":"Which bill is included in the rent?","criteria":{"electricity":null,"gas":null,"water":null,"internet":null,"none":"No bill is included"}},"weekday_viewing":{"type":"noul","instructions":"Can the flat be viewed on a weekday?"},"term":{"type":"score","instructions":"How long is the minimum tenancy?","criteria":["under 6 months","6 months","12 months","over 12 months"]}}},"media":null,"gold":{"bedrooms":"two","lift":"true","furnished":"0","rabbit":"no","parking":"false","bill":"water","weekday_viewing":"false","term":"2"},"note":"flat listing, 8 questions","provenance":{"from":"the Kev port's own records","status":"published in conversion/kev/own_records_public.json","request_sha256":"a41d0363abf69e602ec3e9790075dfd216d1558add68c98b434367b8c3754b16"}},{"id":"own_L01","source":"own_long","public":true,"request":{"state":"BRINDLECOURT LOGISTICS - parcel hub 2, cross-belt sorter S1, controller log (excerpt)\nshift: early (06:00-14:00)   supervisor on duty: Yusuf   maintenance on call: Malik\n\n06:00:00 INFO  throughput 1712 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n06:03:00 INFO  throughput 1698 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n06:06:00 INFO  throughput 1731 parcels/h; lanes 1-12 active; scanner read rate 99.5%\n06:09:00 INFO  throughput 1704 parcels/h; lanes 1-12 active; scanner read rate 99.7%\n06:09:41 INFO  diverter D4 calibration check passed (offset 0.39 mm)\n06:12:00 INFO  throughput 1689 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n06:12:17 WARN  scanner T2 no-read on parcel 48367; parcel sent to manual lane\n06:15:00 INFO  throughput 1720 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n06:18:00 INFO  throughput 1716 parcels/h; lanes 1-12 active; scanner read rate 99.5%\n06:21:00 INFO  throughput 1702 parcels/h; lanes 1-12 active; scanner read rate 99.7%\n06:24:00 INFO  throughput 1695 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n06:27:00 INFO  throughput 1727 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n06:27:41 INFO  diverter D4 calibration check passed (offset 0.57 mm)\n06:30:00 INFO  throughput 1709 parcels/h; lanes 1-12 active; scanner read rate 99.5%\n06:33:00 INFO  throughput 1683 parcels/h; lanes 1-12 active; scanner read rate 99.7%\n06:33:17 WARN  scanner T2 no-read on parcel 48640; parcel sent to manual lane\n06:36:00 INFO  throughput 1718 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n06:39:00 INFO  throughput 1700 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n06:40:00 INFO  throughput 1694 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n06:41:12 WARN  lane 7 drive motor current 8.9 A (limit 8.0 A) for 5 s\n06:41:58 WARN  lane 7 drive motor current 9.4 A (limit 8.0 A) for 12 s\n06:42:40 WARN  lane 7 belt speed 1.62 m/s (setpoint 1.80 m/s)\n06:43:05 ERROR lane 7 jam detected: photo-eye PE7-3 blocked for more than 3 s\n06:43:06 ERROR sorter S1 stopped by safety controller (cause: jam lane 7); all lanes halted\n06:43:06 INFO  infeed conveyors paused; parcels held on accumulation belts\n06:43:40 INFO  alarm acknowledged by Yusuf at the control desk\n06:44:15 INFO  maintenance call raised to Malik (priority 1)\n06:46:02 INFO  scanner T1 and T2 idle (no parcels); self-test passed\n06:47:30 INFO  accumulation belts at 64% capacity\n06:50:12 INFO  Malik on site at lane 7; lock-out applied to lane 7 drive\n06:52:48 INFO  accumulation belts at 81% capacity; infeed doors 3 and 4 closed to new trailers\n06:55:20 INFO  jam cause reported by Malik: crushed carton wedged under the lane 7 belt at the second roller\n06:58:41 INFO  carton removed from lane 7; roller R7-2 inspected, no damage found\n07:00:05 INFO  lane 7 belt tension measured below specification; adjustment started\n07:03:12 INFO  lane 7 belt tension adjusted; test run at low speed 0.5 m/s for 60 s passed\n07:04:25 INFO  lock-out removed from lane 7 drive\n07:05:30 INFO  sorter S1 restarted by Yusuf; stoppage 22 min 24 s\n07:05:31 INFO  infeed conveyors resumed; infeed doors 3 and 4 reopened\n07:06:00 INFO  lane 7 drive motor current 6.1 A (limit 8.0 A)\n07:09:00 INFO  throughput 1840 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.0%\n07:12:00 INFO  throughput 1866 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.3%\n07:15:00 INFO  throughput 1851 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.6%\n07:15:44 WARN  scanner T1 no-read on parcel 49870; parcel sent to manual lane\n07:18:00 INFO  throughput 1792 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.0%\n07:21:00 INFO  throughput 1748 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.3%\n07:21:09 INFO  accumulation belts back below 10% capacity\n07:24:00 INFO  throughput 1731 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.6%\n07:27:00 INFO  throughput 1722 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.0%\n07:27:30 INFO  lane 7 drive motor current 5.9 A (limit 8.0 A); belt speed 1.80 m/s\n07:30:00 INFO  throughput 1715 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.3%\n07:33:00 INFO  shift note by Yusuf: lane 7 belt tension to be checked again at 10:00; carton supplier to be told about weak boxes\n07:33:05 INFO  sorter S1 running normally; no active alarms","questions":{"component":{"type":"choice","instructions":"Which component caused the stoppage?","criteria":{"scanner":"Barcode scanner","lane_7":"Conveyor lane 7","diverter":"Diverter arm","power":"Power supply","software":"Control software"}},"normal_at_end":{"type":"noul","instructions":"Was the sorter running normally at the end of the log?"},"stoppage":{"type":"score","instructions":"How long was the sorter fully stopped?","criteria":["under 2 minutes","2 to 5 minutes","5 to 10 minutes","10 to 20 minutes","20 to 40 minutes","over 40 minutes"]}}},"media":null,"gold":{"component":"lane_7","normal_at_end":"true","stoppage":"4"},"note":"long state: sorter log","provenance":{"from":"the Kev port's own records","status":"published in conversion/kev/own_records_public.json","request_sha256":"2cfe98ea936f73bcbaafe21c1c1ab72ea35e25f70e23ad1015797c66de384238"}},{"id":"own_L03","source":"own_long","public":true,"request":{"state":"WENDLECOTE RESIDENTS' ASSOCIATION\nMinutes of the Annual General Meeting held on Thursday 12 March 2026 at 19:30 in Wendlecote Village Hall\n\nPresent: 47 members (attendance sheet attached) and 3 guests, among them the ward councillor, Adaeze. In the chair: Gareth. Minutes: Priyanka (secretary).\nApologies: 11 members sent their apologies.\n\n1. Welcome and quorum\nThe chair welcomed members and guests and thanked the hall committee for the use of the room. The constitution requires 30 members to be present for a general meeting to be quorate. With 47 members present, the meeting was declared quorate.\n\n2. Minutes of the last annual general meeting\nThe minutes of the meeting held on 13 March 2025 were approved as a true record, proposed by Hamid and seconded by Ruth. There were no matters arising that were not already on the agenda.\n\n3. Chair's report\nGareth summarised the year. The association ran the summer fair, which raised 2,140 for the hall roof fund, and organised two litter picks along the river path, each with more than twenty volunteers. The campaign for a pedestrian crossing on Mill Lane continued, and the council has now agreed to carry out a traffic survey in May. Membership rose from 212 to 236 households. He thanked the committee and the volunteers who deliver the newsletter to every house four times a year.\n\n4. Treasurer's report\nThe outgoing treasurer, Colm, presented the accounts for the year. Income was 6,480: membership fees 2,832, the summer fair 2,140 and donations 1,508. Expenditure was 4,915, mostly newsletter printing (1,620), insurance (740), hall hire (600) and equipment for the river path clean-up (1,955). The year closed with a surplus of 1,565 and a bank balance of 9,310. The accounts had been checked by an independent examiner, who found no issues. Members approved the accounts by a show of hands, with no votes against.\n\n5. Reports from the working groups\nEnvironment group (Hannah): the group planted 150 native bulbs on the verge outside the school and repaired two of the four bird boxes on the green. It would like to survey the river path for invasive plants in the summer and is looking for two more volunteers.\nEvents group (Dominic): besides the summer fair, the group ran a quiz night in November that broke even. A film night in the hall is planned for April if a licence can be arranged at a reasonable price.\nRoads and transport group (Hamid): apart from the crossing campaign, the group reported three damaged street lights and a blocked drain on Station Road to the council; two of the lights have been repaired.\n\n6. Hall roof fund\nRuth reported for the hall committee that the roof repair has been quoted at 18,400. With the 2,140 from the summer fair, the fund now stands at 11,750. The committee has applied to two charitable trusts for the rest and expects their answers by the end of May. Work cannot start before September because bats roost in the roof space during the summer, and an ecologist will check the roof before any work begins. Members asked whether the association would cover any shortfall; Ruth said that the committee would come back to a general meeting before committing any of the association's money. One member offered to organise a sponsored walk for the fund in June, and several members volunteered to help.\n\n7. Election of officers\nGareth agreed to stand again as chair and was re-elected unopposed. Priyanka was re-elected as secretary unopposed. Colm stood down as treasurer after six years. Two members were nominated for the post: Odette and Lewis. After short statements from both candidates, a secret ballot was held. Odette received 29 votes and Lewis 16, with 2 spoiled papers. Odette was declared elected as treasurer. The meeting thanked Colm warmly for his years of service.\n\n8. Motion 1: bicycle racks on the village green\nProposed by Ines, seconded by Dominic: \"That the association spends up to 1,200 from its reserves on four bicycle racks at the edge of the village green, subject to council permission.\" Ines explained that bicycles are currently chained to the railings, which damages the paint and blocks the footpath for pushchairs. One member asked whether the racks would attract theft; Ines replied that they would stand in view of the shop's camera. Vote: 31 for, 9 against, 7 abstentions. The motion was carried.\n\n9. Motion 2: annual membership fee\nProposed by the committee: \"That the annual fee rises from 12 to 15 per household from 1 April 2026.\" The treasurer-elect said that this year's surplus makes an increase unnecessary for now. Several members said that a higher fee would put off new households, especially in the new flats on Station Road. Vote: 18 for, 26 against, 3 abstentions. The motion was defeated, and the fee stays at 12.\n\n10. Motion 3: community garden\nProposed by Kwame, seconded by Hannah: \"That the association creates a community garden on the unused plot behind the hall.\" Members supported the idea in principle but asked who would look after the garden and whether the hall committee owns the plot. It was agreed to set up a working group of Kwame, Hannah and Lewis to report to the next general meeting. No vote was taken.\n\n11. Any other business\n(a) Speeding on Mill Lane: several members reported cars driving well above the limit at school times. The secretary will write to the council asking for the May survey to include speed data.\n(b) Newsletter: volunteers are needed to deliver on the east side of the village.\n(c) The ward councillor said that the council's grant scheme for community projects opens in June and that the garden might qualify.\n(d) Recycling: the council will collect glass separately from July; the collection days will be printed in the next newsletter.\n(e) Defibrillator: Hamid reported that the defibrillator outside the shop was checked in February and that its pads expire in October; the committee will order replacements.\n\n12. Date of the next meeting\nThe next general meeting will be held in September; the date will be announced in the summer newsletter. The meeting closed at 21:10.","questions":{"fee_motion":{"type":"choice","instructions":"What happened to the motion on the annual fee?","criteria":{"approved":null,"rejected":null,"deferred":"Postponed to a later meeting or a working group","withdrawn":"Withdrawn before the vote"}},"quorate":{"type":"noul","instructions":"Was the meeting quorate?"},"treasurer":{"type":"choice","instructions":"Who was elected treasurer?","criteria":{"odette":"Odette","lewis":"Lewis"}},"racks_support":{"type":"score","instructions":"How strong was the support for the bicycle racks motion?","criteria":["opposed by most","split","supported by most","unanimous"]}}},"media":null,"gold":{"fee_motion":"rejected","quorate":"true","treasurer":"odette","racks_support":"2"},"note":"long state: meeting minutes","provenance":{"from":"the Kev port's own records","status":"published in conversion/kev/own_records_public.json","request_sha256":"963f55cb5893a0d1e642ca0355480a2cd9259bc8c0faa8e521eaa5bb143cbd0c"}},{"id":"long_3400","source":"own_long_extended","public":true,"request":{"state":"BRINDLECOURT LOGISTICS - parcel hub 2, cross-belt sorter S1, controller log (excerpt)\nshift: early (06:00-14:00)   supervisor on duty: Yusuf   maintenance on call: Malik\n\n06:00:00 INFO  throughput 1712 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n06:03:00 INFO  throughput 1698 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n06:06:00 INFO  throughput 1731 parcels/h; lanes 1-12 active; scanner read rate 99.5%\n06:09:00 INFO  throughput 1704 parcels/h; lanes 1-12 active; scanner read rate 99.7%\n06:09:41 INFO  diverter D4 calibration check passed (offset 0.39 mm)\n06:12:00 INFO  throughput 1689 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n06:12:17 WARN  scanner T2 no-read on parcel 48367; parcel sent to manual lane\n06:15:00 INFO  throughput 1720 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n06:18:00 INFO  throughput 1716 parcels/h; lanes 1-12 active; scanner read rate 99.5%\n06:21:00 INFO  throughput 1702 parcels/h; lanes 1-12 active; scanner read rate 99.7%\n06:24:00 INFO  throughput 1695 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n06:27:00 INFO  throughput 1727 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n06:27:41 INFO  diverter D4 calibration check passed (offset 0.57 mm)\n06:30:00 INFO  throughput 1709 parcels/h; lanes 1-12 active; scanner read rate 99.5%\n06:33:00 INFO  throughput 1683 parcels/h; lanes 1-12 active; scanner read rate 99.7%\n06:33:17 WARN  scanner T2 no-read on parcel 48640; parcel sent to manual lane\n06:36:00 INFO  throughput 1718 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n06:39:00 INFO  throughput 1700 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n06:40:00 INFO  throughput 1694 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n06:41:12 WARN  lane 7 drive motor current 8.9 A (limit 8.0 A) for 5 s\n06:41:58 WARN  lane 7 drive motor current 9.4 A (limit 8.0 A) for 12 s\n06:42:40 WARN  lane 7 belt speed 1.62 m/s (setpoint 1.80 m/s)\n06:43:05 ERROR lane 7 jam detected: photo-eye PE7-3 blocked for more than 3 s\n06:43:06 ERROR sorter S1 stopped by safety controller (cause: jam lane 7); all lanes halted\n06:43:06 INFO  infeed conveyors paused; parcels held on accumulation belts\n06:43:40 INFO  alarm acknowledged by Yusuf at the control desk\n06:44:15 INFO  maintenance call raised to Malik (priority 1)\n06:46:02 INFO  scanner T1 and T2 idle (no parcels); self-test passed\n06:47:30 INFO  accumulation belts at 64% capacity\n06:50:12 INFO  Malik on site at lane 7; lock-out applied to lane 7 drive\n06:52:48 INFO  accumulation belts at 81% capacity; infeed doors 3 and 4 closed to new trailers\n06:55:20 INFO  jam cause reported by Malik: crushed carton wedged under the lane 7 belt at the second roller\n06:58:41 INFO  carton removed from lane 7; roller R7-2 inspected, no damage found\n07:00:05 INFO  lane 7 belt tension measured below specification; adjustment started\n07:03:12 INFO  lane 7 belt tension adjusted; test run at low speed 0.5 m/s for 60 s passed\n07:04:25 INFO  lock-out removed from lane 7 drive\n07:05:30 INFO  sorter S1 restarted by Yusuf; stoppage 22 min 24 s\n07:05:31 INFO  infeed conveyors resumed; infeed doors 3 and 4 reopened\n07:06:00 INFO  lane 7 drive motor current 6.1 A (limit 8.0 A)\n07:09:00 INFO  throughput 1840 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.0%\n07:12:00 INFO  throughput 1866 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.3%\n07:15:00 INFO  throughput 1851 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.6%\n07:15:44 WARN  scanner T1 no-read on parcel 49870; parcel sent to manual lane\n07:18:00 INFO  throughput 1792 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.0%\n07:21:00 INFO  throughput 1748 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.3%\n07:21:09 INFO  accumulation belts back below 10% capacity\n07:24:00 INFO  throughput 1731 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.6%\n07:27:00 INFO  throughput 1722 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.0%\n07:27:30 INFO  lane 7 drive motor current 5.9 A (limit 8.0 A); belt speed 1.80 m/s\n07:30:00 INFO  throughput 1715 parcels/h (backlog clearing); lanes 1-12 active; scanner read rate 99.3%\n07:33:00 INFO  shift note by Yusuf: lane 7 belt tension to be checked again at 10:00; carton supplier to be told about weak boxes\n07:33:05 INFO  sorter S1 running normally; no active alarms\n07:36:00 INFO  throughput 1708 parcels/h; lanes 1-12 active; scanner read rate 99.0%\n07:40:00 INFO  throughput 1716 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n07:44:00 INFO  throughput 1699 parcels/h; lanes 1-12 active; scanner read rate 99.2%\n07:44:41 INFO  diverter D4 calibration check passed (offset 0.43 mm)\n07:48:00 INFO  throughput 1722 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n07:52:00 INFO  throughput 1704 parcels/h; lanes 1-12 active; scanner read rate 99.4%\n07:52:17 WARN  scanner T1 no-read on parcel 50113; parcel sent to manual lane\n07:56:00 INFO  throughput 1711 parcels/h; lanes 1-12 active; scanner read rate 99.5%\n08:00:00 INFO  throughput 1693 parcels/h; lanes 1-12 active; scanner read rate 99.6%\n08:00:30 INFO  lane 7 drive motor current 5.8 A (limit 8.0 A); belt speed 1.80 m/s\n08:04:00 INFO  throughput 1719 parcels/h; lanes 1-12 active; scanner read rate 99.7%\n08:08:00 INFO  throughput 1702 parcels/h; lanes 1-12 active; scanner read rate 99.0%\n08:08:41 INFO  diverter D4 calibration check passed (offset 0.49 mm)\n08:12:00 INFO  throughput 1714 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n08:12:50 INFO  accumulation belts at 8% capacity\n08:16:00 INFO  throughput 1697 parcels/h; lanes 1-12 active; scanner read rate 99.2%\n08:20:00 INFO  throughput 1725 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n08:20:17 WARN  scanner T2 no-read on parcel 50510; parcel sent to manual lane\n08:24:00 INFO  throughput 1706 parcels/h; lanes 1-12 active; scanner read rate 99.4%\n08:28:00 INFO  throughput 1690 parcels/h; lanes 1-12 active; scanner read rate 99.5%\n08:32:00 INFO  throughput 1718 parcels/h; lanes 1-12 active; scanner read rate 99.6%\n08:32:41 INFO  diverter D4 calibration check passed (offset 0.55 mm)\n08:36:00 INFO  throughput 1701 parcels/h; lanes 1-12 active; scanner read rate 99.7%\n08:40:00 INFO  throughput 1708 parcels/h; lanes 1-12 active; scanner read rate 99.0%\n08:40:30 INFO  lane 7 drive motor current 5.9 A (limit 8.0 A); belt speed 1.80 m/s\n08:44:00 INFO  throughput 1716 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n08:48:00 INFO  throughput 1699 parcels/h; lanes 1-12 active; scanner read rate 99.2%\n08:48:17 WARN  scanner T1 no-read on parcel 50907; parcel sent to manual lane\n08:52:00 INFO  throughput 1722 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n08:56:00 INFO  throughput 1704 parcels/h; lanes 1-12 active; scanner read rate 99.4%\n08:56:41 INFO  diverter D4 calibration check passed (offset 0.31 mm)\n09:00:00 INFO  throughput 1711 parcels/h; lanes 1-12 active; scanner read rate 99.5%\n09:00:50 INFO  accumulation belts at 5% capacity\n09:04:00 INFO  throughput 1693 parcels/h; lanes 1-12 active; scanner read rate 99.6%\n09:08:00 INFO  throughput 1719 parcels/h; lanes 1-12 active; scanner read rate 99.7%\n09:12:00 INFO  throughput 1702 parcels/h; lanes 1-12 active; scanner read rate 99.0%\n09:16:00 INFO  throughput 1714 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n09:16:17 WARN  scanner T2 no-read on parcel 51304; parcel sent to manual lane\n09:20:00 INFO  throughput 1697 parcels/h; lanes 1-12 active; scanner read rate 99.2%\n09:20:41 INFO  diverter D4 calibration check passed (offset 0.37 mm)\n09:20:30 INFO  lane 7 drive motor current 6.0 A (limit 8.0 A); belt speed 1.80 m/s\n09:24:00 INFO  throughput 1725 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n09:28:00 INFO  throughput 1706 parcels/h; lanes 1-12 active; scanner read rate 99.4%\n09:32:00 INFO  throughput 1690 parcels/h; lanes 1-12 active; scanner read rate 99.5%\n09:36:00 INFO  throughput 1718 parcels/h; lanes 1-12 active; scanner read rate 99.6%\n09:40:00 INFO  throughput 1701 parcels/h; lanes 1-12 active; scanner read rate 99.7%\n09:44:00 INFO  throughput 1708 parcels/h; lanes 1-12 active; scanner read rate 99.0%\n09:44:41 INFO  diverter D4 calibration check passed (offset 0.43 mm)\n09:44:17 WARN  scanner T1 no-read on parcel 51701; parcel sent to manual lane\n09:48:00 INFO  throughput 1716 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n09:48:50 INFO  accumulation belts at 7% capacity\n09:52:00 INFO  throughput 1699 parcels/h; lanes 1-12 active; scanner read rate 99.2%\n09:56:00 INFO  throughput 1722 parcels/h; lanes 1-12 active; scanner read rate 99.3%\n10:00:00 INFO  throughput 1704 parcels/h; lanes 1-12 active; scanner read rate 99.4%\n10:00:30 INFO  lane 7 drive motor current 5.8 A (limit 8.0 A); belt speed 1.80 m/s\n10:00:40 INFO  lane 7 belt tension checked by Malik (as planned in the 07:33 shift note): within specification; no adjustment needed\n10:04:00 INFO  throughput 1711 parcels/h; lanes 1-12 active; scanner read rate 99.5%\n10:08:00 INFO  throughput 1693 parcels/h; lanes 1-12 active; scanner read rate 99.6%\n10:08:41 INFO  diverter D4 calibration check passed (offset 0.49 mm)\n10:12:00 INFO  throughput 1719 parcels/h; lanes 1-12 active; scanner read rate 99.7%\n10:12:17 WARN  scanner T2 no-read on parcel 52098; parcel sent to manual lane\n10:16:00 INFO  throughput 1702 parcels/h; lanes 1-12 active; scanner read rate 99.0%\n10:20:00 INFO  throughput 1714 parcels/h; lanes 1-12 active; scanner read rate 99.1%\n10:21:05 INFO  sorter S1 running normally; no active alarms","questions":{"component":{"type":"choice","instructions":"Which component caused the stoppage?","criteria":{"scanner":"Barcode scanner","lane_7":"Conveyor lane 7","diverter":"Diverter arm","power":"Power supply","software":"Control software"}},"normal_at_end":{"type":"noul","instructions":"Was the sorter running normally at the end of the log?"},"stoppage":{"type":"score","instructions":"How long was the sorter fully stopped?","criteria":["under 2 minutes","2 to 5 minutes","5 to 10 minutes","10 to 20 minutes","20 to 40 minutes","over 40 minutes"]},"tension_10":{"type":"noul","instructions":"At the 10:00 check, was the lane 7 belt tension within specification?"}}},"media":null,"gold":{"component":"lane_7","normal_at_end":"true","stoppage":"4","tension_10":"true"},"note":"own_L01's sorter log continued to ~3.4k state tokens (same names, places and equipment); own_L01's questions keep their answers (no new stoppage, the log ends normally); tension_10 is answered in the added lines","provenance":{"from":"own_L01","own_L01_state_sha256":"c2b5ddc001cd500fef76a3f31b060efc87094797e8bfb9e98b37b6948e716758","added_lines":64,"until":"10:20","state_tokens":3410}},{"id":"img_01","source":"own_image","public":true,"request":{"state":null,"questions":{"room":{"type":"choice","instructions":"What kind of room is this?","criteria":{"kitchen":"A kitchen","bedroom":"A bedroom","office":"An office","bathroom":"A bathroom"}},"suitcase":{"type":"noul","instructions":"Is there a suitcase in the room?"},"brightness":{"type":"score","instructions":"How bright is the room?","criteria":["Dark","Dim","Bright"]}}},"media":{"images":["images/img_01.png"]},"gold":{"room":"bedroom","suitcase":"true","brightness":"2"},"note":"a made double bed seen from a doorway in daylight, folded towels, one bedside lamp on, a closed suitcase on the floor (own FLUX.2 klein output; gold = the answer seen in the picture)","provenance":{"image":{"file":"images/img_01.png","sha256":"ae59d05efc3d6990f1a02b728508cd786d140bdccf666f9a8a11c53d1e2191b9","bytes":167859,"px":[384,384],"license":"own model output: FLUX.2 klein 4B (Apache-2.0) on Core AI, no third-party image involved","pool":"flux","from":"room3_05.png","op":"resize 1024x1024 -> 384x384","manifest":"images/manifest.json"},"image_info":{"px":[384,384],"mode":"RGB","prefix_positions":144}}},{"id":"img_02","source":"own_image","public":true,"request":{"state":"Photo from a room check.","questions":{"room":{"type":"choice","instructions":"What kind of room is this?","criteria":{"kitchen":"A kitchen","bedroom":"A bedroom","office":"An office","bathroom":"A bathroom"}},"lamp":{"type":"noul","instructions":"Is a lamp switched on?"},"brightness":{"type":"score","instructions":"How bright is the room?","criteria":["Dark","Dim","Bright"]}}},"media":{"images":["images/img_02.png"]},"gold":{"room":"bedroom","lamp":"true","brightness":"1"},"note":"a hotel bedroom at night lit by two switched-on table lamps; a made bed, an armchair, a desk phone (keypad and clock face not legible) (CC0 photograph; gold = the answer seen in the picture)","provenance":{"image":{"file":"images/img_02.png","sha256":"1864e1c215f23774abbd74860272c5f8e9ce439e75857b48b4cabaabf09195b7","bytes":146091,"px":[384,288],"license":"CC0 1.0 (Wikimedia Commons licence template in the cc-zero family; no attribution required)","pool":"cc0","from":"real_14.png","op":"resize 1024x768 -> 384x288","manifest":"images/manifest.json"},"image_info":{"px":[384,288],"mode":"RGB","prefix_positions":108}}},{"id":"img_03","source":"own_image","public":true,"request":{"state":null,"questions":{"beds":{"type":"choice","instructions":"How many beds are in the room?","criteria":{"one":"One bed","two":"Two beds","more":"Three or more beds"}},"towel":{"type":"noul","instructions":"Is there a folded towel on a bed?"},"tidy":{"type":"score","instructions":"How tidy is the room?","criteria":["Very cluttered","Some clutter","Tidy"]}}},"media":{"images":["images/img_03.png"]},"gold":{"beds":"two","towel":"true","tidy":"2"},"note":"two made single beds side by side, a folded towel on the nearer one, a desk lamp (off) and a wooden lattice on the wall; portrait 768 x 1024 (tiled) (CC0 photograph; gold = the answer seen in the picture)","provenance":{"image":{"file":"images/img_03.png","sha256":"28080dd775920ac0b701d6640eb08e0500542946a19c84ab80e263e595224312","bytes":566869,"px":[768,1024],"license":"CC0 1.0 (Wikimedia Commons licence template in the cc-zero family; no attribution required)","pool":"cc0","from":"real_18.png","op":"copy (the source file, byte for byte)","manifest":"images/manifest.json"},"image_info":{"px":[768,1024],"mode":"RGB","prefix_positions":1770}}},{"id":"aud_01","source":"own_audio","public":true,"request":{"state":"Voice note from a user.","questions":{"topic":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"wants":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"urgency":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]}}},"media":{"audio":"audio/aud_01.wav"},"gold":{"topic":"delivery","wants":"true","urgency":"1"},"note":"synthesised: Kokoro-82M voice am_michael","provenance":{"script":"Hi, the blender I ordered last week arrived with a cracked jug. Could you send me a replacement before Friday? I need it for the weekend.","voice":"am_michael","synth":{"model":"hexgrad/Kokoro-82M","package":"kokoro 0.9.4 + misaki","lang_code":"a","speed":1.0,"seed":0,"native_rate":24000,"rate":16000,"resample":"ffmpeg -ar 16000 -ac 1 -sample_fmt s16","seconds":[6.0,12.0]},"file_present":true,"clip":{"file":"audio/aud_01.wav","voice":"am_michael","seed":0,"text":"Hi, the blender I ordered last week arrived with a cracked jug. Could you send me a replacement before Friday? I need it for the weekend.","phonemes":["hˈI, ðə blˈɛndəɹ ˌI ˈɔɹdəɹd lˈæst wˈik əɹˈIvd wɪð ɐ kɹˈækt ʤˈʌɡ. kˌʊd ju sˈɛnd mˌi ɐ ɹəplˈAsmᵊnt bəfˈɔɹ fɹˈIdˌA? ˌI nˈid ɪt fɔɹ ðə wˈikˌɛnd."],"native_seconds":9.625,"seconds":9.625,"sample_rate":16000,"channels":1,"sample_width_bytes":2,"frames":154000,"in_range_6_12_s":true,"sha256":"ad576e24573dce521c9104d0acb05697395811743b17f0c58882c17d9a27feb4","synth_s":0.72}}},{"id":"aud_02","source":"own_audio","public":true,"request":{"state":"Voice note from a user.","questions":{"topic":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"wants":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"urgency":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]}}},"media":{"audio":"audio/aud_02.wav"},"gold":{"topic":"feedback","wants":"false","urgency":"0"},"note":"synthesised: Kokoro-82M voice bf_emma","provenance":{"script":"I just wanted to say that the new update works really well. The app opens much faster now, and I have not had a single problem this week. Thanks.","voice":"bf_emma","synth":{"model":"hexgrad/Kokoro-82M","package":"kokoro 0.9.4 + misaki","lang_code":"a","speed":1.0,"seed":0,"native_rate":24000,"rate":16000,"resample":"ffmpeg -ar 16000 -ac 1 -sample_fmt s16","seconds":[6.0,12.0]},"file_present":true,"clip":{"file":"audio/aud_02.wav","voice":"bf_emma","seed":0,"text":"I just wanted to say that the new update works really well. The app opens much faster now, and I have not had a single problem this week. Thanks.","phonemes":["ˌI ʤˈʌst wˈɑntᵻd tə sˈA ðæt ðə nˈu ˈʌpdˌAt wˈɜɹks ɹˈiᵊli wˈɛl. ði ˈæp ˈOpᵊnz mˈʌʧ fˈæstəɹ nˈW, ænd ˌI hæv nˌɑt hæd ɐ sˈɪŋɡᵊl pɹˈɑbləm ðɪs wˈik. θˈæŋks."],"native_seconds":8.775,"seconds":8.775,"sample_rate":16000,"channels":1,"sample_width_bytes":2,"frames":140400,"in_range_6_12_s":true,"sha256":"d3128b99481a46e3631fb7fc396c54551a46398da271ed48f0a36c5217467563","synth_s":0.57}}},{"id":"aud_03","source":"own_audio","public":true,"request":{"state":null,"questions":{"topic":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"wants":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"urgency":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]}}},"media":{"audio":"audio/aud_03.wav"},"gold":{"topic":"account","wants":"true","urgency":"2"},"note":"synthesised: Kokoro-82M voice af_heart","provenance":{"script":"I can't log in to my account, and I have a payment due in an hour. Please reset my password right away. The reset email never arrived.","voice":"af_heart","synth":{"model":"hexgrad/Kokoro-82M","package":"kokoro 0.9.4 + misaki","lang_code":"a","speed":1.0,"seed":0,"native_rate":24000,"rate":16000,"resample":"ffmpeg -ar 16000 -ac 1 -sample_fmt s16","seconds":[6.0,12.0]},"file_present":true,"clip":{"file":"audio/aud_03.wav","voice":"af_heart","seed":0,"text":"I can't log in to my account, and I have a payment due in an hour. Please reset my password right away. The reset email never arrived.","phonemes":["ˌI kˈænt lˈɔɡ ˈɪn tə mI əkˈWnt, ænd ˌI hæv ɐ pˈAmᵊnt dˈu ɪn ɐn ˈWəɹ. plˈiz ɹisˈɛt mI pˈæswˌɜɹd ɹˈIt əwˈA. ðə ɹisˈɛt ˈimˌAl nˈɛvəɹ əɹˈIvd."],"native_seconds":8.75,"seconds":8.75,"sample_rate":16000,"channels":1,"sample_width_bytes":2,"frames":140000,"in_range_6_12_s":true,"sha256":"c78244d4ae793e00dc0d82a609f72d14bdd1fe93705e9a99483a9bf3dadf4c6d","synth_s":0.58}}},{"id":"aud_04","source":"own_audio","public":true,"request":{"state":"Voice note from a user.","questions":{"topic":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"wants":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"urgency":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]}}},"media":{"audio":"audio/aud_04.wav"},"gold":{"topic":"account","wants":"true","urgency":"2"},"note":"synthesised: Kokoro-82M voice af_bella, speed 1.0, 2.8 s","provenance":{"script":"Please reset my password right away.","voice":"af_bella","synth":{"model":"hexgrad/Kokoro-82M","package":"kokoro 0.9.4 + misaki","lang_code":"a","seed":0,"native_rate":24000,"rate":16000,"resample":"ffmpeg -ar 16000 -ac 1 -sample_fmt s16","round":6,"speed":"per clip","seconds":"per clip (target_seconds)","trim":{"threshold_abs_int16":328,"margin_samples":320}},"speed":1.0,"target_seconds":[2.5,3.5],"trim":false,"file_present":true,"clip":{"file":"audio/aud_04.wav","voice":"af_bella","seed":0,"speed":1.0,"text":"Please reset my password right away.","phonemes":["plˈiz ɹisˈɛt mI pˈæswˌɜɹd ɹˈIt əwˈA."],"native_seconds":2.8,"seconds":2.8,"sample_rate":16000,"channels":1,"sample_width_bytes":2,"frames":44800,"target_seconds":[2.5,3.5],"in_target":true,"trim":null,"deterministic_two_runs":true,"round":6,"sha256":"efbf3b4287c1ecbfa13b7fc9ab1a52e2a7a5d7b09ad2f8319f0a9bbba11b92b3","synth_s":0.64}}},{"id":"aud_05","source":"own_audio","public":true,"request":{"state":null,"questions":{"topic":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"wants":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"urgency":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]}}},"media":{"audio":"audio/aud_05.wav"},"gold":{"topic":"billing","wants":"true","urgency":"0"},"note":"synthesised: Kokoro-82M voice am_fenrir, speed 1.0, 2.875 s","provenance":{"script":"No rush, but please update my billing address.","voice":"am_fenrir","synth":{"model":"hexgrad/Kokoro-82M","package":"kokoro 0.9.4 + misaki","lang_code":"a","seed":0,"native_rate":24000,"rate":16000,"resample":"ffmpeg -ar 16000 -ac 1 -sample_fmt s16","round":6,"speed":"per clip","seconds":"per clip (target_seconds)","trim":{"threshold_abs_int16":328,"margin_samples":320}},"speed":1.0,"target_seconds":[2.5,3.5],"trim":false,"file_present":true,"clip":{"file":"audio/aud_05.wav","voice":"am_fenrir","seed":0,"speed":1.0,"text":"No rush, but please update my billing address.","phonemes":["nˈO ɹˈʌʃ, bˌʌt plˈiz ˈʌpdˌAt mI bˈɪlɪŋ ədɹˈɛs."],"native_seconds":2.875,"seconds":2.875,"sample_rate":16000,"channels":1,"sample_width_bytes":2,"frames":46000,"target_seconds":[2.5,3.5],"in_target":true,"trim":null,"deterministic_two_runs":true,"round":6,"sha256":"98d200490be9134d28d81f257bdf9e8d99520bb00ac20d68d26e748b0dd28c6c","synth_s":0.58}}},{"id":"aud_06","source":"own_audio","public":true,"request":{"state":"Voice note from a user.","questions":{"topic":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"wants":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"urgency":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]}}},"media":{"audio":"audio/aud_06.wav"},"gold":{"topic":"delivery","wants":"true","urgency":"1"},"note":"synthesised: Kokoro-82M voice bf_emma, speed 1.0, 8.05 s","provenance":{"script":"Hello, my parcel was marked as delivered yesterday, but it is not at my door or with my neighbours. Could you look into it this week?","voice":"bf_emma","synth":{"model":"hexgrad/Kokoro-82M","package":"kokoro 0.9.4 + misaki","lang_code":"a","seed":0,"native_rate":24000,"rate":16000,"resample":"ffmpeg -ar 16000 -ac 1 -sample_fmt s16","round":6,"speed":"per clip","seconds":"per clip (target_seconds)","trim":{"threshold_abs_int16":328,"margin_samples":320}},"speed":1.0,"target_seconds":[7.0,9.0],"trim":false,"file_present":true,"clip":{"file":"audio/aud_06.wav","voice":"bf_emma","seed":0,"speed":1.0,"text":"Hello, my parcel was marked as delivered yesterday, but it is not at my door or with my neighbours. Could you look into it this week?","phonemes":["həlˈO, mI pˈɑɹsᵊl wʌz mˈɑɹkt æz dəlˈɪvəɹd jˈɛstəɹdˌA, bˌʌt ɪt ɪz nˌɑt æt mI dˈɔɹ ɔɹ wɪð mI nˈAbəɹz. kˌʊd ju lˈʊk ˈɪntu ɪt ðɪs wˈik?"],"native_seconds":8.05,"seconds":8.05,"sample_rate":16000,"channels":1,"sample_width_bytes":2,"frames":128800,"target_seconds":[7.0,9.0],"in_target":true,"trim":null,"deterministic_two_runs":true,"round":6,"sha256":"b5b2522295c3cf334ef97fde4c44678ae8017958a2f4395cab0cf8a8ce7f964c","synth_s":1.24}}},{"id":"aud_07","source":"own_audio","public":true,"request":{"state":null,"questions":{"topic":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"wants":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"urgency":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]}}},"media":{"audio":"audio/aud_07.wav"},"gold":{"topic":"feedback","wants":"false","urgency":"0"},"note":"synthesised: Kokoro-82M voice af_heart, speed 1.0, 8.45 s","provenance":{"script":"I just want to say that the support team was very kind on the phone today. Everything is sorted now, and I am very happy. Thank you.","voice":"af_heart","synth":{"model":"hexgrad/Kokoro-82M","package":"kokoro 0.9.4 + misaki","lang_code":"a","seed":0,"native_rate":24000,"rate":16000,"resample":"ffmpeg -ar 16000 -ac 1 -sample_fmt s16","round":6,"speed":"per clip","seconds":"per clip (target_seconds)","trim":{"threshold_abs_int16":328,"margin_samples":320}},"speed":1.0,"target_seconds":[7.0,9.0],"trim":false,"file_present":true,"clip":{"file":"audio/aud_07.wav","voice":"af_heart","seed":0,"speed":1.0,"text":"I just want to say that the support team was very kind on the phone today. Everything is sorted now, and I am very happy. Thank you.","phonemes":["ˌI ʤˈʌst wˈɑnt tə sˈA ðæt ðə səpˈɔɹt tˈim wʌz vˈɛɹi kˈInd ˌɔn ðə fˈOn tədˈA. ˈɛvəɹiθˌɪŋ ɪz sˈɔɹTᵻd nˈW, ænd ˌI ɐm vˈɛɹi hˈæpi. θˈæŋk ju."],"native_seconds":8.45,"seconds":8.45,"sample_rate":16000,"channels":1,"sample_width_bytes":2,"frames":135200,"target_seconds":[7.0,9.0],"in_target":true,"trim":null,"deterministic_two_runs":true,"round":6,"sha256":"cc8e1ed29d98c65e2420106ef620f0c84ef94db37bc410d3fb341128f2dfe538","synth_s":1.27}}},{"id":"aud_08","source":"own_audio","public":true,"request":{"state":"Voice note from a user.","questions":{"topic":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"wants":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"urgency":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]}}},"media":{"audio":"audio/aud_08.wav"},"gold":{"topic":"billing","wants":"true","urgency":"1"},"note":"synthesised: Kokoro-82M voice am_michael, speed 1.0, 15.875 s","provenance":{"script":"Hi there. I looked at my statement this morning, and I was charged twice for the same monthly plan. The second payment went out on the third. I would like the extra charge refunded, ideally before my next statement at the end of the month.","voice":"am_michael","synth":{"model":"hexgrad/Kokoro-82M","package":"kokoro 0.9.4 + misaki","lang_code":"a","seed":0,"native_rate":24000,"rate":16000,"resample":"ffmpeg -ar 16000 -ac 1 -sample_fmt s16","round":6,"speed":"per clip","seconds":"per clip (target_seconds)","trim":{"threshold_abs_int16":328,"margin_samples":320}},"speed":1.0,"target_seconds":[13.5,16.5],"trim":false,"file_present":true,"clip":{"file":"audio/aud_08.wav","voice":"am_michael","seed":0,"speed":1.0,"text":"Hi there. I looked at my statement this morning, and I was charged twice for the same monthly plan. The second payment went out on the third. I would like the extra charge refunded, ideally before my next statement at the end of the month.","phonemes":["hˈI ðˈɛɹ. ˌI lˈʊkt æt mI stˈAtmənt ðɪs mˈɔɹnɪŋ, ænd ˌI wʌz ʧˈɑɹʤd twˈIs fɔɹ ðə sˈAm mˈʌnθli plˈæn. ðə sˈɛkənd pˈAmᵊnt wˈɛnt ˈWt ˌɔn ðə θˈɜɹd. ˌI wʊd lˈIk ði ˈɛkstɹə ʧˈɑɹʤ ɹəfˈʌndᵻd, Idˈiᵊli bəfˈɔɹ mI nˈɛkst stˈAtmənt æt ði ˈɛnd ʌv ðə mˈʌnθ."],"native_seconds":15.875,"seconds":15.875,"sample_rate":16000,"channels":1,"sample_width_bytes":2,"frames":254000,"target_seconds":[13.5,16.5],"in_target":true,"trim":null,"deterministic_two_runs":true,"round":6,"sha256":"3a06935972640ca5f9d2758c35b5de504e2589734d805089f77d3b7742b95bd6","synth_s":2.27}}},{"id":"aud_09","source":"own_audio","public":true,"request":{"state":null,"questions":{"topic":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"wants":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"urgency":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]}}},"media":{"audio":"audio/aud_09.wav"},"gold":{"topic":"account","wants":"true","urgency":"2"},"note":"synthesised: Kokoro-82M voice af_bella, speed 1.0, 13.7 s","provenance":{"script":"Someone changed the email address on my account last night, and it was not me. I have already changed my password, but I am still worried. Please lock the account for now, and call me back as soon as you possibly can.","voice":"af_bella","synth":{"model":"hexgrad/Kokoro-82M","package":"kokoro 0.9.4 + misaki","lang_code":"a","seed":0,"native_rate":24000,"rate":16000,"resample":"ffmpeg -ar 16000 -ac 1 -sample_fmt s16","round":6,"speed":"per clip","seconds":"per clip (target_seconds)","trim":{"threshold_abs_int16":328,"margin_samples":320}},"speed":1.0,"target_seconds":[13.5,16.5],"trim":false,"file_present":true,"clip":{"file":"audio/aud_09.wav","voice":"af_bella","seed":0,"speed":1.0,"text":"Someone changed the email address on my account last night, and it was not me. I have already changed my password, but I am still worried. Please lock the account for now, and call me back as soon as you possibly can.","phonemes":["sˈʌmwˌʌn ʧˈAnʤd ði ˈimˌAl ədɹˈɛs ˌɔn mI əkˈWnt lˈæst nˈIt, ænd ɪt wʌz nˌɑt mˌi. ˌI hæv ˌɔlɹˈɛdi ʧˈAnʤd mI pˈæswˌɜɹd, bˌʌt ˌI ɐm stˈɪl wˈɜɹid. plˈiz lˈɑk ði əkˈWnt fɔɹ nˈW, ænd kˈɔl mˌi bˈæk æz sˈun æz ju pˈɑsəbli kˈæn."],"native_seconds":13.7,"seconds":13.7,"sample_rate":16000,"channels":1,"sample_width_bytes":2,"frames":219200,"target_seconds":[13.5,16.5],"in_target":true,"trim":null,"deterministic_two_runs":true,"round":6,"sha256":"6a9e400bd5dc10207c0482f22ad275ffcc23804e2fd765470c9604aaad0f2428","synth_s":1.82}}},{"id":"aud_10","source":"own_audio","public":true,"request":{"state":"Voice note from a user.","questions":{"topic":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"wants":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"urgency":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]}}},"media":{"audio":"audio/aud_10.wav"},"gold":{"topic":"feedback","wants":"false","urgency":"0"},"note":"synthesised: Kokoro-82M voice bf_emma, speed 1.0, 21.475 s","provenance":{"script":"I wanted to leave some feedback about the new delivery tracking page. It is much clearer than the old one. I can see where my order is on the map, and the time window has been right every time so far. The messages are short and friendly, and I no longer have to call to ask where things are. My neighbours have started using it too, and they all say the same thing. Keep up the good work, and thank you.","voice":"bf_emma","synth":{"model":"hexgrad/Kokoro-82M","package":"kokoro 0.9.4 + misaki","lang_code":"a","seed":0,"native_rate":24000,"rate":16000,"resample":"ffmpeg -ar 16000 -ac 1 -sample_fmt s16","round":6,"speed":"per clip","seconds":"per clip (target_seconds)","trim":{"threshold_abs_int16":328,"margin_samples":320}},"speed":1.0,"target_seconds":[20.5,23.5],"trim":false,"file_present":true,"clip":{"file":"audio/aud_10.wav","voice":"bf_emma","seed":0,"speed":1.0,"text":"I wanted to leave some feedback about the new delivery tracking page. It is much clearer than the old one. I can see where my order is on the map, and the time window has been right every time so far. The messages are short and friendly, and I no longer have to call to ask where things are. My neighbours have started using it too, and they all say the same thing. Keep up the good work, and thank you.","phonemes":["ˌI wˈɑntᵻd tə lˈiv sˌʌm fˈidbˌæk əbˈWt ðə nˈu dəlˈɪvəɹi tɹˈækɪŋ pˈAʤ. ˌɪt ɪz mˈʌʧ klˈɪɹəɹ ðən ði ˈOld wˈʌn. ˌI kæn sˈi wˌɛɹ mI ˈɔɹdəɹ ɪz ˌɔn ðə mˈæp, ænd ðə tˈIm wˈɪndO hæz bɪn ɹˈIt ˈɛvəɹi tˈIm sˌO fˈɑɹ. ðə mˈɛsɪʤᵻz ɑɹ ʃˈɔɹt ænd fɹˈɛndli, ænd ˌI nˈO lˈɔŋɡəɹ hæv tə kˈɔl tʊ ˈæsk wˌɛɹ θˈɪŋz ɑɹ. mˌI nˈAbəɹz hæv stˈɑɹTᵻd jˈuzɪŋ ɪt tˈu, ænd ðA ˈɔl sˈA ðə sˈAm θˈɪŋ. kˈip ˌʌp ðə ɡˈʊd wˈɜɹk, ænd θˈæŋk ju."],"native_seconds":21.475,"seconds":21.475,"sample_rate":16000,"channels":1,"sample_width_bytes":2,"frames":343600,"target_seconds":[20.5,23.5],"in_target":true,"trim":null,"deterministic_two_runs":true,"round":6,"sha256":"e5c5dd555900c9cf5c78bac7cc0a9ab7dd0592aa6425d9af526d873940c73e6a","synth_s":2.68}}},{"id":"aud_11","source":"own_audio","public":true,"request":{"state":null,"questions":{"topic":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"wants":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"urgency":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]}}},"media":{"audio":"audio/aud_11.wav"},"gold":{"topic":"delivery","wants":"true","urgency":"1"},"note":"synthesised: Kokoro-82M voice am_fenrir, speed 1.0, 20.975 s","provenance":{"script":"Good morning. I ordered a set of kitchen chairs two weeks ago, and the delivery date has moved three times now. Each time I took a morning off work to wait for the van, and each time nobody came, and nobody called to tell me. I am having family over next Saturday, and I really need the chairs by then. Could you tell me the real delivery date and, if you can, put them on the earliest van this week? A short text message with the date would be perfect. Thank you very much for your help.","voice":"am_fenrir","synth":{"model":"hexgrad/Kokoro-82M","package":"kokoro 0.9.4 + misaki","lang_code":"a","seed":0,"native_rate":24000,"rate":16000,"resample":"ffmpeg -ar 16000 -ac 1 -sample_fmt s16","round":6,"speed":"per clip","seconds":"per clip (target_seconds)","trim":{"threshold_abs_int16":328,"margin_samples":320}},"speed":1.0,"target_seconds":[20.5,23.5],"trim":false,"file_present":true,"clip":{"file":"audio/aud_11.wav","voice":"am_fenrir","seed":0,"speed":1.0,"text":"Good morning. I ordered a set of kitchen chairs two weeks ago, and the delivery date has moved three times now. Each time I took a morning off work to wait for the van, and each time nobody came, and nobody called to tell me. I am having family over next Saturday, and I really need the chairs by then. Could you tell me the real delivery date and, if you can, put them on the earliest van this week? A short text message with the date would be perfect. Thank you very much for your help.","phonemes":["ɡˈʊd mˈɔɹnɪŋ. ˌI ˈɔɹdəɹd ɐ sˈɛt ʌv kˈɪʧᵊn ʧˈɛɹz tˈu wˈiks əɡˈO, ænd ðə dəlˈɪvəɹi dˈAt hæz mˈuvd θɹˈi tˈImz nˈW. ˈiʧ tˈIm ˌI tˈʊk ɐ mˈɔɹnɪŋ ˈɔf wˈɜɹk tə wˈAt fɔɹ ðə vˈæn, ænd ˈiʧ tˈIm nˈObˌɑdi kˈAm, ænd nˈObˌɑdi kˈɔld tə tˈɛl mˌi. ˌI ɐm hˈævɪŋ fˈæmᵊli ˈOvəɹ nˈɛkst sˈæTəɹdˌA, ænd ˌI ɹˈiᵊli nˈid ðə ʧˈɛɹz bI ðˈɛn. kˌʊd ju tˈɛl mˌi ðə ɹˈiᵊl dəlˈɪvəɹi dˈAt ænd, ɪf ju kˈæn, pˌʊt ðˌɛm ˌɔn ði ˈɜɹliᵻst vˈæn ðɪs wˈik? ɐ ʃˈɔɹt tˈɛkst mˈɛsɪʤ wɪð ðə dˈAt wʊd bi pˈɜɹfəkt. θˈæŋk ju vˈɛɹi mˈʌʧ fɔɹ jʊɹ hˈɛlp."],"native_seconds":20.975,"seconds":20.975,"sample_rate":16000,"channels":1,"sample_width_bytes":2,"frames":335600,"target_seconds":[20.5,23.5],"in_target":true,"trim":null,"deterministic_two_runs":true,"round":6,"sha256":"6d861cce1c398b7ea1783f99ae39bfcf082f082206feade3290570920231a8bb","synth_s":2.7}}},{"id":"aud_12","source":"own_audio","public":true,"request":{"state":"Voice note from a user.","questions":{"topic":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"wants":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"urgency":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]}}},"media":{"audio":"audio/aud_12.wav"},"gold":{"topic":"billing","wants":"true","urgency":"2"},"note":"synthesised: Kokoro-82M voice af_heart, speed 0.95, 27.95 s","provenance":{"script":"Hi, I am calling about my bill. This morning I got a message saying that my card was declined and that my service will be switched off at noon today. The card works everywhere else, and I paid the last three bills on time. I have tried to update the card details in the app twice, and both times it showed an error, so I am stuck. Please take the payment by hand or keep my service on, because I work from home and I need it for a meeting today. Thank you, and please be quick, because noon is not far away.","voice":"af_heart","synth":{"model":"hexgrad/Kokoro-82M","package":"kokoro 0.9.4 + misaki","lang_code":"a","seed":0,"native_rate":24000,"rate":16000,"resample":"ffmpeg -ar 16000 -ac 1 -sample_fmt s16","round":6,"speed":"per clip","seconds":"per clip (target_seconds)","trim":{"threshold_abs_int16":328,"margin_samples":320}},"speed":0.95,"target_seconds":[27.5,29.8],"trim":false,"file_present":true,"clip":{"file":"audio/aud_12.wav","voice":"af_heart","seed":0,"speed":0.95,"text":"Hi, I am calling about my bill. This morning I got a message saying that my card was declined and that my service will be switched off at noon today. The card works everywhere else, and I paid the last three bills on time. I have tried to update the card details in the app twice, and both times it showed an error, so I am stuck. Please take the payment by hand or keep my service on, because I work from home and I need it for a meeting today. Thank you, and please be quick, because noon is not far away.","phonemes":["hˈI, ˌI ɐm kˈɔlɪŋ əbˈWt mI bˈɪl. ðˌɪs mˈɔɹnɪŋ ˌI ɡɑt ɐ mˈɛsɪʤ sˈAɪŋ ðæt mI kˈɑɹd wʌz dəklˈInd ænd ðæt mI sˈɜɹvəs wɪl bi swˈɪʧt ˈɔf æt nˈun tədˈA. ðə kˈɑɹd wˈɜɹks ˈɛvɹiwˌɛɹ ˈɛls, ænd ˌI pˈAd ðə lˈæst θɹˈi bˈɪlz ˌɔn tˈIm. ˌI hæv tɹˈId tʊ ˈʌpdˌAt ðə kˈɑɹd dətˈAlz ɪn ði ˈæp twˈIs, ænd bˈOθ tˈImz ɪt ʃˈOd ɐn ˈɛɹəɹ, sˌO ˌI ɐm stˈʌk. plˈiz tˈAk ðə pˈAmᵊnt bI hˈænd ɔɹ kˈip mI sˈɜɹvəs ˌɔn, bəkˈʌz ˌI wˈɜɹk fɹʌm hˈOm ænd ˌI nˈid ɪt fɔɹ ɐ mˈiTɪŋ tədˈA. θˈæŋk ju, ænd plˈiz bi kwˈɪk, bəkˈʌz nˈun ɪz nˌɑt fˈɑɹ əwˈA."],"native_seconds":27.95,"seconds":27.95,"sample_rate":16000,"channels":1,"sample_width_bytes":2,"frames":447200,"target_seconds":[27.5,29.8],"in_target":true,"trim":null,"deterministic_two_runs":true,"round":6,"sha256":"e012012e8c3dbcd7b9b09d6c208db22bd7e8ccdbd1597985cdc09048a1fbf040","synth_s":3.33}}},{"id":"aud_13","source":"own_audio","public":true,"request":{"state":null,"questions":{"topic":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"wants":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"urgency":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]}}},"media":{"audio":"audio/aud_13.wav"},"gold":{"topic":"account","wants":"true","urgency":"0"},"note":"synthesised: Kokoro-82M voice am_michael, speed 1.0, 28.45 s","provenance":{"script":"Hello. This is not urgent at all, so please take your time. I would like to change the name on my account to the name I use now, and I would also like to add a second phone number. I looked in the settings, but I could not find where to add a second number, and the help page did not mention it either. Whenever you have a moment this month, please tell me what I need to send you, and I will reply as soon as I can. There is no hurry, and next month would also be fine. Thanks a lot, and have a good day.","voice":"am_michael","synth":{"model":"hexgrad/Kokoro-82M","package":"kokoro 0.9.4 + misaki","lang_code":"a","seed":0,"native_rate":24000,"rate":16000,"resample":"ffmpeg -ar 16000 -ac 1 -sample_fmt s16","round":6,"speed":"per clip","seconds":"per clip (target_seconds)","trim":{"threshold_abs_int16":328,"margin_samples":320}},"speed":1.0,"target_seconds":[27.5,29.8],"trim":false,"file_present":true,"clip":{"file":"audio/aud_13.wav","voice":"am_michael","seed":0,"speed":1.0,"text":"Hello. This is not urgent at all, so please take your time. I would like to change the name on my account to the name I use now, and I would also like to add a second phone number. I looked in the settings, but I could not find where to add a second number, and the help page did not mention it either. Whenever you have a moment this month, please tell me what I need to send you, and I will reply as soon as I can. There is no hurry, and next month would also be fine. Thanks a lot, and have a good day.","phonemes":["həlˈO. ðˌɪs ɪz nˌɑt ˈɜɹʤənt æt ˈɔl, sˌO plˈiz tˈAk jʊɹ tˈIm. ˌI wʊd lˈIk tə ʧˈAnʤ ðə nˈAm ˌɔn mI əkˈWnt tə ðə nˈAm ˌI jˈuz nˈW, ænd ˌI wʊd ˈɔlsO lˈIk tʊ ˈæd ɐ sˈɛkənd fˈOn nˈʌmbəɹ. ˌI lˈʊkt ɪn ðə sˈɛTɪŋz, bˌʌt ˌI kʊd nˌɑt fˈInd wˌɛɹ tʊ ˈæd ɐ sˈɛkənd nˈʌmbəɹ, ænd ðə hˈɛlp pˈAʤ dˈɪd nˌɑt mˈɛnʧᵊn ɪt ˈiðəɹ. wənˈɛvəɹ ju hæv ɐ mˈOmᵊnt ðɪs mˈʌnθ, plˈiz tˈɛl mˌi wˌʌt ˌI nˈid tə sˈɛnd ju, ænd ˌI wɪl ɹəplˈI æz sˈun æz ˌI kˈæn. ðˌɛɹ ɪz nˈO hˈɜɹi, ænd nˈɛkst mˈʌnθ wʊd ˈɔlsO bi fˈIn. θˈæŋks ɐ lˈɑt, ænd hæv ɐ ɡˈʊd dˈA."],"native_seconds":28.45,"seconds":28.45,"sample_rate":16000,"channels":1,"sample_width_bytes":2,"frames":455200,"target_seconds":[27.5,29.8],"in_target":true,"trim":null,"deterministic_two_runs":true,"round":6,"sha256":"77e9d3c6bd3a7ed47ae819a8c8632c03da017312dd366a0832ec8f2bfb32926f","synth_s":3.42}}},{"id":"aud_14","source":"own_audio","public":true,"request":{"state":null,"questions":{"topic":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"wants":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"urgency":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]}}},"media":{"audio":"audio/aud_14.wav"},"gold":{"topic":"billing","wants":"true"},"note":"synthesised: Kokoro-82M voice af_bella, speed 1.5, 0.3729 s, speech only (trim)","provenance":{"script":"Refund.","voice":"af_bella","synth":{"model":"hexgrad/Kokoro-82M","package":"kokoro 0.9.4 + misaki","lang_code":"a","seed":0,"native_rate":24000,"rate":16000,"resample":"ffmpeg -ar 16000 -ac 1 -sample_fmt s16","round":6,"speed":"per clip","seconds":"per clip (target_seconds)","trim":{"threshold_abs_int16":328,"margin_samples":320}},"speed":1.5,"target_seconds":[0.2,0.45],"trim":true,"file_present":true,"clip":{"file":"audio/aud_14.wav","voice":"af_bella","seed":0,"speed":1.5,"text":"Refund.","phonemes":["ɹˈifˌʌnd."],"native_seconds":1.0,"seconds":0.3729,"sample_rate":16000,"channels":1,"sample_width_bytes":2,"frames":5967,"target_seconds":[0.2,0.45],"in_target":true,"trim":{"threshold_abs_int16":328,"margin_samples":320,"resampled_samples":16000,"kept":[4620,10587]},"deterministic_two_runs":true,"round":6,"sha256":"ccb80a9da1e09508c72795a5c0d442f024b95a1199c73e6f00699890a1bf730c","synth_s":0.32}}},{"id":"aud_15","source":"own_audio","public":true,"request":{"state":"Voice note from a user.","questions":{"topic":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"wants":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"urgency":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]}}},"media":{"audio":"audio/aud_15.wav"},"gold":{"topic":"delivery","wants":"true","urgency":"2"},"note":"synthesised: Kokoro-82M voice bf_emma, speed 1.0, 36.15 s","provenance":{"script":"Hello, I hope you can help me quickly. My grocery order was meant to arrive between eight and nine this morning, and it is now almost eleven. The tracking page still says the van is on its way, but it has not moved for an hour. I need the milk in that order for my baby's bottles, so please send the driver now, or tell me where the order is so that I can collect it myself. I have already waited two hours, and I cannot go out and leave the baby alone for long. If the van has broken down, I would rather have a refund than wait for another slot tomorrow. My phone is on, and I will keep it next to me all morning. Please call me back as soon as you see this. Thank you.","voice":"bf_emma","synth":{"model":"hexgrad/Kokoro-82M","package":"kokoro 0.9.4 + misaki","lang_code":"a","seed":0,"native_rate":24000,"rate":16000,"resample":"ffmpeg -ar 16000 -ac 1 -sample_fmt s16","round":6,"speed":"per clip","seconds":"per clip (target_seconds)","trim":{"threshold_abs_int16":328,"margin_samples":320}},"speed":1.0,"target_seconds":[33.0,37.0],"trim":false,"file_present":true,"clip":{"file":"audio/aud_15.wav","voice":"bf_emma","seed":0,"speed":1.0,"text":"Hello, I hope you can help me quickly. My grocery order was meant to arrive between eight and nine this morning, and it is now almost eleven. The tracking page still says the van is on its way, but it has not moved for an hour. I need the milk in that order for my baby's bottles, so please send the driver now, or tell me where the order is so that I can collect it myself. I have already waited two hours, and I cannot go out and leave the baby alone for long. If the van has broken down, I would rather have a refund than wait for another slot tomorrow. My phone is on, and I will keep it next to me all morning. Please call me back as soon as you see this. Thank you.","phonemes":["həlˈO, ˌI hˈOp ju kæn hˈɛlp mˌi kwˈɪkli. mˌI ɡɹˈOsəɹi ˈɔɹdəɹ wʌz mˈɛnt tʊ əɹˈIv bətwˈin ˈAt ænd nˈIn ðɪs mˈɔɹnɪŋ, ænd ɪt ɪz nˈW ˈɔlmOst əlˈɛvᵊn. ðə tɹˈækɪŋ pˈAʤ stˈɪl sˈɛz ðə vˈæn ɪz ˌɔn ɪts wˈA, bˌʌt ɪt hæz nˌɑt mˈuvd fɔɹ ɐn ˈWəɹ. ˌI nˈid ðə mˈɪlk ɪn ðˈæt ˈɔɹdəɹ fɔɹ mI bˈAbiz bˈɑTəlz, sˌO plˈiz sˈɛnd ðə dɹˈIvəɹ nˈW, ɔɹ tˈɛl mˌi wˌɛɹ ði ˈɔɹdəɹ ɪz sˌO ðæt ˌI kæn kəlˈɛkt ɪt mIsˈɛlf. ˌI hæv ˌɔlɹˈɛdi wˈATᵻd tˈu ˈWəɹz, ænd ˌI kənˈɑt ɡˌO ˈWt ænd lˈiv ðə bˈAbi əlˈOn fɔɹ lˈɔŋ.","ˌɪf ðə vˈæn hæz bɹˈOkən dˌWn, ˌI wʊd ɹˈæðəɹ hæv ɐ ɹˈifˌʌnd ðən wˈAt fɔɹ ənˈʌðəɹ slˈɑt təmˈɑɹO. mˌI fˈOn ɪz ˌɔn, ænd ˌI wɪl kˈip ɪt nˈɛkst tə mˌi ˈɔl mˈɔɹnɪŋ. plˈiz kˈɔl mˌi bˈæk æz sˈun æz ju sˈi ðˈɪs. θˈæŋk ju."],"native_seconds":36.15,"seconds":36.15,"sample_rate":16000,"channels":1,"sample_width_bytes":2,"frames":578400,"target_seconds":[33.0,37.0],"in_target":true,"trim":null,"deterministic_two_runs":true,"round":6,"sha256":"d2ed03414e9da96d1055afcd92d8b0476d8f1f17c87f70ae9ce57434fb9fafa9","synth_s":6.68}}}],"rows":[{"row_id":"aud_01/audio/topic","fixture_id":"aud_01","mode":"audio","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":["audio/aud_01.wav"],"prefix_rows":121,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,3493,856,779,21009,10789,1337,540,19,16,7832,572,1873,535,835,11746,933,768,15073,8688,20,19,16,7832,572,7778,535,61958,774,810,23325,20,19,16,7832,572,12992,535,21040,3197,20,19,16,7832,572,15296,535,36843,3321,1337,779,3714,20,21],"sequence_length":63,"marker_positions":[18,31,42,51],"positions":184,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-2.8674635887145996,-11.332260131835938,-12.128717422485352,-0.14766040444374084],"probabilities_raw":[0.06181371584534645,1.3027777640672866e-05,5.874532689631451e-06,0.93816739320755],"probabilities":[0.06181371584534645,1.3027777640672866e-05,5.874532689631451e-06,0.93816739320755],"argmax_index":3,"argmax":"feedback","top2_margin":0.8763536773622036,"near_tie":false,"gold":"delivery","official_answer":{"type":"choice","choice":"feedback","confidence":0.93816739320755,"probabilities":{"delivery":0.06181371584534645,"billing":1.3027777640672866e-05,"account":5.874532689631451e-06,"feedback":0.93816739320755}},"usage_input_tokens":502,"oracle_file":"records_ref.json"},{"row_id":"aud_01/audio/wants","fixture_id":"aud_01","mode":"audio","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":["audio/aud_01.wav"],"prefix_rows":121,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":33,"marker_positions":[21,27],"positions":154,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-1.4458585977554321,-0.055500663816928864],"probabilities_raw":[0.19935061037540436,0.8006494045257568],"probabilities":[0.8006494045257568,0.19935061037540436],"argmax_index":0,"argmax":"true","top2_margin":0.6012987941503525,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.8006494045257568},"usage_input_tokens":502,"oracle_file":"records_ref.json"},{"row_id":"aud_01/audio/urgency","fixture_id":"aud_01","mode":"audio","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":["audio/aud_01.wav"],"prefix_rows":121,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":43,"marker_positions":[17,26,34],"positions":164,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-11.046607971191406,-1.6502888202667236,-6.171070098876953],"probabilities_raw":[8.212868851842359e-05,0.9891554713249207,0.010762522928416729],"probabilities":[8.212868851842359e-05,0.9891554713249207,0.010762522928416729],"argmax_index":1,"argmax":"1","top2_margin":0.9783929483965039,"near_tie":false,"gold":"1","official_answer":{"type":"score","score":1.0106805171817541,"confidence":0.9891554713249207,"probabilities":{"0":8.212868851842359e-05,"1":0.9891554713249207,"2":0.010762522928416729},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":502,"oracle_file":"records_ref.json"},{"row_id":"aud_01/text/topic","fixture_id":"aud_01","mode":"text","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,3493,856,779,21009,10789,1337,540,19,16,11746,535,835,11746,933,768,15073,8688,20,19,16,5363,800,535,61958,774,810,23325,20,19,16,3863,535,21040,3197,20,19,16,12779,535,36843,3321,1337,779,3714,20,21],"sequence_length":56,"marker_positions":[18,29,39,46],"positions":56,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.1100118160247803,-1.3073279857635498,-3.290978193283081,1.7141824960708618],"probabilities_raw":[0.020269835367798805,0.04523257911205292,0.0062224725261330605,0.9282751679420471],"probabilities":[0.05386649817228317,0.0955735445022583,0.023170966655015945,0.8273889422416687],"argmax_index":3,"argmax":"feedback","top2_margin":0.7318153977394104,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"feedback","confidence":0.8273889422416687,"probabilities":{"delivery":0.05386649817228317,"billing":0.0955735445022583,"account":0.023170966655015945,"feedback":0.8273889422416687}},"usage_input_tokens":142,"oracle_file":"records_ref.json"},{"row_id":"aud_01/text/wants","fixture_id":"aud_01","mode":"text","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":43,"marker_positions":[21,33],"positions":43,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[2.693018674850464,-4.27215051651001],"probabilities_raw":[0.9990566372871399,0.0009433126542717218],"probabilities":[0.015068508684635162,0.984931468963623],"argmax_index":1,"argmax":"false","top2_margin":0.9698629602789879,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.015068508684635162},"usage_input_tokens":142,"oracle_file":"records_ref.json"},{"row_id":"aud_01/text/urgency","fixture_id":"aud_01","mode":"text","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":43,"marker_positions":[17,26,34],"positions":43,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-8.044438362121582,-6.318297863006592,-8.428503036499023],"probabilities_raw":[0.13698606193065643,0.7697144746780396,0.09329947084188461],"probabilities":[0.22158345580101013,0.6009449362754822,0.17747168242931366],"argmax_index":1,"argmax":"1","top2_margin":0.37936148047447205,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":0.9558883011341095,"confidence":0.6009449362754822,"probabilities":{"0":0.22158345580101013,"1":0.6009449362754822,"2":0.17747168242931366},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":142,"oracle_file":"records_ref.json"},{"row_id":"aud_02/audio/topic","fixture_id":"aud_02","mode":"audio","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":["audio/aud_02.wav"],"prefix_rows":110,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,3493,856,779,21009,10789,1337,540,19,16,7832,572,1873,535,835,11746,933,768,15073,8688,20,19,16,7832,572,7778,535,61958,774,810,23325,20,19,16,7832,572,12992,535,21040,3197,20,19,16,7832,572,15296,535,36843,3321,1337,779,3714,20,21],"sequence_length":63,"marker_positions":[18,31,42,51],"positions":173,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-11.529346466064453,-16.728370666503906,-14.800735473632812,0.060165852308273315],"probabilities_raw":[9.262639650842175e-06,5.114783263593381e-08,3.5155119348928565e-07,0.9999903440475464],"probabilities":[9.262639650842175e-06,5.114783263593381e-08,3.5155119348928565e-07,0.9999903440475464],"argmax_index":3,"argmax":"feedback","top2_margin":0.9999810814078955,"near_tie":false,"gold":"feedback","official_answer":{"type":"choice","choice":"feedback","confidence":0.9999903440475464,"probabilities":{"delivery":9.262639650842175e-06,"billing":5.114783263593381e-08,"account":3.5155119348928565e-07,"feedback":0.9999903440475464}},"usage_input_tokens":469,"oracle_file":"records_ref.json"},{"row_id":"aud_02/audio/wants","fixture_id":"aud_02","mode":"audio","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":["audio/aud_02.wav"],"prefix_rows":110,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":33,"marker_positions":[21,27],"positions":143,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[0.2793175280094147,-3.478671073913574],"probabilities_raw":[0.9772013425827026,0.02279871143400669],"probabilities":[0.02279871143400669,0.9772013425827026],"argmax_index":1,"argmax":"false","top2_margin":0.954402631148696,"near_tie":false,"gold":"false","official_answer":{"type":"noul","noul":0.02279871143400669},"usage_input_tokens":469,"oracle_file":"records_ref.json"},{"row_id":"aud_02/audio/urgency","fixture_id":"aud_02","mode":"audio","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":["audio/aud_02.wav"],"prefix_rows":110,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":43,"marker_positions":[17,26,34],"positions":153,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-12.738178253173828,-5.769467830657959,1.4846340417861938],"probabilities_raw":[6.64972333197511e-07,0.0007067669648677111,0.9992924928665161],"probabilities":[6.64972333197511e-07,0.0007067669648677111,0.9992924928665161],"argmax_index":2,"argmax":"2","top2_margin":0.9985857259016484,"near_tie":false,"gold":"0","official_answer":{"type":"score","score":1.9992917526979,"confidence":0.9992924928665161,"probabilities":{"0":6.64972333197511e-07,"1":0.0007067669648677111,"2":0.9992924928665161},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":469,"oracle_file":"records_ref.json"},{"row_id":"aud_02/text/topic","fixture_id":"aud_02","mode":"text","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,3493,856,779,21009,10789,1337,540,19,16,11746,535,835,11746,933,768,15073,8688,20,19,16,5363,800,535,61958,774,810,23325,20,19,16,3863,535,21040,3197,20,19,16,12779,535,36843,3321,1337,779,3714,20,21],"sequence_length":56,"marker_positions":[18,29,39,46],"positions":56,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.1100118160247803,-1.3073279857635498,-3.290978193283081,1.7141824960708618],"probabilities_raw":[0.020269835367798805,0.04523257911205292,0.0062224725261330605,0.9282751679420471],"probabilities":[0.05386649817228317,0.0955735445022583,0.023170966655015945,0.8273889422416687],"argmax_index":3,"argmax":"feedback","top2_margin":0.7318153977394104,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"feedback","confidence":0.8273889422416687,"probabilities":{"delivery":0.05386649817228317,"billing":0.0955735445022583,"account":0.023170966655015945,"feedback":0.8273889422416687}},"usage_input_tokens":142,"oracle_file":"records_ref.json"},{"row_id":"aud_02/text/wants","fixture_id":"aud_02","mode":"text","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":43,"marker_positions":[21,33],"positions":43,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[2.693018674850464,-4.27215051651001],"probabilities_raw":[0.9990566372871399,0.0009433126542717218],"probabilities":[0.015068508684635162,0.984931468963623],"argmax_index":1,"argmax":"false","top2_margin":0.9698629602789879,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.015068508684635162},"usage_input_tokens":142,"oracle_file":"records_ref.json"},{"row_id":"aud_02/text/urgency","fixture_id":"aud_02","mode":"text","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":43,"marker_positions":[17,26,34],"positions":43,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-8.044438362121582,-6.318297863006592,-8.428503036499023],"probabilities_raw":[0.13698606193065643,0.7697144746780396,0.09329947084188461],"probabilities":[0.22158345580101013,0.6009449362754822,0.17747168242931366],"argmax_index":1,"argmax":"1","top2_margin":0.37936148047447205,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":0.9558883011341095,"confidence":0.6009449362754822,"probabilities":{"0":0.22158345580101013,"1":0.6009449362754822,"2":0.17747168242931366},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":142,"oracle_file":"records_ref.json"},{"row_id":"aud_03/audio/topic","fixture_id":"aud_03","mode":"audio","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":["audio/aud_03.wav"],"prefix_rows":110,"sequence_ids":[1,17,24373,18,3493,856,779,21009,10789,1337,540,19,16,7832,572,1873,535,835,11746,933,768,15073,8688,20,19,16,7832,572,7778,535,61958,774,810,23325,20,19,16,7832,572,12992,535,21040,3197,20,19,16,7832,572,15296,535,36843,3321,1337,779,3714,20,21],"sequence_length":57,"marker_positions":[12,25,36,45],"positions":167,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-8.26899528503418,-5.348294258117676,-9.974102020263672,-1.3370122909545898],"probabilities_raw":[0.0009576162556186318,0.0177678894251585,0.0001740496081765741,0.9811004400253296],"probabilities":[0.0009576162556186318,0.0177678894251585,0.0001740496081765741,0.9811004400253296],"argmax_index":3,"argmax":"feedback","top2_margin":0.9633325506001711,"near_tie":false,"gold":"account","official_answer":{"type":"choice","choice":"feedback","confidence":0.9811004400253296,"probabilities":{"delivery":0.0009576162556186318,"billing":0.0177678894251585,"account":0.0001740496081765741,"feedback":0.9811004400253296}},"usage_input_tokens":451,"oracle_file":"records_ref.json"},{"row_id":"aud_03/audio/wants","fixture_id":"aud_03","mode":"audio","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":["audio/aud_03.wav"],"prefix_rows":110,"sequence_ids":[1,17,24373,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":27,"marker_positions":[15,21],"positions":137,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-1.357582926750183,-1.9369168281555176],"probabilities_raw":[0.6409140825271606,0.3590858578681946],"probabilities":[0.3590858578681946,0.6409140825271606],"argmax_index":1,"argmax":"false","top2_margin":0.28182822465896606,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.3590858578681946},"usage_input_tokens":451,"oracle_file":"records_ref.json"},{"row_id":"aud_03/audio/urgency","fixture_id":"aud_03","mode":"audio","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":["audio/aud_03.wav"],"prefix_rows":110,"sequence_ids":[1,17,24373,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":37,"marker_positions":[11,20,28],"positions":147,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-13.345606803894043,-3.927008628845215,1.5194247961044312],"probabilities_raw":[3.4860235587075294e-07,0.004293142817914486,0.9957064986228943],"probabilities":[3.4860235587075294e-07,0.004293142817914486,0.9957064986228943],"argmax_index":2,"argmax":"2","top2_margin":0.9914133558049798,"near_tie":false,"gold":"2","official_answer":{"type":"score","score":1.995706140063703,"confidence":0.9957064986228943,"probabilities":{"0":3.4860235587075294e-07,"1":0.004293142817914486,"2":0.9957064986228943},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":451,"oracle_file":"records_ref.json"},{"row_id":"aud_03/text/topic","fixture_id":"aud_03","mode":"text","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,3493,856,779,21009,10789,1337,540,19,16,11746,535,835,11746,933,768,15073,8688,20,19,16,5363,800,535,61958,774,810,23325,20,19,16,3863,535,21040,3197,20,19,16,12779,535,36843,3321,1337,779,3714,20,21],"sequence_length":49,"marker_positions":[11,22,32,39],"positions":49,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-4.030765533447266,-2.516629695892334,-4.948546409606934,-1.2068599462509155],"probabilities_raw":[0.04388390854001045,0.19947391748428345,0.017527427524328232,0.7391147613525391],"probabilities":[0.08343102037906647,0.24607208371162415,0.043311357498168945,0.6271855235099792],"argmax_index":3,"argmax":"feedback","top2_margin":0.3811134397983551,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"feedback","confidence":0.6271855235099792,"probabilities":{"delivery":0.08343102037906647,"billing":0.24607208371162415,"account":0.043311357498168945,"feedback":0.6271855235099792}},"usage_input_tokens":121,"oracle_file":"records_ref.json"},{"row_id":"aud_03/text/wants","fixture_id":"aud_03","mode":"text","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":36,"marker_positions":[14,26],"positions":36,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[-0.19816279411315918,3.9951019287109375],"probabilities_raw":[0.014872386120259762,0.9851276278495789],"probabilities":[0.9252889752388,0.07471100240945816],"argmax_index":0,"argmax":"true","top2_margin":0.8505779728293419,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.9252889752388},"usage_input_tokens":121,"oracle_file":"records_ref.json"},{"row_id":"aud_03/text/urgency","fixture_id":"aud_03","mode":"text","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":36,"marker_positions":[10,19,27],"positions":36,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-7.304863452911377,-6.445466995239258,-5.352657318115234],"probabilities_raw":[0.09609891474246979,0.22696013748645782,0.6769409775733948],"probabilities":[0.17439991235733032,0.28659766912460327,0.5390024185180664],"argmax_index":2,"argmax":"2","top2_margin":0.25240474939346313,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":1.364602506160736,"confidence":0.5390024185180664,"probabilities":{"0":0.17439991235733032,"1":0.28659766912460327,"2":0.5390024185180664},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":121,"oracle_file":"records_ref.json"},{"row_id":"aud_04/audio/topic","fixture_id":"aud_04","mode":"audio","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":["audio/aud_04.wav"],"prefix_rows":35,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,3493,856,779,21009,10789,1337,540,19,16,7832,572,1873,535,835,11746,933,768,15073,8688,20,19,16,7832,572,7778,535,61958,774,810,23325,20,19,16,7832,572,12992,535,21040,3197,20,19,16,7832,572,15296,535,36843,3321,1337,779,3714,20,21],"sequence_length":63,"marker_positions":[18,31,42,51],"positions":98,"window":128,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-11.180398941040039,-14.908483505249023,-14.098936080932617,-0.7033980488777161],"probabilities_raw":[2.8176236810395494e-05,6.773240102120326e-07,1.5218730595734087e-06,0.9999696016311646],"probabilities":[2.8176236810395494e-05,6.773240102120326e-07,1.5218730595734087e-06,0.9999696016311646],"argmax_index":3,"argmax":"feedback","top2_margin":0.9999414253943542,"near_tie":false,"gold":"account","official_answer":{"type":"choice","choice":"feedback","confidence":0.9999696016311646,"probabilities":{"delivery":2.8176236810395494e-05,"billing":6.773240102120326e-07,"account":1.5218730595734087e-06,"feedback":0.9999696016311646}},"usage_input_tokens":244,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_04/audio/wants","fixture_id":"aud_04","mode":"audio","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":["audio/aud_04.wav"],"prefix_rows":35,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":33,"marker_positions":[21,27],"positions":68,"window":128,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-0.5335029363632202,-3.4489498138427734],"probabilities_raw":[0.9486048221588135,0.0513952411711216],"probabilities":[0.0513952411711216,0.9486048221588135],"argmax_index":1,"argmax":"false","top2_margin":0.8972095809876919,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.0513952411711216},"usage_input_tokens":244,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_04/audio/urgency","fixture_id":"aud_04","mode":"audio","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":["audio/aud_04.wav"],"prefix_rows":35,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":43,"marker_positions":[17,26,34],"positions":78,"window":128,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-13.666671752929688,-7.450804710388184,2.360034465789795],"probabilities_raw":[1.0956349427715395e-07,5.4850806918693706e-05,0.9999450445175171],"probabilities":[1.0956349427715395e-07,5.4850806918693706e-05,0.9999450445175171],"argmax_index":2,"argmax":"2","top2_margin":0.9998901937105984,"near_tie":false,"gold":"2","official_answer":{"type":"score","score":1.9999449398419529,"confidence":0.9999450445175171,"probabilities":{"0":1.0956349427715395e-07,"1":5.4850806918693706e-05,"2":0.9999450445175171},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":244,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_04/text/topic","fixture_id":"aud_04","mode":"text","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,3493,856,779,21009,10789,1337,540,19,16,11746,535,835,11746,933,768,15073,8688,20,19,16,5363,800,535,61958,774,810,23325,20,19,16,3863,535,21040,3197,20,19,16,12779,535,36843,3321,1337,779,3714,20,21],"sequence_length":56,"marker_positions":[18,29,39,46],"positions":56,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.1100118160247803,-1.3073279857635498,-3.290978193283081,1.7141824960708618],"probabilities_raw":[0.020269835367798805,0.04523257911205292,0.0062224725261330605,0.9282751679420471],"probabilities":[0.05386649817228317,0.0955735445022583,0.023170966655015945,0.8273889422416687],"argmax_index":3,"argmax":"feedback","top2_margin":0.7318153977394104,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"feedback","confidence":0.8273889422416687,"probabilities":{"delivery":0.05386649817228317,"billing":0.0955735445022583,"account":0.023170966655015945,"feedback":0.8273889422416687}},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_04/text/wants","fixture_id":"aud_04","mode":"text","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":43,"marker_positions":[21,33],"positions":43,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[2.693018674850464,-4.27215051651001],"probabilities_raw":[0.9990566372871399,0.0009433126542717218],"probabilities":[0.015068508684635162,0.984931468963623],"argmax_index":1,"argmax":"false","top2_margin":0.9698629602789879,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.015068508684635162},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_04/text/urgency","fixture_id":"aud_04","mode":"text","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":43,"marker_positions":[17,26,34],"positions":43,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-8.044438362121582,-6.318297863006592,-8.428503036499023],"probabilities_raw":[0.13698606193065643,0.7697144746780396,0.09329947084188461],"probabilities":[0.22158345580101013,0.6009449362754822,0.17747168242931366],"argmax_index":1,"argmax":"1","top2_margin":0.37936148047447205,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":0.9558883011341095,"confidence":0.6009449362754822,"probabilities":{"0":0.22158345580101013,"1":0.6009449362754822,"2":0.17747168242931366},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_05/audio/topic","fixture_id":"aud_05","mode":"audio","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":["audio/aud_05.wav"],"prefix_rows":36,"sequence_ids":[1,17,24373,18,3493,856,779,21009,10789,1337,540,19,16,7832,572,1873,535,835,11746,933,768,15073,8688,20,19,16,7832,572,7778,535,61958,774,810,23325,20,19,16,7832,572,12992,535,21040,3197,20,19,16,7832,572,15296,535,36843,3321,1337,779,3714,20,21],"sequence_length":57,"marker_positions":[12,25,36,45],"positions":93,"window":128,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-11.728607177734375,-14.39934253692627,-4.546878814697266,-2.9778451919555664],"probabilities_raw":[0.00013103160017635673,9.067559403774794e-06,0.17233005166053772,0.8275298476219177],"probabilities":[0.00013103160017635673,9.067559403774794e-06,0.17233005166053772,0.8275298476219177],"argmax_index":3,"argmax":"feedback","top2_margin":0.65519979596138,"near_tie":false,"gold":"billing","official_answer":{"type":"choice","choice":"feedback","confidence":0.8275298476219177,"probabilities":{"delivery":0.00013103160017635673,"billing":9.067559403774794e-06,"account":0.17233005166053772,"feedback":0.8275298476219177}},"usage_input_tokens":229,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_05/audio/wants","fixture_id":"aud_05","mode":"audio","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":["audio/aud_05.wav"],"prefix_rows":36,"sequence_ids":[1,17,24373,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":27,"marker_positions":[15,21],"positions":63,"window":64,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-0.6051132678985596,-0.5525320768356323],"probabilities_raw":[0.48685774207115173,0.5131422877311707],"probabilities":[0.5131422877311707,0.48685774207115173],"argmax_index":0,"argmax":"true","top2_margin":0.02628454566001892,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.5131422877311707},"usage_input_tokens":229,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_05/audio/urgency","fixture_id":"aud_05","mode":"audio","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":["audio/aud_05.wav"],"prefix_rows":36,"sequence_ids":[1,17,24373,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":37,"marker_positions":[11,20,28],"positions":73,"window":128,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-13.145566940307617,-6.944738388061523,-7.77335262298584],"probabilities_raw":[0.0014094498474150896,0.6950807571411133,0.30350977182388306],"probabilities":[0.0014094498474150896,0.6950807571411133,0.30350977182388306],"argmax_index":1,"argmax":"1","top2_margin":0.3915709853172302,"near_tie":false,"gold":"0","official_answer":{"type":"score","score":1.3021003007888794,"confidence":0.6950807571411133,"probabilities":{"0":0.0014094498474150896,"1":0.6950807571411133,"2":0.30350977182388306},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":229,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_05/text/topic","fixture_id":"aud_05","mode":"text","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,3493,856,779,21009,10789,1337,540,19,16,11746,535,835,11746,933,768,15073,8688,20,19,16,5363,800,535,61958,774,810,23325,20,19,16,3863,535,21040,3197,20,19,16,12779,535,36843,3321,1337,779,3714,20,21],"sequence_length":49,"marker_positions":[11,22,32,39],"positions":49,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-4.030765533447266,-2.516629695892334,-4.948546409606934,-1.2068599462509155],"probabilities_raw":[0.04388390854001045,0.19947391748428345,0.017527427524328232,0.7391147613525391],"probabilities":[0.08343102037906647,0.24607208371162415,0.043311357498168945,0.6271855235099792],"argmax_index":3,"argmax":"feedback","top2_margin":0.3811134397983551,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"feedback","confidence":0.6271855235099792,"probabilities":{"delivery":0.08343102037906647,"billing":0.24607208371162415,"account":0.043311357498168945,"feedback":0.6271855235099792}},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_05/text/wants","fixture_id":"aud_05","mode":"text","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":36,"marker_positions":[14,26],"positions":36,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[-0.19816279411315918,3.9951019287109375],"probabilities_raw":[0.014872386120259762,0.9851276278495789],"probabilities":[0.9252889752388,0.07471100240945816],"argmax_index":0,"argmax":"true","top2_margin":0.8505779728293419,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.9252889752388},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_05/text/urgency","fixture_id":"aud_05","mode":"text","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":36,"marker_positions":[10,19,27],"positions":36,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-7.304863452911377,-6.445466995239258,-5.352657318115234],"probabilities_raw":[0.09609891474246979,0.22696013748645782,0.6769409775733948],"probabilities":[0.17439991235733032,0.28659766912460327,0.5390024185180664],"argmax_index":2,"argmax":"2","top2_margin":0.25240474939346313,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":1.364602506160736,"confidence":0.5390024185180664,"probabilities":{"0":0.17439991235733032,"1":0.28659766912460327,"2":0.5390024185180664},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_06/audio/topic","fixture_id":"aud_06","mode":"audio","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":["audio/aud_06.wav"],"prefix_rows":101,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,3493,856,779,21009,10789,1337,540,19,16,7832,572,1873,535,835,11746,933,768,15073,8688,20,19,16,7832,572,7778,535,61958,774,810,23325,20,19,16,7832,572,12992,535,21040,3197,20,19,16,7832,572,15296,535,36843,3321,1337,779,3714,20,21],"sequence_length":63,"marker_positions":[18,31,42,51],"positions":164,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-5.219973564147949,-7.757868766784668,-11.41282844543457,0.0007779449224472046],"probabilities_raw":[0.005371889099478722,0.00042455419315956533,1.0980044862662908e-05,0.9941925406455994],"probabilities":[0.005371889099478722,0.00042455419315956533,1.0980044862662908e-05,0.9941925406455994],"argmax_index":3,"argmax":"feedback","top2_margin":0.9888206515461206,"near_tie":false,"gold":"delivery","official_answer":{"type":"choice","choice":"feedback","confidence":0.9941925406455994,"probabilities":{"delivery":0.005371889099478722,"billing":0.00042455419315956533,"account":1.0980044862662908e-05,"feedback":0.9941925406455994}},"usage_input_tokens":442,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_06/audio/wants","fixture_id":"aud_06","mode":"audio","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":["audio/aud_06.wav"],"prefix_rows":101,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":33,"marker_positions":[21,27],"positions":134,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-0.22706714272499084,-3.0375657081604004],"probabilities_raw":[0.9432405829429626,0.05675949156284332],"probabilities":[0.05675949156284332,0.9432405829429626],"argmax_index":1,"argmax":"false","top2_margin":0.8864810913801193,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.05675949156284332},"usage_input_tokens":442,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_06/audio/urgency","fixture_id":"aud_06","mode":"audio","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":["audio/aud_06.wav"],"prefix_rows":101,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":43,"marker_positions":[17,26,34],"positions":144,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-13.540458679199219,-7.2855424880981445,-5.484748840332031],"probabilities_raw":[0.0002722343779169023,0.1417158991098404,0.8580119013786316],"probabilities":[0.0002722343779169023,0.1417158991098404,0.8580119013786316],"argmax_index":2,"argmax":"2","top2_margin":0.7162960022687912,"near_tie":false,"gold":"1","official_answer":{"type":"score","score":1.8577397018671036,"confidence":0.8580119013786316,"probabilities":{"0":0.0002722343779169023,"1":0.1417158991098404,"2":0.8580119013786316},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":442,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_06/text/topic","fixture_id":"aud_06","mode":"text","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,3493,856,779,21009,10789,1337,540,19,16,11746,535,835,11746,933,768,15073,8688,20,19,16,5363,800,535,61958,774,810,23325,20,19,16,3863,535,21040,3197,20,19,16,12779,535,36843,3321,1337,779,3714,20,21],"sequence_length":56,"marker_positions":[18,29,39,46],"positions":56,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.1100118160247803,-1.3073279857635498,-3.290978193283081,1.7141824960708618],"probabilities_raw":[0.020269835367798805,0.04523257911205292,0.0062224725261330605,0.9282751679420471],"probabilities":[0.05386649817228317,0.0955735445022583,0.023170966655015945,0.8273889422416687],"argmax_index":3,"argmax":"feedback","top2_margin":0.7318153977394104,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"feedback","confidence":0.8273889422416687,"probabilities":{"delivery":0.05386649817228317,"billing":0.0955735445022583,"account":0.023170966655015945,"feedback":0.8273889422416687}},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_06/text/wants","fixture_id":"aud_06","mode":"text","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":43,"marker_positions":[21,33],"positions":43,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[2.693018674850464,-4.27215051651001],"probabilities_raw":[0.9990566372871399,0.0009433126542717218],"probabilities":[0.015068508684635162,0.984931468963623],"argmax_index":1,"argmax":"false","top2_margin":0.9698629602789879,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.015068508684635162},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_06/text/urgency","fixture_id":"aud_06","mode":"text","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":43,"marker_positions":[17,26,34],"positions":43,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-8.044438362121582,-6.318297863006592,-8.428503036499023],"probabilities_raw":[0.13698606193065643,0.7697144746780396,0.09329947084188461],"probabilities":[0.22158345580101013,0.6009449362754822,0.17747168242931366],"argmax_index":1,"argmax":"1","top2_margin":0.37936148047447205,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":0.9558883011341095,"confidence":0.6009449362754822,"probabilities":{"0":0.22158345580101013,"1":0.6009449362754822,"2":0.17747168242931366},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_07/audio/topic","fixture_id":"aud_07","mode":"audio","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":["audio/aud_07.wav"],"prefix_rows":106,"sequence_ids":[1,17,24373,18,3493,856,779,21009,10789,1337,540,19,16,7832,572,1873,535,835,11746,933,768,15073,8688,20,19,16,7832,572,7778,535,61958,774,810,23325,20,19,16,7832,572,12992,535,21040,3197,20,19,16,7832,572,15296,535,36843,3321,1337,779,3714,20,21],"sequence_length":57,"marker_positions":[12,25,36,45],"positions":163,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-13.036137580871582,-11.367012023925781,-12.230705261230469,-2.8472213745117188],"probabilities_raw":[3.757253216463141e-05,0.00019941713253501803,8.407468703808263e-05,0.9996789693832397],"probabilities":[3.757253216463141e-05,0.00019941713253501803,8.407468703808263e-05,0.9996789693832397],"argmax_index":3,"argmax":"feedback","top2_margin":0.9994795522507047,"near_tie":false,"gold":"feedback","official_answer":{"type":"choice","choice":"feedback","confidence":0.9996789693832397,"probabilities":{"delivery":3.757253216463141e-05,"billing":0.00019941713253501803,"account":8.407468703808263e-05,"feedback":0.9996789693832397}},"usage_input_tokens":439,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_07/audio/wants","fixture_id":"aud_07","mode":"audio","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":["audio/aud_07.wav"],"prefix_rows":106,"sequence_ids":[1,17,24373,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":27,"marker_positions":[15,21],"positions":133,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[0.03376114368438721,-3.4692351818084717],"probabilities_raw":[0.970772922039032,0.02922709286212921],"probabilities":[0.02922709286212921,0.970772922039032],"argmax_index":1,"argmax":"false","top2_margin":0.9415458291769028,"near_tie":false,"gold":"false","official_answer":{"type":"noul","noul":0.02922709286212921},"usage_input_tokens":439,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_07/audio/urgency","fixture_id":"aud_07","mode":"audio","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":["audio/aud_07.wav"],"prefix_rows":106,"sequence_ids":[1,17,24373,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":37,"marker_positions":[11,20,28],"positions":143,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-9.906222343444824,-5.438223838806152,1.249060869216919],"probabilities_raw":[1.4281534276960883e-05,0.0012450935319066048,0.9987406134605408],"probabilities":[1.4281534276960883e-05,0.0012450935319066048,0.9987406134605408],"argmax_index":2,"argmax":"2","top2_margin":0.9974955199286342,"near_tie":false,"gold":"0","official_answer":{"type":"score","score":1.9987263204529881,"confidence":0.9987406134605408,"probabilities":{"0":1.4281534276960883e-05,"1":0.0012450935319066048,"2":0.9987406134605408},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":439,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_07/text/topic","fixture_id":"aud_07","mode":"text","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,3493,856,779,21009,10789,1337,540,19,16,11746,535,835,11746,933,768,15073,8688,20,19,16,5363,800,535,61958,774,810,23325,20,19,16,3863,535,21040,3197,20,19,16,12779,535,36843,3321,1337,779,3714,20,21],"sequence_length":49,"marker_positions":[11,22,32,39],"positions":49,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-4.030765533447266,-2.516629695892334,-4.948546409606934,-1.2068599462509155],"probabilities_raw":[0.04388390854001045,0.19947391748428345,0.017527427524328232,0.7391147613525391],"probabilities":[0.08343102037906647,0.24607208371162415,0.043311357498168945,0.6271855235099792],"argmax_index":3,"argmax":"feedback","top2_margin":0.3811134397983551,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"feedback","confidence":0.6271855235099792,"probabilities":{"delivery":0.08343102037906647,"billing":0.24607208371162415,"account":0.043311357498168945,"feedback":0.6271855235099792}},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_07/text/wants","fixture_id":"aud_07","mode":"text","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":36,"marker_positions":[14,26],"positions":36,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[-0.19816279411315918,3.9951019287109375],"probabilities_raw":[0.014872386120259762,0.9851276278495789],"probabilities":[0.9252889752388,0.07471100240945816],"argmax_index":0,"argmax":"true","top2_margin":0.8505779728293419,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.9252889752388},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_07/text/urgency","fixture_id":"aud_07","mode":"text","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":36,"marker_positions":[10,19,27],"positions":36,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-7.304863452911377,-6.445466995239258,-5.352657318115234],"probabilities_raw":[0.09609891474246979,0.22696013748645782,0.6769409775733948],"probabilities":[0.17439991235733032,0.28659766912460327,0.5390024185180664],"argmax_index":2,"argmax":"2","top2_margin":0.25240474939346313,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":1.364602506160736,"confidence":0.5390024185180664,"probabilities":{"0":0.17439991235733032,"1":0.28659766912460327,"2":0.5390024185180664},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_08/audio/topic","fixture_id":"aud_08","mode":"audio","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":["audio/aud_08.wav"],"prefix_rows":199,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,3493,856,779,21009,10789,1337,540,19,16,7832,572,1873,535,835,11746,933,768,15073,8688,20,19,16,7832,572,7778,535,61958,774,810,23325,20,19,16,7832,572,12992,535,21040,3197,20,19,16,7832,572,15296,535,36843,3321,1337,779,3714,20,21],"sequence_length":63,"marker_positions":[18,31,42,51],"positions":262,"window":512,"window_r10":512,"temperature_key":null,"temperature":null,"raw_logits":[-8.704425811767578,-3.4039201736450195,-11.297805786132812,-0.0038675889372825623],"probabilities_raw":[0.00016108817362692207,0.03228823095560074,1.204406453325646e-05,0.967538595199585],"probabilities":[0.00016108817362692207,0.03228823095560074,1.204406453325646e-05,0.967538595199585],"argmax_index":3,"argmax":"feedback","top2_margin":0.9352503642439842,"near_tie":false,"gold":"billing","official_answer":{"type":"choice","choice":"feedback","confidence":0.967538595199585,"probabilities":{"delivery":0.00016108817362692207,"billing":0.03228823095560074,"account":1.204406453325646e-05,"feedback":0.967538595199585}},"usage_input_tokens":736,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_08/audio/wants","fixture_id":"aud_08","mode":"audio","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":["audio/aud_08.wav"],"prefix_rows":199,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":33,"marker_positions":[21,27],"positions":232,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-1.890669584274292,1.8680564165115356],"probabilities_raw":[0.022782286629080772,0.9772177338600159],"probabilities":[0.9772177338600159,0.022782286629080772],"argmax_index":0,"argmax":"true","top2_margin":0.9544354472309351,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.9772177338600159},"usage_input_tokens":736,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_08/audio/urgency","fixture_id":"aud_08","mode":"audio","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":["audio/aud_08.wav"],"prefix_rows":199,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":43,"marker_positions":[17,26,34],"positions":242,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-13.792383193969727,-4.434501647949219,-5.2392988204956055],"probabilities_raw":[5.961775968899019e-05,0.6909584999084473,0.30898186564445496],"probabilities":[5.961775968899019e-05,0.6909584999084473,0.30898186564445496],"argmax_index":1,"argmax":"1","top2_margin":0.3819766342639923,"near_tie":false,"gold":"1","official_answer":{"type":"score","score":1.3089222311973572,"confidence":0.6909584999084473,"probabilities":{"0":5.961775968899019e-05,"1":0.6909584999084473,"2":0.30898186564445496},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":736,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_08/text/topic","fixture_id":"aud_08","mode":"text","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,3493,856,779,21009,10789,1337,540,19,16,11746,535,835,11746,933,768,15073,8688,20,19,16,5363,800,535,61958,774,810,23325,20,19,16,3863,535,21040,3197,20,19,16,12779,535,36843,3321,1337,779,3714,20,21],"sequence_length":56,"marker_positions":[18,29,39,46],"positions":56,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.1100118160247803,-1.3073279857635498,-3.290978193283081,1.7141824960708618],"probabilities_raw":[0.020269835367798805,0.04523257911205292,0.0062224725261330605,0.9282751679420471],"probabilities":[0.05386649817228317,0.0955735445022583,0.023170966655015945,0.8273889422416687],"argmax_index":3,"argmax":"feedback","top2_margin":0.7318153977394104,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"feedback","confidence":0.8273889422416687,"probabilities":{"delivery":0.05386649817228317,"billing":0.0955735445022583,"account":0.023170966655015945,"feedback":0.8273889422416687}},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_08/text/wants","fixture_id":"aud_08","mode":"text","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":43,"marker_positions":[21,33],"positions":43,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[2.693018674850464,-4.27215051651001],"probabilities_raw":[0.9990566372871399,0.0009433126542717218],"probabilities":[0.015068508684635162,0.984931468963623],"argmax_index":1,"argmax":"false","top2_margin":0.9698629602789879,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.015068508684635162},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_08/text/urgency","fixture_id":"aud_08","mode":"text","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":43,"marker_positions":[17,26,34],"positions":43,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-8.044438362121582,-6.318297863006592,-8.428503036499023],"probabilities_raw":[0.13698606193065643,0.7697144746780396,0.09329947084188461],"probabilities":[0.22158345580101013,0.6009449362754822,0.17747168242931366],"argmax_index":1,"argmax":"1","top2_margin":0.37936148047447205,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":0.9558883011341095,"confidence":0.6009449362754822,"probabilities":{"0":0.22158345580101013,"1":0.6009449362754822,"2":0.17747168242931366},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_09/audio/topic","fixture_id":"aud_09","mode":"audio","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":["audio/aud_09.wav"],"prefix_rows":172,"sequence_ids":[1,17,24373,18,3493,856,779,21009,10789,1337,540,19,16,7832,572,1873,535,835,11746,933,768,15073,8688,20,19,16,7832,572,7778,535,61958,774,810,23325,20,19,16,7832,572,12992,535,21040,3197,20,19,16,7832,572,15296,535,36843,3321,1337,779,3714,20,21],"sequence_length":57,"marker_positions":[12,25,36,45],"positions":229,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-10.309013366699219,-12.59838581085205,-9.959497451782227,-2.7299599647521973],"probabilities_raw":[0.0005103875882923603,5.1717615860980004e-05,0.0007239239639602602,0.9987140893936157],"probabilities":[0.0005103875882923603,5.1717615860980004e-05,0.0007239239639602602,0.9987140893936157],"argmax_index":3,"argmax":"feedback","top2_margin":0.9979901654296555,"near_tie":false,"gold":"account","official_answer":{"type":"choice","choice":"feedback","confidence":0.9987140893936157,"probabilities":{"delivery":0.0005103875882923603,"billing":5.1717615860980004e-05,"account":0.0007239239639602602,"feedback":0.9987140893936157}},"usage_input_tokens":637,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_09/audio/wants","fixture_id":"aud_09","mode":"audio","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":["audio/aud_09.wav"],"prefix_rows":172,"sequence_ids":[1,17,24373,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":27,"marker_positions":[15,21],"positions":199,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-1.9508432149887085,-2.6538851261138916],"probabilities_raw":[0.6688618659973145,0.33113816380500793],"probabilities":[0.33113816380500793,0.6688618659973145],"argmax_index":1,"argmax":"false","top2_margin":0.3377237021923065,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.33113816380500793},"usage_input_tokens":637,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_09/audio/urgency","fixture_id":"aud_09","mode":"audio","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":["audio/aud_09.wav"],"prefix_rows":172,"sequence_ids":[1,17,24373,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":37,"marker_positions":[11,20,28],"positions":209,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-12.979911804199219,-4.641338348388672,3.067082166671753],"probabilities_raw":[1.0732090771625735e-07,0.00044882844667881727,0.999550998210907],"probabilities":[1.0732090771625735e-07,0.00044882844667881727,0.999550998210907],"argmax_index":2,"argmax":"2","top2_margin":0.9991021697642282,"near_tie":false,"gold":"2","official_answer":{"type":"score","score":1.9995508248684928,"confidence":0.999550998210907,"probabilities":{"0":1.0732090771625735e-07,"1":0.00044882844667881727,"2":0.999550998210907},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":637,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_09/text/topic","fixture_id":"aud_09","mode":"text","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,3493,856,779,21009,10789,1337,540,19,16,11746,535,835,11746,933,768,15073,8688,20,19,16,5363,800,535,61958,774,810,23325,20,19,16,3863,535,21040,3197,20,19,16,12779,535,36843,3321,1337,779,3714,20,21],"sequence_length":49,"marker_positions":[11,22,32,39],"positions":49,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-4.030765533447266,-2.516629695892334,-4.948546409606934,-1.2068599462509155],"probabilities_raw":[0.04388390854001045,0.19947391748428345,0.017527427524328232,0.7391147613525391],"probabilities":[0.08343102037906647,0.24607208371162415,0.043311357498168945,0.6271855235099792],"argmax_index":3,"argmax":"feedback","top2_margin":0.3811134397983551,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"feedback","confidence":0.6271855235099792,"probabilities":{"delivery":0.08343102037906647,"billing":0.24607208371162415,"account":0.043311357498168945,"feedback":0.6271855235099792}},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_09/text/wants","fixture_id":"aud_09","mode":"text","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":36,"marker_positions":[14,26],"positions":36,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[-0.19816279411315918,3.9951019287109375],"probabilities_raw":[0.014872386120259762,0.9851276278495789],"probabilities":[0.9252889752388,0.07471100240945816],"argmax_index":0,"argmax":"true","top2_margin":0.8505779728293419,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.9252889752388},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_09/text/urgency","fixture_id":"aud_09","mode":"text","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":36,"marker_positions":[10,19,27],"positions":36,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-7.304863452911377,-6.445466995239258,-5.352657318115234],"probabilities_raw":[0.09609891474246979,0.22696013748645782,0.6769409775733948],"probabilities":[0.17439991235733032,0.28659766912460327,0.5390024185180664],"argmax_index":2,"argmax":"2","top2_margin":0.25240474939346313,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":1.364602506160736,"confidence":0.5390024185180664,"probabilities":{"0":0.17439991235733032,"1":0.28659766912460327,"2":0.5390024185180664},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_10/audio/topic","fixture_id":"aud_10","mode":"audio","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":["audio/aud_10.wav"],"prefix_rows":269,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,3493,856,779,21009,10789,1337,540,19,16,7832,572,1873,535,835,11746,933,768,15073,8688,20,19,16,7832,572,7778,535,61958,774,810,23325,20,19,16,7832,572,12992,535,21040,3197,20,19,16,7832,572,15296,535,36843,3321,1337,779,3714,20,21],"sequence_length":63,"marker_positions":[18,31,42,51],"positions":332,"window":512,"window_r10":512,"temperature_key":null,"temperature":null,"raw_logits":[-6.047428131103516,-11.045299530029297,-10.394746780395508,-0.31905287504196167],"probabilities_raw":[0.003241606056690216,2.1888312403461896e-05,4.195113797322847e-05,0.9966945648193359],"probabilities":[0.003241606056690216,2.1888312403461896e-05,4.195113797322847e-05,0.9966945648193359],"argmax_index":3,"argmax":"feedback","top2_margin":0.9934529587626457,"near_tie":false,"gold":"feedback","official_answer":{"type":"choice","choice":"feedback","confidence":0.9966945648193359,"probabilities":{"delivery":0.003241606056690216,"billing":2.1888312403461896e-05,"account":4.195113797322847e-05,"feedback":0.9966945648193359}},"usage_input_tokens":946,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_10/audio/wants","fixture_id":"aud_10","mode":"audio","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":["audio/aud_10.wav"],"prefix_rows":269,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":33,"marker_positions":[21,27],"positions":302,"window":512,"window_r10":512,"temperature_key":null,"temperature":null,"raw_logits":[-0.8847565054893494,-3.556145191192627],"probabilities_raw":[0.9353170394897461,0.06468290090560913],"probabilities":[0.06468290090560913,0.9353170394897461],"argmax_index":1,"argmax":"false","top2_margin":0.870634138584137,"near_tie":false,"gold":"false","official_answer":{"type":"noul","noul":0.06468290090560913},"usage_input_tokens":946,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_10/audio/urgency","fixture_id":"aud_10","mode":"audio","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":["audio/aud_10.wav"],"prefix_rows":269,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":43,"marker_positions":[17,26,34],"positions":312,"window":512,"window_r10":512,"temperature_key":null,"temperature":null,"raw_logits":[-12.225658416748047,-3.0319995880126953,-1.7885916233062744],"probabilities_raw":[2.27603468374582e-05,0.22383823990821838,0.776138961315155],"probabilities":[2.27603468374582e-05,0.22383823990821838,0.776138961315155],"argmax_index":2,"argmax":"2","top2_margin":0.5523007214069366,"near_tie":false,"gold":"0","official_answer":{"type":"score","score":1.7761161625385284,"confidence":0.776138961315155,"probabilities":{"0":2.27603468374582e-05,"1":0.22383823990821838,"2":0.776138961315155},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":946,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_10/text/topic","fixture_id":"aud_10","mode":"text","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,3493,856,779,21009,10789,1337,540,19,16,11746,535,835,11746,933,768,15073,8688,20,19,16,5363,800,535,61958,774,810,23325,20,19,16,3863,535,21040,3197,20,19,16,12779,535,36843,3321,1337,779,3714,20,21],"sequence_length":56,"marker_positions":[18,29,39,46],"positions":56,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.1100118160247803,-1.3073279857635498,-3.290978193283081,1.7141824960708618],"probabilities_raw":[0.020269835367798805,0.04523257911205292,0.0062224725261330605,0.9282751679420471],"probabilities":[0.05386649817228317,0.0955735445022583,0.023170966655015945,0.8273889422416687],"argmax_index":3,"argmax":"feedback","top2_margin":0.7318153977394104,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"feedback","confidence":0.8273889422416687,"probabilities":{"delivery":0.05386649817228317,"billing":0.0955735445022583,"account":0.023170966655015945,"feedback":0.8273889422416687}},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_10/text/wants","fixture_id":"aud_10","mode":"text","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":43,"marker_positions":[21,33],"positions":43,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[2.693018674850464,-4.27215051651001],"probabilities_raw":[0.9990566372871399,0.0009433126542717218],"probabilities":[0.015068508684635162,0.984931468963623],"argmax_index":1,"argmax":"false","top2_margin":0.9698629602789879,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.015068508684635162},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_10/text/urgency","fixture_id":"aud_10","mode":"text","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":43,"marker_positions":[17,26,34],"positions":43,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-8.044438362121582,-6.318297863006592,-8.428503036499023],"probabilities_raw":[0.13698606193065643,0.7697144746780396,0.09329947084188461],"probabilities":[0.22158345580101013,0.6009449362754822,0.17747168242931366],"argmax_index":1,"argmax":"1","top2_margin":0.37936148047447205,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":0.9558883011341095,"confidence":0.6009449362754822,"probabilities":{"0":0.22158345580101013,"1":0.6009449362754822,"2":0.17747168242931366},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_11/audio/topic","fixture_id":"aud_11","mode":"audio","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":["audio/aud_11.wav"],"prefix_rows":263,"sequence_ids":[1,17,24373,18,3493,856,779,21009,10789,1337,540,19,16,7832,572,1873,535,835,11746,933,768,15073,8688,20,19,16,7832,572,7778,535,61958,774,810,23325,20,19,16,7832,572,12992,535,21040,3197,20,19,16,7832,572,15296,535,36843,3321,1337,779,3714,20,21],"sequence_length":57,"marker_positions":[12,25,36,45],"positions":320,"window":512,"window_r10":512,"temperature_key":null,"temperature":null,"raw_logits":[-5.378786087036133,-6.809049606323242,-13.527311325073242,-2.2166378498077393],"probabilities_raw":[0.04022393375635147,0.009623407386243343,1.1631209417828359e-05,0.9501410126686096],"probabilities":[0.04022393375635147,0.009623407386243343,1.1631209417828359e-05,0.9501410126686096],"argmax_index":3,"argmax":"feedback","top2_margin":0.9099170789122581,"near_tie":false,"gold":"delivery","official_answer":{"type":"choice","choice":"feedback","confidence":0.9501410126686096,"probabilities":{"delivery":0.04022393375635147,"billing":0.009623407386243343,"account":1.1631209417828359e-05,"feedback":0.9501410126686096}},"usage_input_tokens":910,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_11/audio/wants","fixture_id":"aud_11","mode":"audio","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":["audio/aud_11.wav"],"prefix_rows":263,"sequence_ids":[1,17,24373,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":27,"marker_positions":[15,21],"positions":290,"window":512,"window_r10":512,"temperature_key":null,"temperature":null,"raw_logits":[-2.169802665710449,-2.4261226654052734],"probabilities_raw":[0.5637314319610596,0.43626853823661804],"probabilities":[0.43626853823661804,0.5637314319610596],"argmax_index":1,"argmax":"false","top2_margin":0.12746289372444153,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.43626853823661804},"usage_input_tokens":910,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_11/audio/urgency","fixture_id":"aud_11","mode":"audio","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":["audio/aud_11.wav"],"prefix_rows":263,"sequence_ids":[1,17,24373,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":37,"marker_positions":[11,20,28],"positions":300,"window":512,"window_r10":512,"temperature_key":null,"temperature":null,"raw_logits":[-13.344354629516602,-3.0399885177612305,-0.4444863498210907],"probabilities_raw":[2.3249160676641623e-06,0.06942828744649887,0.9305693507194519],"probabilities":[2.3249160676641623e-06,0.06942828744649887,0.9305693507194519],"argmax_index":2,"argmax":"2","top2_margin":0.861141063272953,"near_tie":false,"gold":"1","official_answer":{"type":"score","score":1.9305669888854027,"confidence":0.9305693507194519,"probabilities":{"0":2.3249160676641623e-06,"1":0.06942828744649887,"2":0.9305693507194519},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":910,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_11/text/topic","fixture_id":"aud_11","mode":"text","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,3493,856,779,21009,10789,1337,540,19,16,11746,535,835,11746,933,768,15073,8688,20,19,16,5363,800,535,61958,774,810,23325,20,19,16,3863,535,21040,3197,20,19,16,12779,535,36843,3321,1337,779,3714,20,21],"sequence_length":49,"marker_positions":[11,22,32,39],"positions":49,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-4.030765533447266,-2.516629695892334,-4.948546409606934,-1.2068599462509155],"probabilities_raw":[0.04388390854001045,0.19947391748428345,0.017527427524328232,0.7391147613525391],"probabilities":[0.08343102037906647,0.24607208371162415,0.043311357498168945,0.6271855235099792],"argmax_index":3,"argmax":"feedback","top2_margin":0.3811134397983551,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"feedback","confidence":0.6271855235099792,"probabilities":{"delivery":0.08343102037906647,"billing":0.24607208371162415,"account":0.043311357498168945,"feedback":0.6271855235099792}},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_11/text/wants","fixture_id":"aud_11","mode":"text","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":36,"marker_positions":[14,26],"positions":36,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[-0.19816279411315918,3.9951019287109375],"probabilities_raw":[0.014872386120259762,0.9851276278495789],"probabilities":[0.9252889752388,0.07471100240945816],"argmax_index":0,"argmax":"true","top2_margin":0.8505779728293419,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.9252889752388},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_11/text/urgency","fixture_id":"aud_11","mode":"text","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":36,"marker_positions":[10,19,27],"positions":36,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-7.304863452911377,-6.445466995239258,-5.352657318115234],"probabilities_raw":[0.09609891474246979,0.22696013748645782,0.6769409775733948],"probabilities":[0.17439991235733032,0.28659766912460327,0.5390024185180664],"argmax_index":2,"argmax":"2","top2_margin":0.25240474939346313,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":1.364602506160736,"confidence":0.5390024185180664,"probabilities":{"0":0.17439991235733032,"1":0.28659766912460327,"2":0.5390024185180664},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_12/audio/topic","fixture_id":"aud_12","mode":"audio","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":["audio/aud_12.wav"],"prefix_rows":350,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,3493,856,779,21009,10789,1337,540,19,16,7832,572,1873,535,835,11746,933,768,15073,8688,20,19,16,7832,572,7778,535,61958,774,810,23325,20,19,16,7832,572,12992,535,21040,3197,20,19,16,7832,572,15296,535,36843,3321,1337,779,3714,20,21],"sequence_length":63,"marker_positions":[18,31,42,51],"positions":413,"window":512,"window_r10":512,"temperature_key":null,"temperature":null,"raw_logits":[-8.041118621826172,-6.483062744140625,-13.2620849609375,-0.41955888271331787],"probabilities_raw":[0.0004884006921201944,0.0023196975234895945,2.638391379150562e-06,0.9971892237663269],"probabilities":[0.0004884006921201944,0.0023196975234895945,2.638391379150562e-06,0.9971892237663269],"argmax_index":3,"argmax":"feedback","top2_margin":0.9948695262428373,"near_tie":false,"gold":"billing","official_answer":{"type":"choice","choice":"feedback","confidence":0.9971892237663269,"probabilities":{"delivery":0.0004884006921201944,"billing":0.0023196975234895945,"account":2.638391379150562e-06,"feedback":0.9971892237663269}},"usage_input_tokens":1189,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_12/audio/wants","fixture_id":"aud_12","mode":"audio","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":["audio/aud_12.wav"],"prefix_rows":350,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":33,"marker_positions":[21,27],"positions":383,"window":512,"window_r10":512,"temperature_key":null,"temperature":null,"raw_logits":[-0.8726661205291748,-2.7932546138763428],"probabilities_raw":[0.8722040057182312,0.12779594957828522],"probabilities":[0.12779594957828522,0.8722040057182312],"argmax_index":1,"argmax":"false","top2_margin":0.744408056139946,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.12779594957828522},"usage_input_tokens":1189,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_12/audio/urgency","fixture_id":"aud_12","mode":"audio","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":["audio/aud_12.wav"],"prefix_rows":350,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":43,"marker_positions":[17,26,34],"positions":393,"window":512,"window_r10":512,"temperature_key":null,"temperature":null,"raw_logits":[-10.186288833618164,-1.5687068700790405,0.817966103553772],"probabilities_raw":[1.5230326425808016e-05,0.08419332653284073,0.915791392326355],"probabilities":[1.5230326425808016e-05,0.08419332653284073,0.915791392326355],"argmax_index":2,"argmax":"2","top2_margin":0.8315980657935143,"near_tie":false,"gold":"2","official_answer":{"type":"score","score":1.9157761111855507,"confidence":0.915791392326355,"probabilities":{"0":1.5230326425808016e-05,"1":0.08419332653284073,"2":0.915791392326355},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":1189,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_12/text/topic","fixture_id":"aud_12","mode":"text","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,3493,856,779,21009,10789,1337,540,19,16,11746,535,835,11746,933,768,15073,8688,20,19,16,5363,800,535,61958,774,810,23325,20,19,16,3863,535,21040,3197,20,19,16,12779,535,36843,3321,1337,779,3714,20,21],"sequence_length":56,"marker_positions":[18,29,39,46],"positions":56,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.1100118160247803,-1.3073279857635498,-3.290978193283081,1.7141824960708618],"probabilities_raw":[0.020269835367798805,0.04523257911205292,0.0062224725261330605,0.9282751679420471],"probabilities":[0.05386649817228317,0.0955735445022583,0.023170966655015945,0.8273889422416687],"argmax_index":3,"argmax":"feedback","top2_margin":0.7318153977394104,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"feedback","confidence":0.8273889422416687,"probabilities":{"delivery":0.05386649817228317,"billing":0.0955735445022583,"account":0.023170966655015945,"feedback":0.8273889422416687}},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_12/text/wants","fixture_id":"aud_12","mode":"text","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":43,"marker_positions":[21,33],"positions":43,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[2.693018674850464,-4.27215051651001],"probabilities_raw":[0.9990566372871399,0.0009433126542717218],"probabilities":[0.015068508684635162,0.984931468963623],"argmax_index":1,"argmax":"false","top2_margin":0.9698629602789879,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.015068508684635162},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_12/text/urgency","fixture_id":"aud_12","mode":"text","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":43,"marker_positions":[17,26,34],"positions":43,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-8.044438362121582,-6.318297863006592,-8.428503036499023],"probabilities_raw":[0.13698606193065643,0.7697144746780396,0.09329947084188461],"probabilities":[0.22158345580101013,0.6009449362754822,0.17747168242931366],"argmax_index":1,"argmax":"1","top2_margin":0.37936148047447205,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":0.9558883011341095,"confidence":0.6009449362754822,"probabilities":{"0":0.22158345580101013,"1":0.6009449362754822,"2":0.17747168242931366},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_13/audio/topic","fixture_id":"aud_13","mode":"audio","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":["audio/aud_13.wav"],"prefix_rows":356,"sequence_ids":[1,17,24373,18,3493,856,779,21009,10789,1337,540,19,16,7832,572,1873,535,835,11746,933,768,15073,8688,20,19,16,7832,572,7778,535,61958,774,810,23325,20,19,16,7832,572,12992,535,21040,3197,20,19,16,7832,572,15296,535,36843,3321,1337,779,3714,20,21],"sequence_length":57,"marker_positions":[12,25,36,45],"positions":413,"window":512,"window_r10":512,"temperature_key":null,"temperature":null,"raw_logits":[-10.233248710632324,-8.533600807189941,-11.744644165039062,-1.8605585098266602],"probabilities_raw":[0.00023073617194313556,0.0012625929666683078,5.0900838687084615e-05,0.9984557628631592],"probabilities":[0.00023073617194313556,0.0012625929666683078,5.0900838687084615e-05,0.9984557628631592],"argmax_index":3,"argmax":"feedback","top2_margin":0.9971931698964909,"near_tie":false,"gold":"account","official_answer":{"type":"choice","choice":"feedback","confidence":0.9984557628631592,"probabilities":{"delivery":0.00023073617194313556,"billing":0.0012625929666683078,"account":5.0900838687084615e-05,"feedback":0.9984557628631592}},"usage_input_tokens":1189,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_13/audio/wants","fixture_id":"aud_13","mode":"audio","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":["audio/aud_13.wav"],"prefix_rows":356,"sequence_ids":[1,17,24373,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":27,"marker_positions":[15,21],"positions":383,"window":512,"window_r10":512,"temperature_key":null,"temperature":null,"raw_logits":[-1.111641764640808,-1.4849694967269897],"probabilities_raw":[0.5922628045082092,0.4077371656894684],"probabilities":[0.4077371656894684,0.5922628045082092],"argmax_index":1,"argmax":"false","top2_margin":0.18452563881874084,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.4077371656894684},"usage_input_tokens":1189,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_13/audio/urgency","fixture_id":"aud_13","mode":"audio","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":["audio/aud_13.wav"],"prefix_rows":356,"sequence_ids":[1,17,24373,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":37,"marker_positions":[11,20,28],"positions":393,"window":512,"window_r10":512,"temperature_key":null,"temperature":null,"raw_logits":[-8.093282699584961,-3.591269016265869,1.4334510564804077],"probabilities_raw":[7.239612750709057e-05,0.006530025042593479,0.99339759349823],"probabilities":[7.239612750709057e-05,0.006530025042593479,0.99339759349823],"argmax_index":2,"argmax":"2","top2_margin":0.9868675684556365,"near_tie":false,"gold":"0","official_answer":{"type":"score","score":1.9933252120390534,"confidence":0.99339759349823,"probabilities":{"0":7.239612750709057e-05,"1":0.006530025042593479,"2":0.99339759349823},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":1189,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_13/text/topic","fixture_id":"aud_13","mode":"text","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,3493,856,779,21009,10789,1337,540,19,16,11746,535,835,11746,933,768,15073,8688,20,19,16,5363,800,535,61958,774,810,23325,20,19,16,3863,535,21040,3197,20,19,16,12779,535,36843,3321,1337,779,3714,20,21],"sequence_length":49,"marker_positions":[11,22,32,39],"positions":49,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-4.030765533447266,-2.516629695892334,-4.948546409606934,-1.2068599462509155],"probabilities_raw":[0.04388390854001045,0.19947391748428345,0.017527427524328232,0.7391147613525391],"probabilities":[0.08343102037906647,0.24607208371162415,0.043311357498168945,0.6271855235099792],"argmax_index":3,"argmax":"feedback","top2_margin":0.3811134397983551,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"feedback","confidence":0.6271855235099792,"probabilities":{"delivery":0.08343102037906647,"billing":0.24607208371162415,"account":0.043311357498168945,"feedback":0.6271855235099792}},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_13/text/wants","fixture_id":"aud_13","mode":"text","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":36,"marker_positions":[14,26],"positions":36,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[-0.19816279411315918,3.9951019287109375],"probabilities_raw":[0.014872386120259762,0.9851276278495789],"probabilities":[0.9252889752388,0.07471100240945816],"argmax_index":0,"argmax":"true","top2_margin":0.8505779728293419,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.9252889752388},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_13/text/urgency","fixture_id":"aud_13","mode":"text","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":36,"marker_positions":[10,19,27],"positions":36,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-7.304863452911377,-6.445466995239258,-5.352657318115234],"probabilities_raw":[0.09609891474246979,0.22696013748645782,0.6769409775733948],"probabilities":[0.17439991235733032,0.28659766912460327,0.5390024185180664],"argmax_index":2,"argmax":"2","top2_margin":0.25240474939346313,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":1.364602506160736,"confidence":0.5390024185180664,"probabilities":{"0":0.17439991235733032,"1":0.28659766912460327,"2":0.5390024185180664},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_14/audio/topic","fixture_id":"aud_14","mode":"audio","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":["audio/aud_14.wav"],"prefix_rows":7,"sequence_ids":[1,17,24373,18,3493,856,779,21009,10789,1337,540,19,16,7832,572,1873,535,835,11746,933,768,15073,8688,20,19,16,7832,572,7778,535,61958,774,810,23325,20,19,16,7832,572,12992,535,21040,3197,20,19,16,7832,572,15296,535,36843,3321,1337,779,3714,20,21],"sequence_length":57,"marker_positions":[12,25,36,45],"positions":64,"window":64,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-14.164146423339844,-10.380624771118164,-15.470945358276367,-2.3223929405212402],"probabilities_raw":[7.1953281803871505e-06,0.0003163830260746181,1.9476683519314975e-06,0.9996744394302368],"probabilities":[7.1953281803871505e-06,0.0003163830260746181,1.9476683519314975e-06,0.9996744394302368],"argmax_index":3,"argmax":"feedback","top2_margin":0.9993580564041622,"near_tie":false,"gold":"billing","official_answer":{"type":"choice","choice":"feedback","confidence":0.9996744394302368,"probabilities":{"delivery":7.1953281803871505e-06,"billing":0.0003163830260746181,"account":1.9476683519314975e-06,"feedback":0.9996744394302368}},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_14/audio/wants","fixture_id":"aud_14","mode":"audio","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":["audio/aud_14.wav"],"prefix_rows":7,"sequence_ids":[1,17,24373,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":27,"marker_positions":[15,21],"positions":34,"window":64,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-1.1729061603546143,-0.18968117237091064],"probabilities_raw":[0.27225232124328613,0.7277476191520691],"probabilities":[0.7277476191520691,0.27225232124328613],"argmax_index":0,"argmax":"true","top2_margin":0.45549529790878296,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.7277476191520691},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_14/audio/urgency","fixture_id":"aud_14","mode":"audio","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":["audio/aud_14.wav"],"prefix_rows":7,"sequence_ids":[1,17,24373,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":37,"marker_positions":[11,20,28],"positions":44,"window":64,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-10.382744789123535,-6.77346658706665,-5.851489067077637],"probabilities_raw":[0.007644413039088249,0.2823798954486847,0.7099757194519043],"probabilities":[0.007644413039088249,0.2823798954486847,0.7099757194519043],"argmax_index":2,"argmax":"2","top2_margin":0.4275958240032196,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":1.7023313343524933,"confidence":0.7099757194519043,"probabilities":{"0":0.007644413039088249,"1":0.2823798954486847,"2":0.7099757194519043},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_14/text/topic","fixture_id":"aud_14","mode":"text","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,3493,856,779,21009,10789,1337,540,19,16,11746,535,835,11746,933,768,15073,8688,20,19,16,5363,800,535,61958,774,810,23325,20,19,16,3863,535,21040,3197,20,19,16,12779,535,36843,3321,1337,779,3714,20,21],"sequence_length":49,"marker_positions":[11,22,32,39],"positions":49,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-4.030765533447266,-2.516629695892334,-4.948546409606934,-1.2068599462509155],"probabilities_raw":[0.04388390854001045,0.19947391748428345,0.017527427524328232,0.7391147613525391],"probabilities":[0.08343102037906647,0.24607208371162415,0.043311357498168945,0.6271855235099792],"argmax_index":3,"argmax":"feedback","top2_margin":0.3811134397983551,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"feedback","confidence":0.6271855235099792,"probabilities":{"delivery":0.08343102037906647,"billing":0.24607208371162415,"account":0.043311357498168945,"feedback":0.6271855235099792}},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_14/text/wants","fixture_id":"aud_14","mode":"text","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":36,"marker_positions":[14,26],"positions":36,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[-0.19816279411315918,3.9951019287109375],"probabilities_raw":[0.014872386120259762,0.9851276278495789],"probabilities":[0.9252889752388,0.07471100240945816],"argmax_index":0,"argmax":"true","top2_margin":0.8505779728293419,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.9252889752388},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_14/text/urgency","fixture_id":"aud_14","mode":"text","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":36,"marker_positions":[10,19,27],"positions":36,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-7.304863452911377,-6.445466995239258,-5.352657318115234],"probabilities_raw":[0.09609891474246979,0.22696013748645782,0.6769409775733948],"probabilities":[0.17439991235733032,0.28659766912460327,0.5390024185180664],"argmax_index":2,"argmax":"2","top2_margin":0.25240474939346313,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":1.364602506160736,"confidence":0.5390024185180664,"probabilities":{"0":0.17439991235733032,"1":0.28659766912460327,"2":0.5390024185180664},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":121,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_15/audio/topic","fixture_id":"aud_15","mode":"audio","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":["audio/aud_15.wav"],"prefix_rows":375,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,3493,856,779,21009,10789,1337,540,19,16,7832,572,1873,535,835,11746,933,768,15073,8688,20,19,16,7832,572,7778,535,61958,774,810,23325,20,19,16,7832,572,12992,535,21040,3197,20,19,16,7832,572,15296,535,36843,3321,1337,779,3714,20,21],"sequence_length":63,"marker_positions":[18,31,42,51],"positions":438,"window":512,"window_r10":512,"temperature_key":null,"temperature":null,"raw_logits":[-7.3204827308654785,-7.056801795959473,-11.541250228881836,-1.0992538928985596],"probabilities_raw":[0.001977700972929597,0.0025743974838405848,2.9047207135590725e-05,0.995418906211853],"probabilities":[0.001977700972929597,0.0025743974838405848,2.9047207135590725e-05,0.995418906211853],"argmax_index":3,"argmax":"feedback","top2_margin":0.9928445087280124,"near_tie":false,"gold":"delivery","official_answer":{"type":"choice","choice":"feedback","confidence":0.995418906211853,"probabilities":{"delivery":0.001977700972929597,"billing":0.0025743974838405848,"account":2.9047207135590725e-05,"feedback":0.995418906211853}},"usage_input_tokens":1264,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_15/audio/wants","fixture_id":"aud_15","mode":"audio","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":["audio/aud_15.wav"],"prefix_rows":375,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":33,"marker_positions":[21,27],"positions":408,"window":512,"window_r10":512,"temperature_key":null,"temperature":null,"raw_logits":[-1.6256054639816284,0.10680553317070007],"probabilities_raw":[0.1502794474363327,0.8497205972671509],"probabilities":[0.8497205972671509,0.1502794474363327],"argmax_index":0,"argmax":"true","top2_margin":0.6994411498308182,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.8497205972671509},"usage_input_tokens":1264,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_15/audio/urgency","fixture_id":"aud_15","mode":"audio","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":["audio/aud_15.wav"],"prefix_rows":375,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":43,"marker_positions":[17,26,34],"positions":418,"window":512,"window_r10":512,"temperature_key":null,"temperature":null,"raw_logits":[-12.868496894836426,-3.0056636333465576,4.093652725219727],"probabilities_raw":[4.2960873969377644e-08,0.0008249877719208598,0.9991748929023743],"probabilities":[4.2960873969377644e-08,0.0008249877719208598,0.9991748929023743],"argmax_index":2,"argmax":"2","top2_margin":0.9983499051304534,"near_tie":false,"gold":"2","official_answer":{"type":"score","score":1.9991747735766694,"confidence":0.9991748929023743,"probabilities":{"0":4.2960873969377644e-08,"1":0.0008249877719208598,"2":0.9991748929023743},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":1264,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_15/text/topic","fixture_id":"aud_15","mode":"text","question_id":"topic","question":{"type":"choice","instructions":"What is the speaker talking about?","criteria":{"delivery":"A delivery or a damaged item","billing":"Charges and payments","account":"Account access","feedback":"Feedback about the service"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,3493,856,779,21009,10789,1337,540,19,16,11746,535,835,11746,933,768,15073,8688,20,19,16,5363,800,535,61958,774,810,23325,20,19,16,3863,535,21040,3197,20,19,16,12779,535,36843,3321,1337,779,3714,20,21],"sequence_length":56,"marker_positions":[18,29,39,46],"positions":56,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.1100118160247803,-1.3073279857635498,-3.290978193283081,1.7141824960708618],"probabilities_raw":[0.020269835367798805,0.04523257911205292,0.0062224725261330605,0.9282751679420471],"probabilities":[0.05386649817228317,0.0955735445022583,0.023170966655015945,0.8273889422416687],"argmax_index":3,"argmax":"feedback","top2_margin":0.7318153977394104,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"feedback","confidence":0.8273889422416687,"probabilities":{"delivery":0.05386649817228317,"billing":0.0955735445022583,"account":0.023170966655015945,"feedback":0.8273889422416687}},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_15/text/wants","fixture_id":"aud_15","mode":"text","question_id":"wants","question":{"type":"noul","instructions":"Is the speaker asking for something to be done?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4940,779,21009,14065,875,3503,811,874,4007,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":43,"marker_positions":[21,33],"positions":43,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[2.693018674850464,-4.27215051651001],"probabilities_raw":[0.9990566372871399,0.0009433126542717218],"probabilities":[0.015068508684635162,0.984931468963623],"argmax_index":1,"argmax":"false","top2_margin":0.9698629602789879,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.015068508684635162},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"aud_15/text/urgency","fixture_id":"aud_15","mode":"text","question_id":"urgency","question":{"type":"score","instructions":"How urgent is the request?","criteria":["Can wait","Soon","Right now"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,27377,1139,6634,988,768,5196,523,18,4023,31149,856,779,5940,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,32238,20,19,16,2119,730,527,535,17231,2166,20,21],"sequence_length":43,"marker_positions":[17,26,34],"positions":43,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-8.044438362121582,-6.318297863006592,-8.428503036499023],"probabilities_raw":[0.13698606193065643,0.7697144746780396,0.09329947084188461],"probabilities":[0.22158345580101013,0.6009449362754822,0.17747168242931366],"argmax_index":1,"argmax":"1","top2_margin":0.37936148047447205,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":0.9558883011341095,"confidence":0.6009449362754822,"probabilities":{"0":0.22158345580101013,"1":0.6009449362754822,"2":0.17747168242931366},"legend":{"0":"Can wait","1":"Soon","2":"Right now"}},"usage_input_tokens":142,"oracle_file":"records_ref_audio.json"},{"row_id":"card_batch_00/text/team","fixture_id":"card_batch_00","mode":"text","question_id":"team","question":{"type":"choice","instructions":"Which team should handle this?","criteria":{"billing":"Charges, refunds, invoices","technical":"App or site faults","fraud":"Suspected unauthorised use"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,15090,856,1727,61026,540,1311,953,2888,17980,523,18,28828,3410,1753,9936,1033,540,19,16,5363,800,535,61958,774,521,53467,592,521,60827,1864,20,19,16,10017,535,5135,933,3475,50022,20,19,16,24744,535,11631,7937,59006,2851,1517,20,21],"sequence_length":53,"marker_positions":[20,34,43],"positions":53,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-4.570059776306152,-1.822887659072876,-7.762394428253174],"probabilities_raw":[0.060097839683294296,0.937433660030365,0.0024685710668563843],"probabilities":[0.12167388945817947,0.865885853767395,0.012440279126167297],"argmax_index":1,"argmax":"technical","top2_margin":0.7442119643092155,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"technical","confidence":0.865885853767395,"probabilities":{"billing":0.12167388945817947,"technical":0.865885853767395,"fraud":0.012440279126167297}},"usage_input_tokens":53,"oracle_file":"records_ref.json"},{"row_id":"card_batch_01/text/team","fixture_id":"card_batch_01","mode":"text","question_id":"team","question":{"type":"choice","instructions":"Which team should handle this?","criteria":{"billing":"Charges, refunds, invoices","technical":"App or site faults","fraud":"Suspected unauthorised use"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,1495,49085,1403,859,2714,11063,523,18,28828,3410,1753,9936,1033,540,19,16,5363,800,535,61958,774,521,53467,592,521,60827,1864,20,19,16,10017,535,5135,933,3475,50022,20,19,16,24744,535,11631,7937,59006,2851,1517,20,21],"sequence_length":51,"marker_positions":[18,32,41],"positions":51,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-4.752628326416016,4.058452606201172,-4.859061241149902],"probabilities_raw":[0.00014902983093634248,0.9997169375419617,0.00013398309238255024],"probabilities":[0.0018406440503895283,0.9964535236358643,0.0017058887751773],"argmax_index":1,"argmax":"technical","top2_margin":0.9946128795854747,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"technical","confidence":0.9964535236358643,"probabilities":{"billing":0.0018406440503895283,"technical":0.9964535236358643,"fraud":0.0017058887751773}},"usage_input_tokens":51,"oracle_file":"records_ref.json"},{"row_id":"card_text/text/refund","fixture_id":"card_text","mode":"text","question_id":"refund","question":{"type":"noul","instructions":"Is the customer asking for a refund?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,550,953,15418,11126,1033,5805,521,8694,53467,1235,803,1377,523,18,4940,779,13608,14065,875,768,53467,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":47,"marker_positions":[25,37],"positions":47,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[-2.0755720138549805,8.723648071289062],"probabilities_raw":[2.0415003746165894e-05,0.9999796152114868],"probabilities":[0.998469889163971,0.001530130160972476],"argmax_index":0,"argmax":"true","top2_margin":0.9969397590029985,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.998469889163971},"usage_input_tokens":154,"oracle_file":"records_ref.json"},{"row_id":"card_text/text/team","fixture_id":"card_text","mode":"text","question_id":"team","question":{"type":"choice","instructions":"Which team should handle this?","criteria":{"billing":"Charges, refunds, invoices","technical":"App or site faults","fraud":"Suspected unauthorised use"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,550,953,15418,11126,1033,5805,521,8694,53467,1235,803,1377,523,18,28828,3410,1753,9936,1033,540,19,16,5363,800,535,61958,774,521,53467,592,521,60827,1864,20,19,16,10017,535,5135,933,3475,50022,20,19,16,24744,535,11631,7937,59006,2851,1517,20,21],"sequence_length":56,"marker_positions":[23,37,46],"positions":56,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[6.0775146484375,-1.2129172086715698,-2.832930088043213],"probabilities_raw":[0.999183714389801,0.0006814766093157232,0.00013486169336829334],"probabilities":[0.9928572177886963,0.005434439517557621,0.0017083368729799986],"argmax_index":0,"argmax":"billing","top2_margin":0.9874227782711387,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"billing","confidence":0.9928572177886963,"probabilities":{"billing":0.9928572177886963,"technical":0.005434439517557621,"fraud":0.0017083368729799986}},"usage_input_tokens":154,"oracle_file":"records_ref.json"},{"row_id":"card_text/text/urgency","fixture_id":"card_text","mode":"text","question_id":"urgency","question":{"type":"score","instructions":"How urgent is this?","criteria":["Can wait","Today","Blocking the customer now"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,550,953,15418,11126,1033,5805,521,8694,53467,1235,803,1377,523,18,4023,31149,856,1033,540,19,16,2119,730,525,535,2577,7498,20,19,16,2119,730,526,535,11268,20,19,16,2119,730,527,535,18578,800,779,13608,2166,20,21],"sequence_length":51,"marker_positions":[22,31,39],"positions":51,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-12.788542747497559,-6.237281322479248,-3.7804148197174072],"probabilities_raw":[0.00011273521522525698,0.07892896980047226,0.9209583401679993],"probabilities":[0.004393829498440027,0.19379673898220062,0.8018094301223755],"argmax_index":2,"argmax":"2","top2_margin":0.6080126911401749,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":1.7974155992269516,"confidence":0.8018094301223755,"probabilities":{"0":0.004393829498440027,"1":0.19379673898220062,"2":0.8018094301223755},"legend":{"0":"Can wait","1":"Today","2":"Blocking the customer now"}},"usage_input_tokens":154,"oracle_file":"records_ref.json"},{"row_id":"img_01/image/room","fixture_id":"img_01","mode":"image","question_id":"room","question":{"type":"choice","instructions":"What kind of room is this?","criteria":{"kitchen":"A kitchen","bedroom":"A bedroom","office":"An office","bathroom":"A bathroom"}},"type":"choice","qtype":0,"K":4,"media":["images/img_01.png"],"prefix_rows":144,"sequence_ids":[1,17,18,3493,3746,803,5362,856,1033,540,19,16,15936,535,835,15936,20,19,16,32874,535,835,32874,20,19,16,6756,535,1592,6756,20,19,16,32062,535,835,32062,20,21],"sequence_length":39,"marker_positions":[11,18,25,32],"positions":183,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-13.372673034667969,2.3188087940216064,-10.872926712036133,-5.046749114990234],"probabilities_raw":[1.5310895662423718e-07,0.9993657469749451,1.8647758679435356e-06,0.0006322711706161499],"probabilities":[1.5310895662423718e-07,0.9993657469749451,1.8647758679435356e-06,0.0006322711706161499],"argmax_index":1,"argmax":"bedroom","top2_margin":0.9987334758043289,"near_tie":false,"gold":"bedroom","official_answer":{"type":"choice","choice":"bedroom","confidence":0.9993657469749451,"probabilities":{"kitchen":1.5310895662423718e-07,"bedroom":0.9993657469749451,"office":1.8647758679435356e-06,"bathroom":0.0006322711706161499}},"usage_input_tokens":530,"oracle_file":"records_ref_images.json"},{"row_id":"img_01/image/suitcase","fixture_id":"img_01","mode":"image","question_id":"suitcase","question":{"type":"noul","instructions":"Is there a suitcase in the room?"},"type":"noul","qtype":2,"K":2,"media":["images/img_01.png"],"prefix_rows":144,"sequence_ids":[1,17,18,4940,1413,768,14503,10511,797,779,5362,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":25,"marker_positions":[13,19],"positions":169,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-3.17026948928833,11.013753890991211],"probabilities_raw":[6.91762181759259e-07,0.9999992847442627],"probabilities":[0.9999992847442627,6.91762181759259e-07],"argmax_index":0,"argmax":"true","top2_margin":0.9999985929820809,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.9999992847442627},"usage_input_tokens":530,"oracle_file":"records_ref_images.json"},{"row_id":"img_01/image/brightness","fixture_id":"img_01","mode":"image","question_id":"brightness","question":{"type":"score","instructions":"How bright is the room?","criteria":["Dark","Dim","Bright"]},"type":"score","qtype":1,"K":3,"media":["images/img_01.png"],"prefix_rows":144,"sequence_ids":[1,17,18,4023,9899,856,779,5362,540,19,16,2119,730,525,535,18003,20,19,16,2119,730,526,535,19640,20,19,16,2119,730,527,535,32989,20,21],"sequence_length":34,"marker_positions":[10,18,26],"positions":178,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-0.548412561416626,-6.016256332397461,-5.538527011871338],"probabilities_raw":[0.9890949130058289,0.004174298141151667,0.006730679422616959],"probabilities":[0.9890949130058289,0.004174298141151667,0.006730679422616959],"argmax_index":0,"argmax":"0","top2_margin":0.9823642335832119,"near_tie":false,"gold":"2","official_answer":{"type":"score","score":0.017635656986385584,"confidence":0.9890949130058289,"probabilities":{"0":0.9890949130058289,"1":0.004174298141151667,"2":0.006730679422616959},"legend":{"0":"Dark","1":"Dim","2":"Bright"}},"usage_input_tokens":530,"oracle_file":"records_ref_images.json"},{"row_id":"img_01/text/room","fixture_id":"img_01","mode":"text","question_id":"room","question":{"type":"choice","instructions":"What kind of room is this?","criteria":{"kitchen":"A kitchen","bedroom":"A bedroom","office":"An office","bathroom":"A bathroom"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,3493,3746,803,5362,856,1033,540,19,16,15936,535,835,15936,20,19,16,32874,535,835,32874,20,19,16,6756,535,1592,6756,20,19,16,32062,535,835,32062,20,21],"sequence_length":39,"marker_positions":[11,18,25,32],"positions":39,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.9235021471977234,1.1862629652023315,-0.5316015481948853,0.6049119234085083],"probabilities_raw":[0.0652020052075386,0.5376754999160767,0.09648530185222626,0.3006371855735779],"probabilities":[0.1018717959523201,0.4598037302494049,0.13478277623653412,0.3035416603088379],"argmax_index":1,"argmax":"bedroom","top2_margin":0.15626206994056702,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"bedroom","confidence":0.4598037302494049,"probabilities":{"kitchen":0.1018717959523201,"bedroom":0.4598037302494049,"office":0.13478277623653412,"bathroom":0.3035416603088379}},"usage_input_tokens":108,"oracle_file":"records_ref_images.json"},{"row_id":"img_01/text/suitcase","fixture_id":"img_01","mode":"text","question_id":"suitcase","question":{"type":"noul","instructions":"Is there a suitcase in the room?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4940,1413,768,14503,10511,797,779,5362,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":35,"marker_positions":[13,25],"positions":35,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[-0.22425425052642822,1.6848607063293457],"probabilities_raw":[0.12908032536506653,0.8709197044372559],"probabilities":[0.7587257027626038,0.24127434194087982],"argmax_index":0,"argmax":"true","top2_margin":0.5174513608217239,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.7587257027626038},"usage_input_tokens":108,"oracle_file":"records_ref_images.json"},{"row_id":"img_01/text/brightness","fixture_id":"img_01","mode":"text","question_id":"brightness","question":{"type":"score","instructions":"How bright is the room?","criteria":["Dark","Dim","Bright"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4023,9899,856,779,5362,540,19,16,2119,730,525,535,18003,20,19,16,2119,730,526,535,19640,20,19,16,2119,730,527,535,32989,20,21],"sequence_length":34,"marker_positions":[10,18,26],"positions":34,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-2.512366533279419,-0.007647328078746796,1.206966519355774],"probabilities_raw":[0.01835636794567108,0.2246842086315155,0.7569594383239746],"probabilities":[0.07227402925491333,0.30741074681282043,0.6203151941299438],"argmax_index":2,"argmax":"2","top2_margin":0.3129044473171234,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":1.5480411350727081,"confidence":0.6203151941299438,"probabilities":{"0":0.07227402925491333,"1":0.30741074681282043,"2":0.6203151941299438},"legend":{"0":"Dark","1":"Dim","2":"Bright"}},"usage_input_tokens":108,"oracle_file":"records_ref_images.json"},{"row_id":"img_02/image/room","fixture_id":"img_02","mode":"image","question_id":"room","question":{"type":"choice","instructions":"What kind of room is this?","criteria":{"kitchen":"A kitchen","bedroom":"A bedroom","office":"An office","bathroom":"A bathroom"}},"type":"choice","qtype":0,"K":4,"media":["images/img_02.png"],"prefix_rows":108,"sequence_ids":[1,17,26252,988,768,5362,3999,523,18,3493,3746,803,5362,856,1033,540,19,16,15936,535,835,15936,20,19,16,32874,535,835,32874,20,19,16,6756,535,1592,6756,20,19,16,32062,535,835,32062,20,21],"sequence_length":45,"marker_positions":[17,24,31,38],"positions":153,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-13.370351791381836,4.531220436096191,-10.323932647705078,-0.2855755388736725],"probabilities_raw":[1.6670359315185124e-08,0.9919719696044922,3.5074251059086237e-07,0.008027708157896996],"probabilities":[1.6670359315185124e-08,0.9919719696044922,3.5074251059086237e-07,0.008027708157896996],"argmax_index":1,"argmax":"bedroom","top2_margin":0.9839442614465952,"near_tie":false,"gold":"bedroom","official_answer":{"type":"choice","choice":"bedroom","confidence":0.9919719696044922,"probabilities":{"kitchen":1.6670359315185124e-08,"bedroom":0.9919719696044922,"office":3.5074251059086237e-07,"bathroom":0.008027708157896996}},"usage_input_tokens":437,"oracle_file":"records_ref_images.json"},{"row_id":"img_02/image/lamp","fixture_id":"img_02","mode":"image","question_id":"lamp","question":{"type":"noul","instructions":"Is a lamp switched on?"},"type":"noul","qtype":2,"K":2,"media":["images/img_02.png"],"prefix_rows":108,"sequence_ids":[1,17,26252,988,768,5362,3999,523,18,4940,768,29114,35589,884,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":28,"marker_positions":[16,22],"positions":136,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[1.247429609298706,-0.5311287641525269],"probabilities_raw":[0.8555187582969666,0.14448124170303345],"probabilities":[0.14448124170303345,0.8555187582969666],"argmax_index":1,"argmax":"false","top2_margin":0.7110375165939331,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.14448124170303345},"usage_input_tokens":437,"oracle_file":"records_ref_images.json"},{"row_id":"img_02/image/brightness","fixture_id":"img_02","mode":"image","question_id":"brightness","question":{"type":"score","instructions":"How bright is the room?","criteria":["Dark","Dim","Bright"]},"type":"score","qtype":1,"K":3,"media":["images/img_02.png"],"prefix_rows":108,"sequence_ids":[1,17,26252,988,768,5362,3999,523,18,4023,9899,856,779,5362,540,19,16,2119,730,525,535,18003,20,19,16,2119,730,526,535,19640,20,19,16,2119,730,527,535,32989,20,21],"sequence_length":40,"marker_positions":[16,24,32],"positions":148,"window":256,"window_r10":256,"temperature_key":null,"temperature":null,"raw_logits":[-2.5174143314361572,-0.759251594543457,-3.9761927127838135],"probabilities_raw":[0.1421608030796051,0.824783980846405,0.033055249601602554],"probabilities":[0.1421608030796051,0.824783980846405,0.033055249601602554],"argmax_index":1,"argmax":"1","top2_margin":0.6826231777667999,"near_tie":false,"gold":"1","official_answer":{"type":"score","score":0.8908944800496101,"confidence":0.824783980846405,"probabilities":{"0":0.1421608030796051,"1":0.824783980846405,"2":0.033055249601602554},"legend":{"0":"Dark","1":"Dim","2":"Bright"}},"usage_input_tokens":437,"oracle_file":"records_ref_images.json"},{"row_id":"img_02/text/room","fixture_id":"img_02","mode":"text","question_id":"room","question":{"type":"choice","instructions":"What kind of room is this?","criteria":{"kitchen":"A kitchen","bedroom":"A bedroom","office":"An office","bathroom":"A bathroom"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,26252,988,768,5362,3999,523,18,3493,3746,803,5362,856,1033,540,19,16,15936,535,835,15936,20,19,16,32874,535,835,32874,20,19,16,6756,535,1592,6756,20,19,16,32062,535,835,32062,20,21],"sequence_length":45,"marker_positions":[17,24,31,38],"positions":45,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.2342491149902344,1.2450541257858276,-1.7813855409622192,-1.1000628471374512],"probabilities_raw":[0.026233933866024017,0.8509531617164612,0.041260987520217896,0.0815519317984581],"probabilities":[0.0601099468767643,0.7216733694076538,0.0830691009759903,0.1351475715637207],"argmax_index":1,"argmax":"bedroom","top2_margin":0.5865257978439331,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"bedroom","confidence":0.7216733694076538,"probabilities":{"kitchen":0.0601099468767643,"bedroom":0.7216733694076538,"office":0.0830691009759903,"bathroom":0.1351475715637207}},"usage_input_tokens":123,"oracle_file":"records_ref_images.json"},{"row_id":"img_02/text/lamp","fixture_id":"img_02","mode":"text","question_id":"lamp","question":{"type":"noul","instructions":"Is a lamp switched on?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,26252,988,768,5362,3999,523,18,4940,768,29114,35589,884,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":38,"marker_positions":[16,28],"positions":38,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[1.744201421737671,-1.7224774360656738],"probabilities_raw":[0.9697246551513672,0.030275333672761917],"probabilities":[0.11101281642913818,0.8889872431755066],"argmax_index":1,"argmax":"false","top2_margin":0.7779744267463684,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.11101281642913818},"usage_input_tokens":123,"oracle_file":"records_ref_images.json"},{"row_id":"img_02/text/brightness","fixture_id":"img_02","mode":"text","question_id":"brightness","question":{"type":"score","instructions":"How bright is the room?","criteria":["Dark","Dim","Bright"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,26252,988,768,5362,3999,523,18,4023,9899,856,779,5362,540,19,16,2119,730,525,535,18003,20,19,16,2119,730,526,535,19640,20,19,16,2119,730,527,535,32989,20,21],"sequence_length":40,"marker_positions":[16,24,32],"positions":40,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-1.4011543989181519,0.5800229907035828,0.1053333580493927],"probabilities_raw":[0.0783567875623703,0.5681866407394409,0.3534564673900604],"probabilities":[0.15310442447662354,0.48117661476135254,0.3657189905643463],"argmax_index":1,"argmax":"1","top2_margin":0.11545762419700623,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":1.2126145958900452,"confidence":0.48117661476135254,"probabilities":{"0":0.15310442447662354,"1":0.48117661476135254,"2":0.3657189905643463},"legend":{"0":"Dark","1":"Dim","2":"Bright"}},"usage_input_tokens":123,"oracle_file":"records_ref_images.json"},{"row_id":"img_03/image/beds","fixture_id":"img_03","mode":"image","question_id":"beds","question":{"type":"choice","instructions":"How many beds are in the room?","criteria":{"one":"One bed","two":"Two beds","more":"Three or more beds"}},"type":"choice","qtype":0,"K":3,"media":["images/img_03.png"],"prefix_rows":1770,"sequence_ids":[1,17,18,4023,1697,25893,938,797,779,5362,540,19,16,1235,535,3732,6100,20,19,16,1547,535,8726,25893,20,19,16,1209,535,13016,933,1209,25893,20,21],"sequence_length":35,"marker_positions":[12,19,26],"positions":1805,"window":2048,"window_r10":2048,"temperature_key":null,"temperature":null,"raw_logits":[1.1830202341079712,3.3079018592834473,-7.577269554138184],"probabilities_raw":[0.10670007765293121,0.8932831883430481,1.673474434937816e-05],"probabilities":[0.10670007765293121,0.8932831883430481,1.673474434937816e-05],"argmax_index":1,"argmax":"two","top2_margin":0.7865831106901169,"near_tie":false,"gold":"two","official_answer":{"type":"choice","choice":"two","confidence":0.8932831883430481,"probabilities":{"one":0.10670007765293121,"two":0.8932831883430481,"more":1.673474434937816e-05}},"usage_input_tokens":5411,"oracle_file":"records_ref_images.json"},{"row_id":"img_03/image/towel","fixture_id":"img_03","mode":"image","question_id":"towel","question":{"type":"noul","instructions":"Is there a folded towel on a bed?"},"type":"noul","qtype":2,"K":2,"media":["images/img_03.png"],"prefix_rows":1770,"sequence_ids":[1,17,18,4940,1413,768,41478,50624,884,768,6100,540,19,16,4831,535,1295,20,19,16,3347,535,18672,20,21],"sequence_length":25,"marker_positions":[13,19],"positions":1795,"window":2048,"window_r10":2048,"temperature_key":null,"temperature":null,"raw_logits":[3.5223562717437744,-2.9612913131713867],"probabilities_raw":[0.9984740614891052,0.0015258945059031248],"probabilities":[0.0015258945059031248,0.9984740614891052],"argmax_index":1,"argmax":"false","top2_margin":0.9969481669832021,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.0015258945059031248},"usage_input_tokens":5411,"oracle_file":"records_ref_images.json"},{"row_id":"img_03/image/tidy","fixture_id":"img_03","mode":"image","question_id":"tidy","question":{"type":"score","instructions":"How tidy is the room?","criteria":["Very cluttered","Some clutter","Tidy"]},"type":"score","qtype":1,"K":3,"media":["images/img_03.png"],"prefix_rows":1770,"sequence_ids":[1,17,18,4023,18584,598,856,779,5362,540,19,16,2119,730,525,535,25913,1064,11119,791,20,19,16,2119,730,526,535,4260,1064,11119,20,19,16,2119,730,527,535,837,48307,20,21],"sequence_length":41,"marker_positions":[11,22,32],"positions":1811,"window":2048,"window_r10":2048,"temperature_key":null,"temperature":null,"raw_logits":[-4.784424304962158,-1.4155879020690918,3.0489602088928223],"probabilities_raw":[0.00039161930908448994,0.01137446891516447,0.9882339239120483],"probabilities":[0.00039161930908448994,0.01137446891516447,0.9882339239120483],"argmax_index":2,"argmax":"2","top2_margin":0.9768594549968839,"near_tie":false,"gold":"2","official_answer":{"type":"score","score":1.9878423167392612,"confidence":0.9882339239120483,"probabilities":{"0":0.00039161930908448994,"1":0.01137446891516447,"2":0.9882339239120483},"legend":{"0":"Very cluttered","1":"Some clutter","2":"Tidy"}},"usage_input_tokens":5411,"oracle_file":"records_ref_images.json"},{"row_id":"img_03/text/beds","fixture_id":"img_03","mode":"text","question_id":"beds","question":{"type":"choice","instructions":"How many beds are in the room?","criteria":{"one":"One bed","two":"Two beds","more":"Three or more beds"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4023,1697,25893,938,797,779,5362,540,19,16,1235,535,3732,6100,20,19,16,1547,535,8726,25893,20,19,16,1209,535,13016,933,1209,25893,20,21],"sequence_length":35,"marker_positions":[12,19,26],"positions":35,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.25700244307518005,0.2678792476654053,1.0923843383789062],"probabilities_raw":[0.1527809202671051,0.2582393288612366,0.5889797806739807],"probabilities":[0.19697225093841553,0.2865765392780304,0.5164511799812317],"argmax_index":2,"argmax":"more","top2_margin":0.2298746407032013,"near_tie":false,"gold":null,"official_answer":{"type":"choice","choice":"more","confidence":0.5164511799812317,"probabilities":{"one":0.19697225093841553,"two":0.2865765392780304,"more":0.5164511799812317}},"usage_input_tokens":111,"oracle_file":"records_ref_images.json"},{"row_id":"img_03/text/towel","fixture_id":"img_03","mode":"text","question_id":"towel","question":{"type":"noul","instructions":"Is there a folded towel on a bed?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4940,1413,768,41478,50624,884,768,6100,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":35,"marker_positions":[13,25],"positions":35,"window":64,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[-0.3175913095474243,7.188986301422119],"probabilities_raw":[0.0005491566262207925,0.9994508624076843],"probabilities":[0.9890658855438232,0.0109340725466609],"argmax_index":0,"argmax":"true","top2_margin":0.9781318129971623,"near_tie":false,"gold":null,"official_answer":{"type":"noul","noul":0.9890658855438232},"usage_input_tokens":111,"oracle_file":"records_ref_images.json"},{"row_id":"img_03/text/tidy","fixture_id":"img_03","mode":"text","question_id":"tidy","question":{"type":"score","instructions":"How tidy is the room?","criteria":["Very cluttered","Some clutter","Tidy"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,18,4023,18584,598,856,779,5362,540,19,16,2119,730,525,535,25913,1064,11119,791,20,19,16,2119,730,526,535,4260,1064,11119,20,19,16,2119,730,527,535,837,48307,20,21],"sequence_length":41,"marker_positions":[11,22,32],"positions":41,"window":64,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-3.0232226848602295,-0.7932671308517456,3.672762393951416],"probabilities_raw":[0.0012203309452161193,0.011348405852913857,0.9874312877655029],"probabilities":[0.01901734061539173,0.06901048868894577,0.911972165107727],"argmax_index":2,"argmax":"2","top2_margin":0.8429616764187813,"near_tie":false,"gold":null,"official_answer":{"type":"score","score":1.8929548189043999,"confidence":0.911972165107727,"probabilities":{"0":0.01901734061539173,"1":0.06901048868894577,"2":0.911972165107727},"legend":{"0":"Very cluttered","1":"Some clutter","2":"Tidy"}},"usage_input_tokens":111,"oracle_file":"records_ref_images.json"},{"row_id":"long_3400/text/component","fixture_id":"long_3400","mode":"text","question_id":"component","question":{"type":"choice","instructions":"Which component caused the stoppage?","criteria":{"scanner":"Barcode scanner","lane_7":"Conveyor lane 7","diverter":"Diverter arm","power":"Power supply","software":"Control software"}},"type":"choice","qtype":0,"K":5,"media":[],"prefix_rows":0,"sequence_ids":[1,17,15991,23707,3613,4425,4857,561,33371,12292,38620,1141,61026,17824,730,527,521,5008,1985,2647,8411,886,826,526,521,22694,3446,843,1513,31225,1015,27464,535,2960,843,6468,535,1056,522,1536,535,1056,518,767,52574,884,15774,535,1267,868,2290,767,12003,884,1868,535,4582,1055,509,6468,535,1056,535,1056,7174,17261,730,63688,730,10426,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,6135,535,1056,7174,17261,730,63688,730,11233,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,6468,535,1056,7174,17261,730,63688,730,10556,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,4905,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,4905,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3188,8061,1015,6468,535,1374,535,1056,7174,17261,730,63688,730,10173,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,1374,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,33118,3375,536,61026,3567,811,16687,43497,708,6468,535,1386,535,1056,7174,17261,730,63688,730,10520,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,1133,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,2197,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,1826,535,1056,7174,17261,730,63688,730,11233,530,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,2142,535,1056,7174,17261,730,63688,730,10520,532,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,2142,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,5489,8061,1015,6468,535,1664,535,1056,7174,17261,730,63688,730,8380,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,2625,535,1056,7174,17261,730,63688,730,10173,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,2625,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,30664,2108,536,61026,3567,811,16687,43497,708,6468,535,3024,535,1056,7174,17261,730,63688,730,10426,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,3188,535,1056,7174,17261,730,63688,730,8380,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,2108,535,1056,7174,17261,730,63688,730,11233,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,4536,535,1374,923,32535,730,43497,730,532,8323,7925,2835,730,533,523,534,835,843,35436,730,533,523,525,835,518,875,730,530,775,708,6468,535,4536,535,5046,923,32535,730,43497,730,532,8323,7925,2835,730,534,523,529,835,843,35436,730,533,523,525,835,518,875,730,1374,775,708,6468,535,4344,535,2108,923,32535,730,43497,730,532,23961,6586,730,526,523,5370,796,5266,843,2390,5723,730,526,523,2346,796,5266,1015,6468,535,4613,535,3654,59687,2850,43497,730,532,16017,12365,535,11697,38993,578,27493,532,522,528,26781,875,1209,1399,730,528,775,708,6468,535,4613,535,6468,59687,2850,8411,886,826,526,13525,968,6155,22694,843,7188,535,16017,43497,730,532,4032,1113,50027,23993,791,708,6468,535,4613,535,6468,7174,17261,730,1641,13053,19461,1189,60433,536,19331,1706,4908,884,24163,57506,708,6468,535,4613,535,2108,7174,17261,730,16753,28984,968,1267,868,2290,963,779,2669,26862,708,6468,535,3337,535,1386,7174,17261,730,12003,1868,9388,811,4582,1055,843,589,60088,730,526,1015,6468,535,4195,535,5843,7174,17261,730,49883,837,526,810,837,527,54126,843,2744,19331,1706,4032,2515,29067,7410,708,6468,535,3745,535,1664,7174,17261,730,24163,57506,963,730,2752,514,7614,708,6468,535,1838,535,1374,7174,17261,730,4582,1055,884,3475,963,43497,730,532,536,11023,17652,6944,811,43497,730,532,8323,708,6468,535,7058,535,3167,7174,17261,730,24163,57506,963,730,9731,514,7614,536,1641,13053,18483,730,528,810,730,529,8557,811,1361,19082,871,708,6468,535,3878,535,898,7174,17261,730,16017,3360,4468,968,4582,1055,535,38169,8227,772,28782,4563,1490,779,43497,730,532,23961,963,779,2341,46129,708,6468,535,5046,535,4536,7174,17261,730,8227,772,8365,988,43497,730,532,536,46129,909,532,522,527,2083,7937,521,1295,5896,1833,708,4972,535,1056,535,3654,7174,17261,730,43497,730,532,23961,17828,8744,4174,35036,536,32215,4984,708,4972,535,6135,535,1374,7174,17261,730,43497,730,532,23961,17828,23848,536,1927,2685,963,2920,6586,730,525,523,530,796,5266,875,730,2198,775,7410,708,4972,535,6326,535,1698,7174,17261,730,11023,17652,8365,988,43497,730,532,8323,708,4972,535,3654,535,1664,7174,17261,730,8411,886,826,526,2280,33480,968,1267,868,2290,536,878,4318,1047,730,2019,1750,730,1826,775,708,4972,535,3654,535,2999,7174,17261,730,1641,13053,19461,1189,49826,536,1641,13053,18483,730,528,810,730,529,840,59188,708,4972,535,6468,535,1056,7174,17261,730,43497,730,532,8323,7925,2835,730,531,523,526,835,843,35436,730,533,523,525,835,1015,4972,535,4905,535,1056,7174,17261,730,63688,730,4787,525,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,1374,535,1056,7174,17261,730,63688,730,3818,531,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,1386,535,1056,7174,17261,730,63688,730,4420,526,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4972,535,1386,535,3337,923,32535,730,49883,837,526,1295,49321,884,61026,730,32928,2476,536,61026,3567,811,16687,43497,708,4972,535,1133,535,1056,7174,17261,730,63688,730,6583,527,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2197,535,1056,7174,17261,730,63688,730,9887,533,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,2197,535,4905,7174,17261,730,24163,57506,1892,4174,730,1155,514,7614,708,4972,535,1826,535,1056,7174,17261,730,63688,730,10556,526,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4972,535,2142,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2142,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,534,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4972,535,1664,535,1056,7174,17261,730,63688,730,10426,530,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,2625,535,1056,7174,17261,730,8501,6634,968,1267,868,2290,535,43497,730,532,23961,17828,811,874,20425,2030,963,730,1155,535,1056,536,8227,772,42806,811,874,5689,1337,8233,20842,708,4972,535,2625,535,3654,7174,17261,730,8411,886,826,526,6840,12465,536,1295,5592,16753,592,708,4972,535,3024,535,1056,7174,17261,730,63688,730,8380,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2108,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4972,535,3337,535,1056,7174,17261,730,63688,730,11233,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4972,535,3337,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,4613,8061,1015,4972,535,3167,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,7058,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,4972,535,7058,535,1417,923,32535,730,49883,837,526,1295,49321,884,61026,730,22218,1532,536,61026,3567,811,16687,43497,708,4972,535,7568,535,1056,7174,17261,730,63688,730,10426,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,4528,535,1056,535,1056,7174,17261,730,63688,730,11233,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4528,535,1056,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,533,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4528,535,6326,535,1056,7174,17261,730,63688,730,10426,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,4528,535,4528,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4528,535,4528,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3756,8061,1015,4528,535,1374,535,1056,7174,17261,730,63688,730,10426,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4528,535,1374,535,1838,7174,17261,730,24163,57506,963,730,533,514,7614,708,4528,535,1443,535,1056,7174,17261,730,63688,730,11233,532,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4528,535,898,535,1056,7174,17261,730,63688,730,10520,530,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4528,535,898,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,28701,1155,536,61026,3567,811,16687,43497,708,4528,535,1826,535,1056,7174,17261,730,63688,730,8380,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,4528,535,2138,535,1056,7174,17261,730,63688,730,11233,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,4528,535,2925,535,1056,7174,17261,730,63688,730,10426,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4528,535,2925,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3878,8061,1015,4528,535,3024,535,1056,7174,17261,730,63688,730,8380,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,4528,535,2108,535,1056,7174,17261,730,63688,730,8380,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4528,535,2108,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,534,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4528,535,3337,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4528,535,3167,535,1056,7174,17261,730,63688,730,11233,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4528,535,3167,535,1417,923,32535,730,49883,837,526,1295,49321,884,61026,730,26465,4972,536,61026,3567,811,16687,43497,708,4528,535,7058,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4528,535,7568,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,4528,535,7568,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,2999,8061,1015,4905,535,1056,535,1056,7174,17261,730,63688,730,10426,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,4905,535,1056,535,1838,7174,17261,730,24163,57506,963,730,530,514,7614,708,4905,535,6326,535,1056,7174,17261,730,63688,730,11233,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4905,535,4528,535,1056,7174,17261,730,63688,730,10426,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,4905,535,1374,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4905,535,1443,535,1056,7174,17261,730,63688,730,10426,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4905,535,1443,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,32522,6326,536,61026,3567,811,16687,43497,708,4905,535,898,535,1056,7174,17261,730,63688,730,11233,532,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4905,535,898,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3092,8061,1015,4905,535,898,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,531,523,525,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4905,535,1826,535,1056,7174,17261,730,63688,730,10520,530,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4905,535,2138,535,1056,7174,17261,730,63688,730,8380,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,4905,535,2925,535,1056,7174,17261,730,63688,730,11233,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,4905,535,3024,535,1056,7174,17261,730,63688,730,10426,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4905,535,2108,535,1056,7174,17261,730,63688,730,8380,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,4905,535,3337,535,1056,7174,17261,730,63688,730,8380,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4905,535,3337,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,4613,8061,1015,4905,535,3337,535,1417,923,32535,730,49883,837,526,1295,49321,884,61026,730,33231,4541,536,61026,3567,811,16687,43497,708,4905,535,3167,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4905,535,3167,535,1838,7174,17261,730,24163,57506,963,730,532,514,7614,708,4905,535,7058,535,1056,7174,17261,730,63688,730,11233,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4905,535,7568,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,1155,535,1056,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,1155,535,1056,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,533,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,1155,535,1056,535,2108,7174,17261,730,43497,730,532,23961,17828,20425,968,4582,1055,843,802,13573,797,779,730,4972,535,2625,8501,6634,3271,2638,35036,536,1295,32215,4602,708,1155,535,6326,535,1056,7174,17261,730,63688,730,10426,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,1155,535,4528,535,1056,7174,17261,730,63688,730,11233,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,1155,535,4528,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3756,8061,1015,1155,535,1374,535,1056,7174,17261,730,63688,730,10426,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,1155,535,1374,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,24656,3361,536,61026,3567,811,16687,43497,708,1155,535,1443,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,1155,535,898,535,1056,7174,17261,730,63688,730,10426,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,1155,535,2197,535,3654,7174,17261,730,8411,886,826,526,6840,12465,536,1295,5592,16753,592,18,28828,8150,5457,779,878,4318,1047,540,19,16,49883,535,3492,3955,49883,20,19,16,43497,572,532,535,2020,4674,777,43497,730,532,20,19,16,49574,886,535,889,1991,886,5716,20,19,16,2410,535,9642,7018,20,19,16,5901,535,10478,5901,20,21],"sequence_length":3467,"marker_positions":[3422,3430,3443,3453,3460],"positions":3467,"window":4096,"window_r10":4096,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.666999101638794,-0.14830780029296875,0.5828009843826294,-1.5615222454071045,-2.4719622135162354],"probabilities_raw":[0.023023677989840508,0.2857777774333954,0.5936698317527771,0.06954680383205414,0.027981961145997047],"probabilities":[0.048573244363069534,0.29361486434936523,0.4949849545955658,0.10699225217103958,0.05583466589450836],"argmax_index":2,"argmax":"diverter","top2_margin":0.20137009024620056,"near_tie":false,"gold":"lane_7","official_answer":{"type":"choice","choice":"diverter","confidence":0.4949849545955658,"probabilities":{"scanner":0.048573244363069534,"lane_7":0.29361486434936523,"diverter":0.4949849545955658,"power":0.10699225217103958,"software":0.05583466589450836}},"usage_input_tokens":13867,"oracle_file":"records_ref.json"},{"row_id":"long_3400/text/normal_at_end","fixture_id":"long_3400","mode":"text","question_id":"normal_at_end","question":{"type":"noul","instructions":"Was the sorter running normally at the end of the log?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,15991,23707,3613,4425,4857,561,33371,12292,38620,1141,61026,17824,730,527,521,5008,1985,2647,8411,886,826,526,521,22694,3446,843,1513,31225,1015,27464,535,2960,843,6468,535,1056,522,1536,535,1056,518,767,52574,884,15774,535,1267,868,2290,767,12003,884,1868,535,4582,1055,509,6468,535,1056,535,1056,7174,17261,730,63688,730,10426,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,6135,535,1056,7174,17261,730,63688,730,11233,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,6468,535,1056,7174,17261,730,63688,730,10556,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,4905,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,4905,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3188,8061,1015,6468,535,1374,535,1056,7174,17261,730,63688,730,10173,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,1374,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,33118,3375,536,61026,3567,811,16687,43497,708,6468,535,1386,535,1056,7174,17261,730,63688,730,10520,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,1133,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,2197,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,1826,535,1056,7174,17261,730,63688,730,11233,530,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,2142,535,1056,7174,17261,730,63688,730,10520,532,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,2142,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,5489,8061,1015,6468,535,1664,535,1056,7174,17261,730,63688,730,8380,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,2625,535,1056,7174,17261,730,63688,730,10173,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,2625,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,30664,2108,536,61026,3567,811,16687,43497,708,6468,535,3024,535,1056,7174,17261,730,63688,730,10426,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,3188,535,1056,7174,17261,730,63688,730,8380,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,2108,535,1056,7174,17261,730,63688,730,11233,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,4536,535,1374,923,32535,730,43497,730,532,8323,7925,2835,730,533,523,534,835,843,35436,730,533,523,525,835,518,875,730,530,775,708,6468,535,4536,535,5046,923,32535,730,43497,730,532,8323,7925,2835,730,534,523,529,835,843,35436,730,533,523,525,835,518,875,730,1374,775,708,6468,535,4344,535,2108,923,32535,730,43497,730,532,23961,6586,730,526,523,5370,796,5266,843,2390,5723,730,526,523,2346,796,5266,1015,6468,535,4613,535,3654,59687,2850,43497,730,532,16017,12365,535,11697,38993,578,27493,532,522,528,26781,875,1209,1399,730,528,775,708,6468,535,4613,535,6468,59687,2850,8411,886,826,526,13525,968,6155,22694,843,7188,535,16017,43497,730,532,4032,1113,50027,23993,791,708,6468,535,4613,535,6468,7174,17261,730,1641,13053,19461,1189,60433,536,19331,1706,4908,884,24163,57506,708,6468,535,4613,535,2108,7174,17261,730,16753,28984,968,1267,868,2290,963,779,2669,26862,708,6468,535,3337,535,1386,7174,17261,730,12003,1868,9388,811,4582,1055,843,589,60088,730,526,1015,6468,535,4195,535,5843,7174,17261,730,49883,837,526,810,837,527,54126,843,2744,19331,1706,4032,2515,29067,7410,708,6468,535,3745,535,1664,7174,17261,730,24163,57506,963,730,2752,514,7614,708,6468,535,1838,535,1374,7174,17261,730,4582,1055,884,3475,963,43497,730,532,536,11023,17652,6944,811,43497,730,532,8323,708,6468,535,7058,535,3167,7174,17261,730,24163,57506,963,730,9731,514,7614,536,1641,13053,18483,730,528,810,730,529,8557,811,1361,19082,871,708,6468,535,3878,535,898,7174,17261,730,16017,3360,4468,968,4582,1055,535,38169,8227,772,28782,4563,1490,779,43497,730,532,23961,963,779,2341,46129,708,6468,535,5046,535,4536,7174,17261,730,8227,772,8365,988,43497,730,532,536,46129,909,532,522,527,2083,7937,521,1295,5896,1833,708,4972,535,1056,535,3654,7174,17261,730,43497,730,532,23961,17828,8744,4174,35036,536,32215,4984,708,4972,535,6135,535,1374,7174,17261,730,43497,730,532,23961,17828,23848,536,1927,2685,963,2920,6586,730,525,523,530,796,5266,875,730,2198,775,7410,708,4972,535,6326,535,1698,7174,17261,730,11023,17652,8365,988,43497,730,532,8323,708,4972,535,3654,535,1664,7174,17261,730,8411,886,826,526,2280,33480,968,1267,868,2290,536,878,4318,1047,730,2019,1750,730,1826,775,708,4972,535,3654,535,2999,7174,17261,730,1641,13053,19461,1189,49826,536,1641,13053,18483,730,528,810,730,529,840,59188,708,4972,535,6468,535,1056,7174,17261,730,43497,730,532,8323,7925,2835,730,531,523,526,835,843,35436,730,533,523,525,835,1015,4972,535,4905,535,1056,7174,17261,730,63688,730,4787,525,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,1374,535,1056,7174,17261,730,63688,730,3818,531,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,1386,535,1056,7174,17261,730,63688,730,4420,526,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4972,535,1386,535,3337,923,32535,730,49883,837,526,1295,49321,884,61026,730,32928,2476,536,61026,3567,811,16687,43497,708,4972,535,1133,535,1056,7174,17261,730,63688,730,6583,527,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2197,535,1056,7174,17261,730,63688,730,9887,533,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,2197,535,4905,7174,17261,730,24163,57506,1892,4174,730,1155,514,7614,708,4972,535,1826,535,1056,7174,17261,730,63688,730,10556,526,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4972,535,2142,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2142,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,534,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4972,535,1664,535,1056,7174,17261,730,63688,730,10426,530,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,2625,535,1056,7174,17261,730,8501,6634,968,1267,868,2290,535,43497,730,532,23961,17828,811,874,20425,2030,963,730,1155,535,1056,536,8227,772,42806,811,874,5689,1337,8233,20842,708,4972,535,2625,535,3654,7174,17261,730,8411,886,826,526,6840,12465,536,1295,5592,16753,592,708,4972,535,3024,535,1056,7174,17261,730,63688,730,8380,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2108,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4972,535,3337,535,1056,7174,17261,730,63688,730,11233,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4972,535,3337,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,4613,8061,1015,4972,535,3167,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,7058,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,4972,535,7058,535,1417,923,32535,730,49883,837,526,1295,49321,884,61026,730,22218,1532,536,61026,3567,811,16687,43497,708,4972,535,7568,535,1056,7174,17261,730,63688,730,10426,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,4528,535,1056,535,1056,7174,17261,730,63688,730,11233,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4528,535,1056,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,533,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4528,535,6326,535,1056,7174,17261,730,63688,730,10426,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,4528,535,4528,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4528,535,4528,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3756,8061,1015,4528,535,1374,535,1056,7174,17261,730,63688,730,10426,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4528,535,1374,535,1838,7174,17261,730,24163,57506,963,730,533,514,7614,708,4528,535,1443,535,1056,7174,17261,730,63688,730,11233,532,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4528,535,898,535,1056,7174,17261,730,63688,730,10520,530,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4528,535,898,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,28701,1155,536,61026,3567,811,16687,43497,708,4528,535,1826,535,1056,7174,17261,730,63688,730,8380,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,4528,535,2138,535,1056,7174,17261,730,63688,730,11233,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,4528,535,2925,535,1056,7174,17261,730,63688,730,10426,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4528,535,2925,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3878,8061,1015,4528,535,3024,535,1056,7174,17261,730,63688,730,8380,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,4528,535,2108,535,1056,7174,17261,730,63688,730,8380,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4528,535,2108,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,534,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4528,535,3337,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4528,535,3167,535,1056,7174,17261,730,63688,730,11233,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4528,535,3167,535,1417,923,32535,730,49883,837,526,1295,49321,884,61026,730,26465,4972,536,61026,3567,811,16687,43497,708,4528,535,7058,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4528,535,7568,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,4528,535,7568,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,2999,8061,1015,4905,535,1056,535,1056,7174,17261,730,63688,730,10426,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,4905,535,1056,535,1838,7174,17261,730,24163,57506,963,730,530,514,7614,708,4905,535,6326,535,1056,7174,17261,730,63688,730,11233,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4905,535,4528,535,1056,7174,17261,730,63688,730,10426,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,4905,535,1374,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4905,535,1443,535,1056,7174,17261,730,63688,730,10426,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4905,535,1443,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,32522,6326,536,61026,3567,811,16687,43497,708,4905,535,898,535,1056,7174,17261,730,63688,730,11233,532,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4905,535,898,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3092,8061,1015,4905,535,898,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,531,523,525,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4905,535,1826,535,1056,7174,17261,730,63688,730,10520,530,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4905,535,2138,535,1056,7174,17261,730,63688,730,8380,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,4905,535,2925,535,1056,7174,17261,730,63688,730,11233,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,4905,535,3024,535,1056,7174,17261,730,63688,730,10426,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4905,535,2108,535,1056,7174,17261,730,63688,730,8380,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,4905,535,3337,535,1056,7174,17261,730,63688,730,8380,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4905,535,3337,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,4613,8061,1015,4905,535,3337,535,1417,923,32535,730,49883,837,526,1295,49321,884,61026,730,33231,4541,536,61026,3567,811,16687,43497,708,4905,535,3167,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4905,535,3167,535,1838,7174,17261,730,24163,57506,963,730,532,514,7614,708,4905,535,7058,535,1056,7174,17261,730,63688,730,11233,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4905,535,7568,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,1155,535,1056,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,1155,535,1056,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,533,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,1155,535,1056,535,2108,7174,17261,730,43497,730,532,23961,17828,20425,968,4582,1055,843,802,13573,797,779,730,4972,535,2625,8501,6634,3271,2638,35036,536,1295,32215,4602,708,1155,535,6326,535,1056,7174,17261,730,63688,730,10426,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,1155,535,4528,535,1056,7174,17261,730,63688,730,11233,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,1155,535,4528,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3756,8061,1015,1155,535,1374,535,1056,7174,17261,730,63688,730,10426,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,1155,535,1374,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,24656,3361,536,61026,3567,811,16687,43497,708,1155,535,1443,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,1155,535,898,535,1056,7174,17261,730,63688,730,10426,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,1155,535,2197,535,3654,7174,17261,730,8411,886,826,526,6840,12465,536,1295,5592,16753,592,18,28519,779,8411,886,6840,12465,963,779,1764,803,779,3446,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":3449,"marker_positions":[3427,3439],"positions":3449,"window":4096,"window_r10":4096,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[0.9065542221069336,1.8131377696990967],"probabilities_raw":[0.28769946098327637,0.7123005390167236],"probabilities":[0.6327569484710693,0.36724305152893066],"argmax_index":0,"argmax":"true","top2_margin":0.26551389694213867,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.6327569484710693},"usage_input_tokens":13867,"oracle_file":"records_ref.json"},{"row_id":"long_3400/text/stoppage","fixture_id":"long_3400","mode":"text","question_id":"stoppage","question":{"type":"score","instructions":"How long was the sorter fully stopped?","criteria":["under 2 minutes","2 to 5 minutes","5 to 10 minutes","10 to 20 minutes","20 to 40 minutes","over 40 minutes"]},"type":"score","qtype":1,"K":6,"media":[],"prefix_rows":0,"sequence_ids":[1,17,15991,23707,3613,4425,4857,561,33371,12292,38620,1141,61026,17824,730,527,521,5008,1985,2647,8411,886,826,526,521,22694,3446,843,1513,31225,1015,27464,535,2960,843,6468,535,1056,522,1536,535,1056,518,767,52574,884,15774,535,1267,868,2290,767,12003,884,1868,535,4582,1055,509,6468,535,1056,535,1056,7174,17261,730,63688,730,10426,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,6135,535,1056,7174,17261,730,63688,730,11233,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,6468,535,1056,7174,17261,730,63688,730,10556,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,4905,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,4905,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3188,8061,1015,6468,535,1374,535,1056,7174,17261,730,63688,730,10173,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,1374,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,33118,3375,536,61026,3567,811,16687,43497,708,6468,535,1386,535,1056,7174,17261,730,63688,730,10520,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,1133,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,2197,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,1826,535,1056,7174,17261,730,63688,730,11233,530,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,2142,535,1056,7174,17261,730,63688,730,10520,532,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,2142,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,5489,8061,1015,6468,535,1664,535,1056,7174,17261,730,63688,730,8380,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,2625,535,1056,7174,17261,730,63688,730,10173,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,2625,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,30664,2108,536,61026,3567,811,16687,43497,708,6468,535,3024,535,1056,7174,17261,730,63688,730,10426,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,3188,535,1056,7174,17261,730,63688,730,8380,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,2108,535,1056,7174,17261,730,63688,730,11233,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,4536,535,1374,923,32535,730,43497,730,532,8323,7925,2835,730,533,523,534,835,843,35436,730,533,523,525,835,518,875,730,530,775,708,6468,535,4536,535,5046,923,32535,730,43497,730,532,8323,7925,2835,730,534,523,529,835,843,35436,730,533,523,525,835,518,875,730,1374,775,708,6468,535,4344,535,2108,923,32535,730,43497,730,532,23961,6586,730,526,523,5370,796,5266,843,2390,5723,730,526,523,2346,796,5266,1015,6468,535,4613,535,3654,59687,2850,43497,730,532,16017,12365,535,11697,38993,578,27493,532,522,528,26781,875,1209,1399,730,528,775,708,6468,535,4613,535,6468,59687,2850,8411,886,826,526,13525,968,6155,22694,843,7188,535,16017,43497,730,532,4032,1113,50027,23993,791,708,6468,535,4613,535,6468,7174,17261,730,1641,13053,19461,1189,60433,536,19331,1706,4908,884,24163,57506,708,6468,535,4613,535,2108,7174,17261,730,16753,28984,968,1267,868,2290,963,779,2669,26862,708,6468,535,3337,535,1386,7174,17261,730,12003,1868,9388,811,4582,1055,843,589,60088,730,526,1015,6468,535,4195,535,5843,7174,17261,730,49883,837,526,810,837,527,54126,843,2744,19331,1706,4032,2515,29067,7410,708,6468,535,3745,535,1664,7174,17261,730,24163,57506,963,730,2752,514,7614,708,6468,535,1838,535,1374,7174,17261,730,4582,1055,884,3475,963,43497,730,532,536,11023,17652,6944,811,43497,730,532,8323,708,6468,535,7058,535,3167,7174,17261,730,24163,57506,963,730,9731,514,7614,536,1641,13053,18483,730,528,810,730,529,8557,811,1361,19082,871,708,6468,535,3878,535,898,7174,17261,730,16017,3360,4468,968,4582,1055,535,38169,8227,772,28782,4563,1490,779,43497,730,532,23961,963,779,2341,46129,708,6468,535,5046,535,4536,7174,17261,730,8227,772,8365,988,43497,730,532,536,46129,909,532,522,527,2083,7937,521,1295,5896,1833,708,4972,535,1056,535,3654,7174,17261,730,43497,730,532,23961,17828,8744,4174,35036,536,32215,4984,708,4972,535,6135,535,1374,7174,17261,730,43497,730,532,23961,17828,23848,536,1927,2685,963,2920,6586,730,525,523,530,796,5266,875,730,2198,775,7410,708,4972,535,6326,535,1698,7174,17261,730,11023,17652,8365,988,43497,730,532,8323,708,4972,535,3654,535,1664,7174,17261,730,8411,886,826,526,2280,33480,968,1267,868,2290,536,878,4318,1047,730,2019,1750,730,1826,775,708,4972,535,3654,535,2999,7174,17261,730,1641,13053,19461,1189,49826,536,1641,13053,18483,730,528,810,730,529,840,59188,708,4972,535,6468,535,1056,7174,17261,730,43497,730,532,8323,7925,2835,730,531,523,526,835,843,35436,730,533,523,525,835,1015,4972,535,4905,535,1056,7174,17261,730,63688,730,4787,525,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,1374,535,1056,7174,17261,730,63688,730,3818,531,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,1386,535,1056,7174,17261,730,63688,730,4420,526,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4972,535,1386,535,3337,923,32535,730,49883,837,526,1295,49321,884,61026,730,32928,2476,536,61026,3567,811,16687,43497,708,4972,535,1133,535,1056,7174,17261,730,63688,730,6583,527,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2197,535,1056,7174,17261,730,63688,730,9887,533,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,2197,535,4905,7174,17261,730,24163,57506,1892,4174,730,1155,514,7614,708,4972,535,1826,535,1056,7174,17261,730,63688,730,10556,526,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4972,535,2142,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2142,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,534,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4972,535,1664,535,1056,7174,17261,730,63688,730,10426,530,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,2625,535,1056,7174,17261,730,8501,6634,968,1267,868,2290,535,43497,730,532,23961,17828,811,874,20425,2030,963,730,1155,535,1056,536,8227,772,42806,811,874,5689,1337,8233,20842,708,4972,535,2625,535,3654,7174,17261,730,8411,886,826,526,6840,12465,536,1295,5592,16753,592,708,4972,535,3024,535,1056,7174,17261,730,63688,730,8380,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2108,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4972,535,3337,535,1056,7174,17261,730,63688,730,11233,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4972,535,3337,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,4613,8061,1015,4972,535,3167,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,7058,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,4972,535,7058,535,1417,923,32535,730,49883,837,526,1295,49321,884,61026,730,22218,1532,536,61026,3567,811,16687,43497,708,4972,535,7568,535,1056,7174,17261,730,63688,730,10426,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,4528,535,1056,535,1056,7174,17261,730,63688,730,11233,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4528,535,1056,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,533,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4528,535,6326,535,1056,7174,17261,730,63688,730,10426,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,4528,535,4528,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4528,535,4528,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3756,8061,1015,4528,535,1374,535,1056,7174,17261,730,63688,730,10426,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4528,535,1374,535,1838,7174,17261,730,24163,57506,963,730,533,514,7614,708,4528,535,1443,535,1056,7174,17261,730,63688,730,11233,532,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4528,535,898,535,1056,7174,17261,730,63688,730,10520,530,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4528,535,898,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,28701,1155,536,61026,3567,811,16687,43497,708,4528,535,1826,535,1056,7174,17261,730,63688,730,8380,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,4528,535,2138,535,1056,7174,17261,730,63688,730,11233,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,4528,535,2925,535,1056,7174,17261,730,63688,730,10426,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4528,535,2925,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3878,8061,1015,4528,535,3024,535,1056,7174,17261,730,63688,730,8380,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,4528,535,2108,535,1056,7174,17261,730,63688,730,8380,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4528,535,2108,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,534,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4528,535,3337,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4528,535,3167,535,1056,7174,17261,730,63688,730,11233,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4528,535,3167,535,1417,923,32535,730,49883,837,526,1295,49321,884,61026,730,26465,4972,536,61026,3567,811,16687,43497,708,4528,535,7058,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4528,535,7568,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,4528,535,7568,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,2999,8061,1015,4905,535,1056,535,1056,7174,17261,730,63688,730,10426,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,4905,535,1056,535,1838,7174,17261,730,24163,57506,963,730,530,514,7614,708,4905,535,6326,535,1056,7174,17261,730,63688,730,11233,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4905,535,4528,535,1056,7174,17261,730,63688,730,10426,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,4905,535,1374,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4905,535,1443,535,1056,7174,17261,730,63688,730,10426,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4905,535,1443,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,32522,6326,536,61026,3567,811,16687,43497,708,4905,535,898,535,1056,7174,17261,730,63688,730,11233,532,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4905,535,898,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3092,8061,1015,4905,535,898,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,531,523,525,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4905,535,1826,535,1056,7174,17261,730,63688,730,10520,530,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4905,535,2138,535,1056,7174,17261,730,63688,730,8380,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,4905,535,2925,535,1056,7174,17261,730,63688,730,11233,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,4905,535,3024,535,1056,7174,17261,730,63688,730,10426,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4905,535,2108,535,1056,7174,17261,730,63688,730,8380,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,4905,535,3337,535,1056,7174,17261,730,63688,730,8380,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4905,535,3337,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,4613,8061,1015,4905,535,3337,535,1417,923,32535,730,49883,837,526,1295,49321,884,61026,730,33231,4541,536,61026,3567,811,16687,43497,708,4905,535,3167,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4905,535,3167,535,1838,7174,17261,730,24163,57506,963,730,532,514,7614,708,4905,535,7058,535,1056,7174,17261,730,63688,730,11233,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4905,535,7568,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,1155,535,1056,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,1155,535,1056,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,533,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,1155,535,1056,535,2108,7174,17261,730,43497,730,532,23961,17828,20425,968,4582,1055,843,802,13573,797,779,730,4972,535,2625,8501,6634,3271,2638,35036,536,1295,32215,4602,708,1155,535,6326,535,1056,7174,17261,730,63688,730,10426,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,1155,535,4528,535,1056,7174,17261,730,63688,730,11233,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,1155,535,4528,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3756,8061,1015,1155,535,1374,535,1056,7174,17261,730,63688,730,10426,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,1155,535,1374,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,24656,3361,536,61026,3567,811,16687,43497,708,1155,535,1443,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,1155,535,898,535,1056,7174,17261,730,63688,730,10426,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,1155,535,2197,535,3654,7174,17261,730,8411,886,826,526,6840,12465,536,1295,5592,16753,592,18,4023,1787,953,779,8411,886,7408,13525,540,19,16,2119,730,525,535,1490,730,527,5112,20,19,16,2119,730,526,535,730,527,811,730,530,5112,20,19,16,2119,730,527,535,730,530,811,730,1155,5112,20,19,16,2119,730,528,535,730,1155,811,730,898,5112,20,19,16,2119,730,529,535,730,898,811,730,2108,5112,20,19,16,2119,730,530,535,1391,730,2108,5112,20,21],"sequence_length":3497,"marker_positions":[3423,3434,3447,3460,3473,3486],"positions":3497,"window":4096,"window_r10":4096,"temperature_key":"score:6-10","temperature":1.0,"raw_logits":[-5.387547016143799,-3.613497495651245,-2.67159366607666,-1.864549160003662,0.024332530796527863,-3.945512294769287],"probabilities_raw":[0.00351888220757246,0.020742669701576233,0.05320204421877861,0.11924036592245102,0.7884137034416199,0.014882382936775684],"probabilities":[0.00351888220757246,0.020742669701576233,0.05320204421877861,0.11924036592245102,0.7884137034416199,0.014882382936775684],"argmax_index":4,"argmax":"4","top2_margin":0.6691733375191689,"near_tie":false,"gold":"4","official_answer":{"type":"score","score":3.7129345843568444,"confidence":0.7884137034416199,"probabilities":{"0":0.00351888220757246,"1":0.020742669701576233,"2":0.05320204421877861,"3":0.11924036592245102,"4":0.7884137034416199,"5":0.014882382936775684},"legend":{"0":"under 2 minutes","1":"2 to 5 minutes","2":"5 to 10 minutes","3":"10 to 20 minutes","4":"20 to 40 minutes","5":"over 40 minutes"}},"usage_input_tokens":13867,"oracle_file":"records_ref.json"},{"row_id":"long_3400/text/tension_10","fixture_id":"long_3400","mode":"text","question_id":"tension_10","question":{"type":"noul","instructions":"At the 10:00 check, was the lane 7 belt tension within specification?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,15991,23707,3613,4425,4857,561,33371,12292,38620,1141,61026,17824,730,527,521,5008,1985,2647,8411,886,826,526,521,22694,3446,843,1513,31225,1015,27464,535,2960,843,6468,535,1056,522,1536,535,1056,518,767,52574,884,15774,535,1267,868,2290,767,12003,884,1868,535,4582,1055,509,6468,535,1056,535,1056,7174,17261,730,63688,730,10426,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,6135,535,1056,7174,17261,730,63688,730,11233,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,6468,535,1056,7174,17261,730,63688,730,10556,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,4905,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,4905,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3188,8061,1015,6468,535,1374,535,1056,7174,17261,730,63688,730,10173,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,1374,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,33118,3375,536,61026,3567,811,16687,43497,708,6468,535,1386,535,1056,7174,17261,730,63688,730,10520,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,1133,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,2197,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,1826,535,1056,7174,17261,730,63688,730,11233,530,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,2142,535,1056,7174,17261,730,63688,730,10520,532,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,2142,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,5489,8061,1015,6468,535,1664,535,1056,7174,17261,730,63688,730,8380,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,2625,535,1056,7174,17261,730,63688,730,10173,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,2625,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,30664,2108,536,61026,3567,811,16687,43497,708,6468,535,3024,535,1056,7174,17261,730,63688,730,10426,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,3188,535,1056,7174,17261,730,63688,730,8380,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,2108,535,1056,7174,17261,730,63688,730,11233,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,4536,535,1374,923,32535,730,43497,730,532,8323,7925,2835,730,533,523,534,835,843,35436,730,533,523,525,835,518,875,730,530,775,708,6468,535,4536,535,5046,923,32535,730,43497,730,532,8323,7925,2835,730,534,523,529,835,843,35436,730,533,523,525,835,518,875,730,1374,775,708,6468,535,4344,535,2108,923,32535,730,43497,730,532,23961,6586,730,526,523,5370,796,5266,843,2390,5723,730,526,523,2346,796,5266,1015,6468,535,4613,535,3654,59687,2850,43497,730,532,16017,12365,535,11697,38993,578,27493,532,522,528,26781,875,1209,1399,730,528,775,708,6468,535,4613,535,6468,59687,2850,8411,886,826,526,13525,968,6155,22694,843,7188,535,16017,43497,730,532,4032,1113,50027,23993,791,708,6468,535,4613,535,6468,7174,17261,730,1641,13053,19461,1189,60433,536,19331,1706,4908,884,24163,57506,708,6468,535,4613,535,2108,7174,17261,730,16753,28984,968,1267,868,2290,963,779,2669,26862,708,6468,535,3337,535,1386,7174,17261,730,12003,1868,9388,811,4582,1055,843,589,60088,730,526,1015,6468,535,4195,535,5843,7174,17261,730,49883,837,526,810,837,527,54126,843,2744,19331,1706,4032,2515,29067,7410,708,6468,535,3745,535,1664,7174,17261,730,24163,57506,963,730,2752,514,7614,708,6468,535,1838,535,1374,7174,17261,730,4582,1055,884,3475,963,43497,730,532,536,11023,17652,6944,811,43497,730,532,8323,708,6468,535,7058,535,3167,7174,17261,730,24163,57506,963,730,9731,514,7614,536,1641,13053,18483,730,528,810,730,529,8557,811,1361,19082,871,708,6468,535,3878,535,898,7174,17261,730,16017,3360,4468,968,4582,1055,535,38169,8227,772,28782,4563,1490,779,43497,730,532,23961,963,779,2341,46129,708,6468,535,5046,535,4536,7174,17261,730,8227,772,8365,988,43497,730,532,536,46129,909,532,522,527,2083,7937,521,1295,5896,1833,708,4972,535,1056,535,3654,7174,17261,730,43497,730,532,23961,17828,8744,4174,35036,536,32215,4984,708,4972,535,6135,535,1374,7174,17261,730,43497,730,532,23961,17828,23848,536,1927,2685,963,2920,6586,730,525,523,530,796,5266,875,730,2198,775,7410,708,4972,535,6326,535,1698,7174,17261,730,11023,17652,8365,988,43497,730,532,8323,708,4972,535,3654,535,1664,7174,17261,730,8411,886,826,526,2280,33480,968,1267,868,2290,536,878,4318,1047,730,2019,1750,730,1826,775,708,4972,535,3654,535,2999,7174,17261,730,1641,13053,19461,1189,49826,536,1641,13053,18483,730,528,810,730,529,840,59188,708,4972,535,6468,535,1056,7174,17261,730,43497,730,532,8323,7925,2835,730,531,523,526,835,843,35436,730,533,523,525,835,1015,4972,535,4905,535,1056,7174,17261,730,63688,730,4787,525,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,1374,535,1056,7174,17261,730,63688,730,3818,531,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,1386,535,1056,7174,17261,730,63688,730,4420,526,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4972,535,1386,535,3337,923,32535,730,49883,837,526,1295,49321,884,61026,730,32928,2476,536,61026,3567,811,16687,43497,708,4972,535,1133,535,1056,7174,17261,730,63688,730,6583,527,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2197,535,1056,7174,17261,730,63688,730,9887,533,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,2197,535,4905,7174,17261,730,24163,57506,1892,4174,730,1155,514,7614,708,4972,535,1826,535,1056,7174,17261,730,63688,730,10556,526,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4972,535,2142,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2142,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,534,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4972,535,1664,535,1056,7174,17261,730,63688,730,10426,530,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,2625,535,1056,7174,17261,730,8501,6634,968,1267,868,2290,535,43497,730,532,23961,17828,811,874,20425,2030,963,730,1155,535,1056,536,8227,772,42806,811,874,5689,1337,8233,20842,708,4972,535,2625,535,3654,7174,17261,730,8411,886,826,526,6840,12465,536,1295,5592,16753,592,708,4972,535,3024,535,1056,7174,17261,730,63688,730,8380,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2108,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4972,535,3337,535,1056,7174,17261,730,63688,730,11233,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4972,535,3337,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,4613,8061,1015,4972,535,3167,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,7058,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,4972,535,7058,535,1417,923,32535,730,49883,837,526,1295,49321,884,61026,730,22218,1532,536,61026,3567,811,16687,43497,708,4972,535,7568,535,1056,7174,17261,730,63688,730,10426,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,4528,535,1056,535,1056,7174,17261,730,63688,730,11233,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4528,535,1056,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,533,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4528,535,6326,535,1056,7174,17261,730,63688,730,10426,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,4528,535,4528,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4528,535,4528,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3756,8061,1015,4528,535,1374,535,1056,7174,17261,730,63688,730,10426,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4528,535,1374,535,1838,7174,17261,730,24163,57506,963,730,533,514,7614,708,4528,535,1443,535,1056,7174,17261,730,63688,730,11233,532,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4528,535,898,535,1056,7174,17261,730,63688,730,10520,530,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4528,535,898,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,28701,1155,536,61026,3567,811,16687,43497,708,4528,535,1826,535,1056,7174,17261,730,63688,730,8380,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,4528,535,2138,535,1056,7174,17261,730,63688,730,11233,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,4528,535,2925,535,1056,7174,17261,730,63688,730,10426,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4528,535,2925,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3878,8061,1015,4528,535,3024,535,1056,7174,17261,730,63688,730,8380,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,4528,535,2108,535,1056,7174,17261,730,63688,730,8380,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4528,535,2108,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,534,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4528,535,3337,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4528,535,3167,535,1056,7174,17261,730,63688,730,11233,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4528,535,3167,535,1417,923,32535,730,49883,837,526,1295,49321,884,61026,730,26465,4972,536,61026,3567,811,16687,43497,708,4528,535,7058,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4528,535,7568,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,4528,535,7568,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,2999,8061,1015,4905,535,1056,535,1056,7174,17261,730,63688,730,10426,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,4905,535,1056,535,1838,7174,17261,730,24163,57506,963,730,530,514,7614,708,4905,535,6326,535,1056,7174,17261,730,63688,730,11233,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4905,535,4528,535,1056,7174,17261,730,63688,730,10426,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,4905,535,1374,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4905,535,1443,535,1056,7174,17261,730,63688,730,10426,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4905,535,1443,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,32522,6326,536,61026,3567,811,16687,43497,708,4905,535,898,535,1056,7174,17261,730,63688,730,11233,532,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4905,535,898,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3092,8061,1015,4905,535,898,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,531,523,525,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4905,535,1826,535,1056,7174,17261,730,63688,730,10520,530,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4905,535,2138,535,1056,7174,17261,730,63688,730,8380,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,4905,535,2925,535,1056,7174,17261,730,63688,730,11233,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,4905,535,3024,535,1056,7174,17261,730,63688,730,10426,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4905,535,2108,535,1056,7174,17261,730,63688,730,8380,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,4905,535,3337,535,1056,7174,17261,730,63688,730,8380,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4905,535,3337,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,4613,8061,1015,4905,535,3337,535,1417,923,32535,730,49883,837,526,1295,49321,884,61026,730,33231,4541,536,61026,3567,811,16687,43497,708,4905,535,3167,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,4905,535,3167,535,1838,7174,17261,730,24163,57506,963,730,532,514,7614,708,4905,535,7058,535,1056,7174,17261,730,63688,730,11233,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,527,12079,4905,535,7568,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,1155,535,1056,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,529,12079,1155,535,1056,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,533,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,1155,535,1056,535,2108,7174,17261,730,43497,730,532,23961,17828,20425,968,4582,1055,843,802,13573,797,779,730,4972,535,2625,8501,6634,3271,2638,35036,536,1295,32215,4602,708,1155,535,6326,535,1056,7174,17261,730,63688,730,10426,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,1155,535,4528,535,1056,7174,17261,730,63688,730,11233,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,1155,535,4528,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3756,8061,1015,1155,535,1374,535,1056,7174,17261,730,63688,730,10426,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,1155,535,1374,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,24656,3361,536,61026,3567,811,16687,43497,708,1155,535,1443,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,1155,535,898,535,1056,7174,17261,730,63688,730,10426,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,1155,535,2197,535,3654,7174,17261,730,8411,886,826,526,6840,12465,536,1295,5592,16753,592,18,5257,779,730,1155,535,1056,3999,521,953,779,43497,730,532,23961,17828,2638,35036,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":3454,"marker_positions":[3432,3444],"positions":3454,"window":4096,"window_r10":4096,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[0.22075581550598145,0.5800595879554749],"probabilities_raw":[0.41112810373306274,0.5888718366622925],"probabilities":[0.5536988377571106,0.4463012218475342],"argmax_index":0,"argmax":"true","top2_margin":0.10739761590957642,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.5536988377571106},"usage_input_tokens":13867,"oracle_file":"records_ref.json"},{"row_id":"own_L01/text/component","fixture_id":"own_L01","mode":"text","question_id":"component","question":{"type":"choice","instructions":"Which component caused the stoppage?","criteria":{"scanner":"Barcode scanner","lane_7":"Conveyor lane 7","diverter":"Diverter arm","power":"Power supply","software":"Control software"}},"type":"choice","qtype":0,"K":5,"media":[],"prefix_rows":0,"sequence_ids":[1,17,15991,23707,3613,4425,4857,561,33371,12292,38620,1141,61026,17824,730,527,521,5008,1985,2647,8411,886,826,526,521,22694,3446,843,1513,31225,1015,27464,535,2960,843,6468,535,1056,522,1536,535,1056,518,767,52574,884,15774,535,1267,868,2290,767,12003,884,1868,535,4582,1055,509,6468,535,1056,535,1056,7174,17261,730,63688,730,10426,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,6135,535,1056,7174,17261,730,63688,730,11233,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,6468,535,1056,7174,17261,730,63688,730,10556,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,4905,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,4905,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3188,8061,1015,6468,535,1374,535,1056,7174,17261,730,63688,730,10173,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,1374,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,33118,3375,536,61026,3567,811,16687,43497,708,6468,535,1386,535,1056,7174,17261,730,63688,730,10520,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,1133,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,2197,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,1826,535,1056,7174,17261,730,63688,730,11233,530,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,2142,535,1056,7174,17261,730,63688,730,10520,532,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,2142,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,5489,8061,1015,6468,535,1664,535,1056,7174,17261,730,63688,730,8380,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,2625,535,1056,7174,17261,730,63688,730,10173,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,2625,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,30664,2108,536,61026,3567,811,16687,43497,708,6468,535,3024,535,1056,7174,17261,730,63688,730,10426,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,3188,535,1056,7174,17261,730,63688,730,8380,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,2108,535,1056,7174,17261,730,63688,730,11233,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,4536,535,1374,923,32535,730,43497,730,532,8323,7925,2835,730,533,523,534,835,843,35436,730,533,523,525,835,518,875,730,530,775,708,6468,535,4536,535,5046,923,32535,730,43497,730,532,8323,7925,2835,730,534,523,529,835,843,35436,730,533,523,525,835,518,875,730,1374,775,708,6468,535,4344,535,2108,923,32535,730,43497,730,532,23961,6586,730,526,523,5370,796,5266,843,2390,5723,730,526,523,2346,796,5266,1015,6468,535,4613,535,3654,59687,2850,43497,730,532,16017,12365,535,11697,38993,578,27493,532,522,528,26781,875,1209,1399,730,528,775,708,6468,535,4613,535,6468,59687,2850,8411,886,826,526,13525,968,6155,22694,843,7188,535,16017,43497,730,532,4032,1113,50027,23993,791,708,6468,535,4613,535,6468,7174,17261,730,1641,13053,19461,1189,60433,536,19331,1706,4908,884,24163,57506,708,6468,535,4613,535,2108,7174,17261,730,16753,28984,968,1267,868,2290,963,779,2669,26862,708,6468,535,3337,535,1386,7174,17261,730,12003,1868,9388,811,4582,1055,843,589,60088,730,526,1015,6468,535,4195,535,5843,7174,17261,730,49883,837,526,810,837,527,54126,843,2744,19331,1706,4032,2515,29067,7410,708,6468,535,3745,535,1664,7174,17261,730,24163,57506,963,730,2752,514,7614,708,6468,535,1838,535,1374,7174,17261,730,4582,1055,884,3475,963,43497,730,532,536,11023,17652,6944,811,43497,730,532,8323,708,6468,535,7058,535,3167,7174,17261,730,24163,57506,963,730,9731,514,7614,536,1641,13053,18483,730,528,810,730,529,8557,811,1361,19082,871,708,6468,535,3878,535,898,7174,17261,730,16017,3360,4468,968,4582,1055,535,38169,8227,772,28782,4563,1490,779,43497,730,532,23961,963,779,2341,46129,708,6468,535,5046,535,4536,7174,17261,730,8227,772,8365,988,43497,730,532,536,46129,909,532,522,527,2083,7937,521,1295,5896,1833,708,4972,535,1056,535,3654,7174,17261,730,43497,730,532,23961,17828,8744,4174,35036,536,32215,4984,708,4972,535,6135,535,1374,7174,17261,730,43497,730,532,23961,17828,23848,536,1927,2685,963,2920,6586,730,525,523,530,796,5266,875,730,2198,775,7410,708,4972,535,6326,535,1698,7174,17261,730,11023,17652,8365,988,43497,730,532,8323,708,4972,535,3654,535,1664,7174,17261,730,8411,886,826,526,2280,33480,968,1267,868,2290,536,878,4318,1047,730,2019,1750,730,1826,775,708,4972,535,3654,535,2999,7174,17261,730,1641,13053,19461,1189,49826,536,1641,13053,18483,730,528,810,730,529,840,59188,708,4972,535,6468,535,1056,7174,17261,730,43497,730,532,8323,7925,2835,730,531,523,526,835,843,35436,730,533,523,525,835,1015,4972,535,4905,535,1056,7174,17261,730,63688,730,4787,525,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,1374,535,1056,7174,17261,730,63688,730,3818,531,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,1386,535,1056,7174,17261,730,63688,730,4420,526,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4972,535,1386,535,3337,923,32535,730,49883,837,526,1295,49321,884,61026,730,32928,2476,536,61026,3567,811,16687,43497,708,4972,535,1133,535,1056,7174,17261,730,63688,730,6583,527,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2197,535,1056,7174,17261,730,63688,730,9887,533,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,2197,535,4905,7174,17261,730,24163,57506,1892,4174,730,1155,514,7614,708,4972,535,1826,535,1056,7174,17261,730,63688,730,10556,526,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4972,535,2142,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2142,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,534,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4972,535,1664,535,1056,7174,17261,730,63688,730,10426,530,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,2625,535,1056,7174,17261,730,8501,6634,968,1267,868,2290,535,43497,730,532,23961,17828,811,874,20425,2030,963,730,1155,535,1056,536,8227,772,42806,811,874,5689,1337,8233,20842,708,4972,535,2625,535,3654,7174,17261,730,8411,886,826,526,6840,12465,536,1295,5592,16753,592,18,28828,8150,5457,779,878,4318,1047,540,19,16,49883,535,3492,3955,49883,20,19,16,43497,572,532,535,2020,4674,777,43497,730,532,20,19,16,49574,886,535,889,1991,886,5716,20,19,16,2410,535,9642,7018,20,19,16,5901,535,10478,5901,20,21],"sequence_length":1606,"marker_positions":[1561,1569,1582,1592,1599],"positions":1606,"window":2048,"window_r10":2048,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.840394139289856,1.6203656196594238,0.25618529319763184,-2.4982593059539795,-3.887787103652954],"probabilities_raw":[0.06271307170391083,0.7346024513244629,0.18775710463523865,0.011949683539569378,0.0029777754098176956],"probabilities":[0.10629419982433319,0.6164789795875549,0.23265084624290466,0.032522596418857574,0.012053357437252998],"argmax_index":1,"argmax":"lane_7","top2_margin":0.38382813334465027,"near_tie":false,"gold":"lane_7","official_answer":{"type":"choice","choice":"lane_7","confidence":0.6164789795875549,"probabilities":{"scanner":0.10629419982433319,"lane_7":0.6164789795875549,"diverter":0.23265084624290466,"power":0.032522596418857574,"software":0.012053357437252998}},"usage_input_tokens":4830,"oracle_file":"records_ref.json"},{"row_id":"own_L01/text/normal_at_end","fixture_id":"own_L01","mode":"text","question_id":"normal_at_end","question":{"type":"noul","instructions":"Was the sorter running normally at the end of the log?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,15991,23707,3613,4425,4857,561,33371,12292,38620,1141,61026,17824,730,527,521,5008,1985,2647,8411,886,826,526,521,22694,3446,843,1513,31225,1015,27464,535,2960,843,6468,535,1056,522,1536,535,1056,518,767,52574,884,15774,535,1267,868,2290,767,12003,884,1868,535,4582,1055,509,6468,535,1056,535,1056,7174,17261,730,63688,730,10426,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,6135,535,1056,7174,17261,730,63688,730,11233,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,6468,535,1056,7174,17261,730,63688,730,10556,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,4905,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,4905,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3188,8061,1015,6468,535,1374,535,1056,7174,17261,730,63688,730,10173,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,1374,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,33118,3375,536,61026,3567,811,16687,43497,708,6468,535,1386,535,1056,7174,17261,730,63688,730,10520,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,1133,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,2197,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,1826,535,1056,7174,17261,730,63688,730,11233,530,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,2142,535,1056,7174,17261,730,63688,730,10520,532,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,2142,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,5489,8061,1015,6468,535,1664,535,1056,7174,17261,730,63688,730,8380,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,2625,535,1056,7174,17261,730,63688,730,10173,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,2625,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,30664,2108,536,61026,3567,811,16687,43497,708,6468,535,3024,535,1056,7174,17261,730,63688,730,10426,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,3188,535,1056,7174,17261,730,63688,730,8380,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,2108,535,1056,7174,17261,730,63688,730,11233,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,4536,535,1374,923,32535,730,43497,730,532,8323,7925,2835,730,533,523,534,835,843,35436,730,533,523,525,835,518,875,730,530,775,708,6468,535,4536,535,5046,923,32535,730,43497,730,532,8323,7925,2835,730,534,523,529,835,843,35436,730,533,523,525,835,518,875,730,1374,775,708,6468,535,4344,535,2108,923,32535,730,43497,730,532,23961,6586,730,526,523,5370,796,5266,843,2390,5723,730,526,523,2346,796,5266,1015,6468,535,4613,535,3654,59687,2850,43497,730,532,16017,12365,535,11697,38993,578,27493,532,522,528,26781,875,1209,1399,730,528,775,708,6468,535,4613,535,6468,59687,2850,8411,886,826,526,13525,968,6155,22694,843,7188,535,16017,43497,730,532,4032,1113,50027,23993,791,708,6468,535,4613,535,6468,7174,17261,730,1641,13053,19461,1189,60433,536,19331,1706,4908,884,24163,57506,708,6468,535,4613,535,2108,7174,17261,730,16753,28984,968,1267,868,2290,963,779,2669,26862,708,6468,535,3337,535,1386,7174,17261,730,12003,1868,9388,811,4582,1055,843,589,60088,730,526,1015,6468,535,4195,535,5843,7174,17261,730,49883,837,526,810,837,527,54126,843,2744,19331,1706,4032,2515,29067,7410,708,6468,535,3745,535,1664,7174,17261,730,24163,57506,963,730,2752,514,7614,708,6468,535,1838,535,1374,7174,17261,730,4582,1055,884,3475,963,43497,730,532,536,11023,17652,6944,811,43497,730,532,8323,708,6468,535,7058,535,3167,7174,17261,730,24163,57506,963,730,9731,514,7614,536,1641,13053,18483,730,528,810,730,529,8557,811,1361,19082,871,708,6468,535,3878,535,898,7174,17261,730,16017,3360,4468,968,4582,1055,535,38169,8227,772,28782,4563,1490,779,43497,730,532,23961,963,779,2341,46129,708,6468,535,5046,535,4536,7174,17261,730,8227,772,8365,988,43497,730,532,536,46129,909,532,522,527,2083,7937,521,1295,5896,1833,708,4972,535,1056,535,3654,7174,17261,730,43497,730,532,23961,17828,8744,4174,35036,536,32215,4984,708,4972,535,6135,535,1374,7174,17261,730,43497,730,532,23961,17828,23848,536,1927,2685,963,2920,6586,730,525,523,530,796,5266,875,730,2198,775,7410,708,4972,535,6326,535,1698,7174,17261,730,11023,17652,8365,988,43497,730,532,8323,708,4972,535,3654,535,1664,7174,17261,730,8411,886,826,526,2280,33480,968,1267,868,2290,536,878,4318,1047,730,2019,1750,730,1826,775,708,4972,535,3654,535,2999,7174,17261,730,1641,13053,19461,1189,49826,536,1641,13053,18483,730,528,810,730,529,840,59188,708,4972,535,6468,535,1056,7174,17261,730,43497,730,532,8323,7925,2835,730,531,523,526,835,843,35436,730,533,523,525,835,1015,4972,535,4905,535,1056,7174,17261,730,63688,730,4787,525,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,1374,535,1056,7174,17261,730,63688,730,3818,531,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,1386,535,1056,7174,17261,730,63688,730,4420,526,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4972,535,1386,535,3337,923,32535,730,49883,837,526,1295,49321,884,61026,730,32928,2476,536,61026,3567,811,16687,43497,708,4972,535,1133,535,1056,7174,17261,730,63688,730,6583,527,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2197,535,1056,7174,17261,730,63688,730,9887,533,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,2197,535,4905,7174,17261,730,24163,57506,1892,4174,730,1155,514,7614,708,4972,535,1826,535,1056,7174,17261,730,63688,730,10556,526,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4972,535,2142,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2142,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,534,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4972,535,1664,535,1056,7174,17261,730,63688,730,10426,530,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,2625,535,1056,7174,17261,730,8501,6634,968,1267,868,2290,535,43497,730,532,23961,17828,811,874,20425,2030,963,730,1155,535,1056,536,8227,772,42806,811,874,5689,1337,8233,20842,708,4972,535,2625,535,3654,7174,17261,730,8411,886,826,526,6840,12465,536,1295,5592,16753,592,18,28519,779,8411,886,6840,12465,963,779,1764,803,779,3446,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":1588,"marker_positions":[1566,1578],"positions":1588,"window":2048,"window_r10":2048,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[0.6751614809036255,1.5158710479736328],"probabilities_raw":[0.3013853430747986,0.6986145973205566],"probabilities":[0.6235232949256897,0.3764767050743103],"argmax_index":0,"argmax":"true","top2_margin":0.2470465898513794,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.6235232949256897},"usage_input_tokens":4830,"oracle_file":"records_ref.json"},{"row_id":"own_L01/text/stoppage","fixture_id":"own_L01","mode":"text","question_id":"stoppage","question":{"type":"score","instructions":"How long was the sorter fully stopped?","criteria":["under 2 minutes","2 to 5 minutes","5 to 10 minutes","10 to 20 minutes","20 to 40 minutes","over 40 minutes"]},"type":"score","qtype":1,"K":6,"media":[],"prefix_rows":0,"sequence_ids":[1,17,15991,23707,3613,4425,4857,561,33371,12292,38620,1141,61026,17824,730,527,521,5008,1985,2647,8411,886,826,526,521,22694,3446,843,1513,31225,1015,27464,535,2960,843,6468,535,1056,522,1536,535,1056,518,767,52574,884,15774,535,1267,868,2290,767,12003,884,1868,535,4582,1055,509,6468,535,1056,535,1056,7174,17261,730,63688,730,10426,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,6135,535,1056,7174,17261,730,63688,730,11233,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,6468,535,1056,7174,17261,730,63688,730,10556,526,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,4905,535,1056,7174,17261,730,63688,730,8380,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,4905,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,3188,8061,1015,6468,535,1374,535,1056,7174,17261,730,63688,730,10173,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,1374,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,33118,3375,536,61026,3567,811,16687,43497,708,6468,535,1386,535,1056,7174,17261,730,63688,730,10520,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,1133,535,1056,7174,17261,730,63688,730,10426,531,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,2197,535,1056,7174,17261,730,63688,730,8380,527,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,1826,535,1056,7174,17261,730,63688,730,11233,530,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,2142,535,1056,7174,17261,730,63688,730,10520,532,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,2142,535,4536,7174,17261,730,49574,886,889,529,41172,3999,7410,843,22500,730,525,523,5489,8061,1015,6468,535,1664,535,1056,7174,17261,730,63688,730,8380,534,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,530,12079,6468,535,2625,535,1056,7174,17261,730,63688,730,10173,528,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,532,12079,6468,535,2625,535,1417,923,32535,730,49883,837,527,1295,49321,884,61026,730,30664,2108,536,61026,3567,811,16687,43497,708,6468,535,3024,535,1056,7174,17261,730,63688,730,10426,533,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,526,12079,6468,535,3188,535,1056,7174,17261,730,63688,730,8380,525,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,2108,535,1056,7174,17261,730,63688,730,11233,529,19331,1706,10012,536,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,6468,535,4536,535,1374,923,32535,730,43497,730,532,8323,7925,2835,730,533,523,534,835,843,35436,730,533,523,525,835,518,875,730,530,775,708,6468,535,4536,535,5046,923,32535,730,43497,730,532,8323,7925,2835,730,534,523,529,835,843,35436,730,533,523,525,835,518,875,730,1374,775,708,6468,535,4344,535,2108,923,32535,730,43497,730,532,23961,6586,730,526,523,5370,796,5266,843,2390,5723,730,526,523,2346,796,5266,1015,6468,535,4613,535,3654,59687,2850,43497,730,532,16017,12365,535,11697,38993,578,27493,532,522,528,26781,875,1209,1399,730,528,775,708,6468,535,4613,535,6468,59687,2850,8411,886,826,526,13525,968,6155,22694,843,7188,535,16017,43497,730,532,4032,1113,50027,23993,791,708,6468,535,4613,535,6468,7174,17261,730,1641,13053,19461,1189,60433,536,19331,1706,4908,884,24163,57506,708,6468,535,4613,535,2108,7174,17261,730,16753,28984,968,1267,868,2290,963,779,2669,26862,708,6468,535,3337,535,1386,7174,17261,730,12003,1868,9388,811,4582,1055,843,589,60088,730,526,1015,6468,535,4195,535,5843,7174,17261,730,49883,837,526,810,837,527,54126,843,2744,19331,1706,4032,2515,29067,7410,708,6468,535,3745,535,1664,7174,17261,730,24163,57506,963,730,2752,514,7614,708,6468,535,1838,535,1374,7174,17261,730,4582,1055,884,3475,963,43497,730,532,536,11023,17652,6944,811,43497,730,532,8323,708,6468,535,7058,535,3167,7174,17261,730,24163,57506,963,730,9731,514,7614,536,1641,13053,18483,730,528,810,730,529,8557,811,1361,19082,871,708,6468,535,3878,535,898,7174,17261,730,16017,3360,4468,968,4582,1055,535,38169,8227,772,28782,4563,1490,779,43497,730,532,23961,963,779,2341,46129,708,6468,535,5046,535,4536,7174,17261,730,8227,772,8365,988,43497,730,532,536,46129,909,532,522,527,2083,7937,521,1295,5896,1833,708,4972,535,1056,535,3654,7174,17261,730,43497,730,532,23961,17828,8744,4174,35036,536,32215,4984,708,4972,535,6135,535,1374,7174,17261,730,43497,730,532,23961,17828,23848,536,1927,2685,963,2920,6586,730,525,523,530,796,5266,875,730,2198,775,7410,708,4972,535,6326,535,1698,7174,17261,730,11023,17652,8365,988,43497,730,532,8323,708,4972,535,3654,535,1664,7174,17261,730,8411,886,826,526,2280,33480,968,1267,868,2290,536,878,4318,1047,730,2019,1750,730,1826,775,708,4972,535,3654,535,2999,7174,17261,730,1641,13053,19461,1189,49826,536,1641,13053,18483,730,528,810,730,529,840,59188,708,4972,535,6468,535,1056,7174,17261,730,43497,730,532,8323,7925,2835,730,531,523,526,835,843,35436,730,533,523,525,835,1015,4972,535,4905,535,1056,7174,17261,730,63688,730,4787,525,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,1374,535,1056,7174,17261,730,63688,730,3818,531,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,1386,535,1056,7174,17261,730,63688,730,4420,526,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4972,535,1386,535,3337,923,32535,730,49883,837,526,1295,49321,884,61026,730,32928,2476,536,61026,3567,811,16687,43497,708,4972,535,1133,535,1056,7174,17261,730,63688,730,6583,527,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2197,535,1056,7174,17261,730,63688,730,9887,533,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,2197,535,4905,7174,17261,730,24163,57506,1892,4174,730,1155,514,7614,708,4972,535,1826,535,1056,7174,17261,730,63688,730,10556,526,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,531,12079,4972,535,2142,535,1056,7174,17261,730,63688,730,10520,527,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,525,12079,4972,535,2142,535,1664,7174,17261,730,43497,730,532,8323,7925,2835,730,530,523,534,835,843,35436,730,533,523,525,835,4032,23961,6586,730,526,523,2346,796,5266,708,4972,535,1664,535,1056,7174,17261,730,63688,730,10426,530,19331,1706,10012,843,3321,3600,38689,4032,50027,730,526,522,1374,5592,536,49883,2062,4002,730,2962,523,528,12079,4972,535,2625,535,1056,7174,17261,730,8501,6634,968,1267,868,2290,535,43497,730,532,23961,17828,811,874,20425,2030,963,730,1155,535,1056,536,8227,772,42806,811,874,5689,1337,8233,20842,708,4972,535,2625,535,3654,7174,17261,730,8411,886,826,526,6840,12465,536,1295,5592,16753,592,18,4023,1787,953,779,8411,886,7408,13525,540,19,16,2119,730,525,535,1490,730,527,5112,20,19,16,2119,730,526,535,730,527,811,730,530,5112,20,19,16,2119,730,527,535,730,530,811,730,1155,5112,20,19,16,2119,730,528,535,730,1155,811,730,898,5112,20,19,16,2119,730,529,535,730,898,811,730,2108,5112,20,19,16,2119,730,530,535,1391,730,2108,5112,20,21],"sequence_length":1636,"marker_positions":[1562,1573,1586,1599,1612,1625],"positions":1636,"window":2048,"window_r10":2048,"temperature_key":"score:6-10","temperature":1.0,"raw_logits":[-6.378607273101807,-3.3338048458099365,-2.862976551055908,-1.375028133392334,0.9950432181358337,-0.7478825449943542],"probabilities_raw":[0.00048148856149055064,0.010114093311131,0.01619589701294899,0.07171545922756195,0.7672231793403625,0.13426990807056427],"probabilities":[0.00048148856149055064,0.010114093311131,0.01619589701294899,0.07171545922756195,0.7672231793403625,0.13426990807056427],"argmax_index":4,"argmax":"4","top2_margin":0.6329532712697983,"near_tie":false,"gold":"4","official_answer":{"type":"score","score":3.9978945227339864,"confidence":0.7672231793403625,"probabilities":{"0":0.00048148856149055064,"1":0.010114093311131,"2":0.01619589701294899,"3":0.07171545922756195,"4":0.7672231793403625,"5":0.13426990807056427},"legend":{"0":"under 2 minutes","1":"2 to 5 minutes","2":"5 to 10 minutes","3":"10 to 20 minutes","4":"20 to 40 minutes","5":"over 40 minutes"}},"usage_input_tokens":4830,"oracle_file":"records_ref.json"},{"row_id":"own_L03/text/fee_motion","fixture_id":"own_L03","mode":"text","question_id":"fee_motion","question":{"type":"choice","instructions":"What happened to the motion on the annual fee?","criteria":{"approved":null,"rejected":null,"deferred":"Postponed to a later meeting or a working group","withdrawn":"Withdrawn before the vote"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,564,17769,3613,4425,12411,34632,2730,41891,516,31289,7272,550,10877,708,13635,3962,803,779,27845,5525,35590,4908,884,22045,730,1374,4690,730,1718,531,963,730,913,535,1664,797,33864,54898,2176,24579,7329,509,41678,535,730,3745,3967,843,1615,959,1143,13975,12930,518,810,730,528,24753,521,3136,1377,779,29780,12548,50804,777,521,2100,3852,1543,523,1030,779,12377,535,918,54069,523,3617,3962,535,9413,598,19718,843,52709,1126,1635,10412,6650,535,730,1632,3967,3567,1149,1514,6650,523,509,526,523,56975,810,967,24339,708,1098,12377,41975,3967,810,24753,810,14954,791,779,11917,15671,875,779,1517,803,779,5362,523,941,12778,6835,730,1664,3967,811,874,2251,875,768,3177,8324,811,874,967,777,931,523,3877,730,3745,3967,2251,521,779,8324,953,13096,967,777,931,523,509,527,523,3617,3962,803,779,2659,8900,3177,8324,708,1098,5112,803,779,8324,4908,884,730,1532,4690,730,1718,530,1166,12043,906,768,3347,3084,521,7885,968,9213,833,810,2341,791,968,28682,523,2479,1166,1295,14332,35517,896,1166,1014,4001,884,779,26168,523,509,528,523,21640,1090,2312,708,548,54069,17422,2851,779,1423,523,941,10039,9657,779,6844,7405,521,1144,9388,730,527,521,9300,875,779,11917,12562,3549,521,810,35130,1547,27297,45546,3256,779,8008,3509,521,1895,916,1209,1399,11583,20470,523,941,7982,875,768,55097,22212,884,8995,27784,6806,521,810,779,17677,1178,2166,11860,811,6715,1230,768,11013,8153,797,3354,523,49427,5313,14118,988,730,18833,811,730,22989,16671,523,1405,14954,791,779,15671,810,779,20470,1316,5514,779,38666,811,1973,3622,2759,3217,768,1423,523,509,529,523,9312,802,9233,1090,2312,708,1098,1230,9932,3185,802,9233,521,2345,586,521,6892,779,12324,875,779,1423,523,53813,953,730,531,521,21957,535,22398,23023,730,527,521,26547,521,779,6844,7405,730,527,521,9300,810,41634,730,526,521,30542,523,2315,2124,11761,953,730,529,521,25803,521,8787,38666,17236,843,526,521,21405,1109,12235,843,23617,1109,11917,33265,843,7537,518,810,7494,875,779,8008,3509,5158,6688,843,526,521,26668,1191,941,1423,8557,916,768,41938,803,730,526,521,32647,810,768,7493,8386,803,730,534,521,19222,523,941,12324,1320,1381,20425,968,902,7054,1924,5725,521,1316,1833,1295,4662,523,32826,12043,779,12324,968,768,1914,803,6108,521,916,1295,20402,2723,523,509,530,523,32964,988,779,3822,3892,708,26691,2514,843,549,37986,3271,779,2514,21440,730,5411,8267,35304,884,779,1375,977,5329,779,2254,810,50398,1547,803,779,2759,5854,20842,884,779,5253,523,1311,1499,1672,811,8153,779,8008,3509,875,21275,4476,797,779,6844,810,856,5400,875,1547,1209,20470,819,30193,2514,843,545,6310,793,3271,29504,779,6844,7405,521,779,2514,9657,768,30780,4915,797,4843,896,17322,1919,523,835,3316,4915,797,779,11917,856,13573,875,4471,1288,768,35563,1011,874,20560,963,768,16336,7264,819,35798,4977,810,5600,2514,843,549,831,833,3271,10325,988,779,22212,7982,521,779,2514,4468,2204,15073,11513,15489,810,768,26781,15761,884,13878,9690,811,779,17677,536,1547,803,779,15489,1052,1381,50398,523,509,531,523,7329,12562,3549,708,559,4182,4468,875,779,11917,15671,896,779,12562,12940,1178,1381,29878,963,730,1133,521,6019,523,3877,779,730,527,521,9300,988,779,6844,7405,521,779,3549,2166,12973,963,730,1632,521,14671,523,941,15671,1178,6944,811,1547,56302,8810,592,875,779,2280,810,45777,1149,13041,968,779,1764,803,3354,523,6162,4584,2155,2076,4526,1929,30541,1232,1000,797,779,12562,3494,2013,779,6844,521,810,902,8535,8923,1188,3999,779,12562,2076,1523,1336,10309,523,32826,5751,3984,779,10039,1499,3644,1523,3290,8372,536,28682,1845,896,779,15671,1499,2743,1892,811,768,3177,8324,2076,54526,1523,803,779,10039,1090,4636,523,3732,5484,9303,811,49904,768,32673,5043,875,779,3549,797,4963,521,810,2993,3967,12547,3501,811,1801,523,509,532,523,53776,803,12900,708,548,54069,11860,811,2472,2030,906,12377,810,953,840,5508,3800,946,920,4346,523,9413,598,19718,953,840,5508,3800,906,27246,946,920,4346,523,2345,586,13682,2208,906,3185,802,9233,1720,4250,1779,523,8726,3967,1166,33256,875,779,2375,535,10593,3102,810,15642,523,4162,3290,15450,988,2059,16397,521,768,7155,48506,953,4908,523,10593,3102,5068,730,2235,20402,810,15642,730,1443,521,916,730,527,20069,9244,12452,523,10593,3102,953,13096,11510,906,3185,802,9233,523,941,8324,14954,791,2345,586,6248,876,875,1210,1779,803,3714,523,509,533,523,41453,730,526,535,39391,1465,14122,884,779,7314,5253,708,2364,4346,968,1030,774,521,2341,791,968,15821,793,535,997,7464,779,10039,47882,1287,811,730,526,521,1086,988,1352,25511,884,2759,39391,1465,14122,963,779,9028,803,779,7314,5253,521,3910,811,17677,15001,2927,1030,774,10201,896,28285,5419,938,6654,979,3213,811,779,8746,1327,521,1144,30809,779,5799,810,12785,779,3789,8034,875,10415,824,5569,523,3732,5484,5751,3984,779,1465,14122,1499,8524,38883,536,1030,774,25723,896,1186,1499,2472,797,3160,803,779,15565,1090,11404,523,1019,2176,535,730,2999,875,521,730,534,2723,521,730,532,41754,806,1114,523,941,9557,953,7818,523,509,534,523,41453,730,527,535,8900,22398,22819,708,2364,4346,968,779,15671,535,997,7464,779,8900,22819,27870,988,730,1374,811,730,1386,1094,8461,988,730,526,4471,730,1718,531,2927,941,3185,802,9233,59339,1845,896,1033,1423,1090,41938,4201,902,3761,27759,875,2166,523,17857,3967,1845,896,768,3585,22819,1499,3151,1612,1361,16671,521,4155,797,779,1361,1434,1872,884,13878,9690,523,1019,2176,535,730,1133,875,521,730,2117,2723,521,730,528,41754,806,1114,523,941,9557,953,17602,521,810,779,22819,32660,963,730,1374,523,509,1155,523,41453,730,528,535,3681,9041,708,2364,4346,968,38992,1068,521,2341,791,968,43842,535,997,7464,779,10039,12881,768,3681,9041,884,779,52352,13646,5598,779,11917,2927,32826,7951,779,4588,797,11618,1203,5751,1316,1499,2144,1720,779,9041,810,3984,779,11917,15671,40917,779,13646,523,1311,953,11860,811,2021,1287,768,3822,2514,803,38992,1068,521,43842,810,15642,811,2312,811,779,3280,3177,8324,523,3253,11386,953,4281,523,509,1632,523,10972,1312,3436,708,13792,518,7223,4729,884,8995,27784,535,2993,3967,4468,11242,10978,1714,3537,779,5630,963,2254,3217,523,941,27246,1188,5773,811,779,17677,14065,875,779,3354,8153,811,2957,6586,1818,819,10388,518,54894,535,20470,938,4602,811,5514,884,779,9168,3392,803,779,7314,819,10120,518,941,29780,12548,50804,777,1845,896,779,17677,1090,8825,14018,875,3681,7092,21981,797,4963,810,896,779,9041,2724,33460,819,9285,518,1416,48053,535,779,17677,1188,5840,7980,23878,988,5105,536,779,6111,3378,1188,874,16022,797,779,3280,38666,819,11634,518,8528,3711,937,1888,535,9213,833,4468,896,779,1629,3711,937,1888,5329,779,15565,953,20425,797,6060,810,896,1352,46967,2716,1046,797,5277,536,779,15671,1188,2565,8297,1667,523,509,1374,523,13887,803,779,3280,8324,708,1098,3280,3177,8324,1188,874,4908,797,4526,536,779,5562,1188,874,9139,797,779,6844,38666,523,941,8324,8557,963,730,2197,535,1155,523,18,3493,9862,811,779,9557,884,779,8900,22819,540,19,16,12043,20,19,16,20691,20,19,16,825,25745,535,6953,2939,791,811,768,2919,8324,933,768,3822,2514,20,19,16,47927,535,3877,577,29058,2076,779,11386,20,21],"sequence_length":1463,"marker_positions":[1427,1431,1435,1452],"positions":1463,"window":2048,"window_r10":2048,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.615821123123169,-2.7980968952178955,0.5164243578910828,-2.294139862060547],"probabilities_raw":[0.09758160263299942,0.029916591942310333,0.8229821920394897,0.04951968789100647],"probabilities":[0.15077616274356842,0.0647965595126152,0.6915529370307922,0.09287437796592712],"argmax_index":2,"argmax":"deferred","top2_margin":0.5407767742872238,"near_tie":false,"gold":"rejected","official_answer":{"type":"choice","choice":"deferred","confidence":0.6915529370307922,"probabilities":{"approved":0.15077616274356842,"rejected":0.0647965595126152,"deferred":0.6915529370307922,"withdrawn":0.09287437796592712}},"usage_input_tokens":5815,"oracle_file":"records_ref.json"},{"row_id":"own_L03/text/quorate","fixture_id":"own_L03","mode":"text","question_id":"quorate","question":{"type":"noul","instructions":"Was the meeting quorate?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,564,17769,3613,4425,12411,34632,2730,41891,516,31289,7272,550,10877,708,13635,3962,803,779,27845,5525,35590,4908,884,22045,730,1374,4690,730,1718,531,963,730,913,535,1664,797,33864,54898,2176,24579,7329,509,41678,535,730,3745,3967,843,1615,959,1143,13975,12930,518,810,730,528,24753,521,3136,1377,779,29780,12548,50804,777,521,2100,3852,1543,523,1030,779,12377,535,918,54069,523,3617,3962,535,9413,598,19718,843,52709,1126,1635,10412,6650,535,730,1632,3967,3567,1149,1514,6650,523,509,526,523,56975,810,967,24339,708,1098,12377,41975,3967,810,24753,810,14954,791,779,11917,15671,875,779,1517,803,779,5362,523,941,12778,6835,730,1664,3967,811,874,2251,875,768,3177,8324,811,874,967,777,931,523,3877,730,3745,3967,2251,521,779,8324,953,13096,967,777,931,523,509,527,523,3617,3962,803,779,2659,8900,3177,8324,708,1098,5112,803,779,8324,4908,884,730,1532,4690,730,1718,530,1166,12043,906,768,3347,3084,521,7885,968,9213,833,810,2341,791,968,28682,523,2479,1166,1295,14332,35517,896,1166,1014,4001,884,779,26168,523,509,528,523,21640,1090,2312,708,548,54069,17422,2851,779,1423,523,941,10039,9657,779,6844,7405,521,1144,9388,730,527,521,9300,875,779,11917,12562,3549,521,810,35130,1547,27297,45546,3256,779,8008,3509,521,1895,916,1209,1399,11583,20470,523,941,7982,875,768,55097,22212,884,8995,27784,6806,521,810,779,17677,1178,2166,11860,811,6715,1230,768,11013,8153,797,3354,523,49427,5313,14118,988,730,18833,811,730,22989,16671,523,1405,14954,791,779,15671,810,779,20470,1316,5514,779,38666,811,1973,3622,2759,3217,768,1423,523,509,529,523,9312,802,9233,1090,2312,708,1098,1230,9932,3185,802,9233,521,2345,586,521,6892,779,12324,875,779,1423,523,53813,953,730,531,521,21957,535,22398,23023,730,527,521,26547,521,779,6844,7405,730,527,521,9300,810,41634,730,526,521,30542,523,2315,2124,11761,953,730,529,521,25803,521,8787,38666,17236,843,526,521,21405,1109,12235,843,23617,1109,11917,33265,843,7537,518,810,7494,875,779,8008,3509,5158,6688,843,526,521,26668,1191,941,1423,8557,916,768,41938,803,730,526,521,32647,810,768,7493,8386,803,730,534,521,19222,523,941,12324,1320,1381,20425,968,902,7054,1924,5725,521,1316,1833,1295,4662,523,32826,12043,779,12324,968,768,1914,803,6108,521,916,1295,20402,2723,523,509,530,523,32964,988,779,3822,3892,708,26691,2514,843,549,37986,3271,779,2514,21440,730,5411,8267,35304,884,779,1375,977,5329,779,2254,810,50398,1547,803,779,2759,5854,20842,884,779,5253,523,1311,1499,1672,811,8153,779,8008,3509,875,21275,4476,797,779,6844,810,856,5400,875,1547,1209,20470,819,30193,2514,843,545,6310,793,3271,29504,779,6844,7405,521,779,2514,9657,768,30780,4915,797,4843,896,17322,1919,523,835,3316,4915,797,779,11917,856,13573,875,4471,1288,768,35563,1011,874,20560,963,768,16336,7264,819,35798,4977,810,5600,2514,843,549,831,833,3271,10325,988,779,22212,7982,521,779,2514,4468,2204,15073,11513,15489,810,768,26781,15761,884,13878,9690,811,779,17677,536,1547,803,779,15489,1052,1381,50398,523,509,531,523,7329,12562,3549,708,559,4182,4468,875,779,11917,15671,896,779,12562,12940,1178,1381,29878,963,730,1133,521,6019,523,3877,779,730,527,521,9300,988,779,6844,7405,521,779,3549,2166,12973,963,730,1632,521,14671,523,941,15671,1178,6944,811,1547,56302,8810,592,875,779,2280,810,45777,1149,13041,968,779,1764,803,3354,523,6162,4584,2155,2076,4526,1929,30541,1232,1000,797,779,12562,3494,2013,779,6844,521,810,902,8535,8923,1188,3999,779,12562,2076,1523,1336,10309,523,32826,5751,3984,779,10039,1499,3644,1523,3290,8372,536,28682,1845,896,779,15671,1499,2743,1892,811,768,3177,8324,2076,54526,1523,803,779,10039,1090,4636,523,3732,5484,9303,811,49904,768,32673,5043,875,779,3549,797,4963,521,810,2993,3967,12547,3501,811,1801,523,509,532,523,53776,803,12900,708,548,54069,11860,811,2472,2030,906,12377,810,953,840,5508,3800,946,920,4346,523,9413,598,19718,953,840,5508,3800,906,27246,946,920,4346,523,2345,586,13682,2208,906,3185,802,9233,1720,4250,1779,523,8726,3967,1166,33256,875,779,2375,535,10593,3102,810,15642,523,4162,3290,15450,988,2059,16397,521,768,7155,48506,953,4908,523,10593,3102,5068,730,2235,20402,810,15642,730,1443,521,916,730,527,20069,9244,12452,523,10593,3102,953,13096,11510,906,3185,802,9233,523,941,8324,14954,791,2345,586,6248,876,875,1210,1779,803,3714,523,509,533,523,41453,730,526,535,39391,1465,14122,884,779,7314,5253,708,2364,4346,968,1030,774,521,2341,791,968,15821,793,535,997,7464,779,10039,47882,1287,811,730,526,521,1086,988,1352,25511,884,2759,39391,1465,14122,963,779,9028,803,779,7314,5253,521,3910,811,17677,15001,2927,1030,774,10201,896,28285,5419,938,6654,979,3213,811,779,8746,1327,521,1144,30809,779,5799,810,12785,779,3789,8034,875,10415,824,5569,523,3732,5484,5751,3984,779,1465,14122,1499,8524,38883,536,1030,774,25723,896,1186,1499,2472,797,3160,803,779,15565,1090,11404,523,1019,2176,535,730,2999,875,521,730,534,2723,521,730,532,41754,806,1114,523,941,9557,953,7818,523,509,534,523,41453,730,527,535,8900,22398,22819,708,2364,4346,968,779,15671,535,997,7464,779,8900,22819,27870,988,730,1374,811,730,1386,1094,8461,988,730,526,4471,730,1718,531,2927,941,3185,802,9233,59339,1845,896,1033,1423,1090,41938,4201,902,3761,27759,875,2166,523,17857,3967,1845,896,768,3585,22819,1499,3151,1612,1361,16671,521,4155,797,779,1361,1434,1872,884,13878,9690,523,1019,2176,535,730,1133,875,521,730,2117,2723,521,730,528,41754,806,1114,523,941,9557,953,17602,521,810,779,22819,32660,963,730,1374,523,509,1155,523,41453,730,528,535,3681,9041,708,2364,4346,968,38992,1068,521,2341,791,968,43842,535,997,7464,779,10039,12881,768,3681,9041,884,779,52352,13646,5598,779,11917,2927,32826,7951,779,4588,797,11618,1203,5751,1316,1499,2144,1720,779,9041,810,3984,779,11917,15671,40917,779,13646,523,1311,953,11860,811,2021,1287,768,3822,2514,803,38992,1068,521,43842,810,15642,811,2312,811,779,3280,3177,8324,523,3253,11386,953,4281,523,509,1632,523,10972,1312,3436,708,13792,518,7223,4729,884,8995,27784,535,2993,3967,4468,11242,10978,1714,3537,779,5630,963,2254,3217,523,941,27246,1188,5773,811,779,17677,14065,875,779,3354,8153,811,2957,6586,1818,819,10388,518,54894,535,20470,938,4602,811,5514,884,779,9168,3392,803,779,7314,819,10120,518,941,29780,12548,50804,777,1845,896,779,17677,1090,8825,14018,875,3681,7092,21981,797,4963,810,896,779,9041,2724,33460,819,9285,518,1416,48053,535,779,17677,1188,5840,7980,23878,988,5105,536,779,6111,3378,1188,874,16022,797,779,3280,38666,819,11634,518,8528,3711,937,1888,535,9213,833,4468,896,779,1629,3711,937,1888,5329,779,15565,953,20425,797,6060,810,896,1352,46967,2716,1046,797,5277,536,779,15671,1188,2565,8297,1667,523,509,1374,523,13887,803,779,3280,8324,708,1098,3280,3177,8324,1188,874,4908,797,4526,536,779,5562,1188,874,9139,797,779,6844,38666,523,941,8324,8557,963,730,2197,535,1155,523,18,28519,779,8324,967,777,931,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":1446,"marker_positions":[1424,1436],"positions":1446,"window":2048,"window_r10":2048,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[0.2696681618690491,5.161313533782959],"probabilities_raw":[0.007453090511262417,0.9925468564033508],"probabilities":[0.9495782256126404,0.0504218228161335],"argmax_index":0,"argmax":"true","top2_margin":0.8991564027965069,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.9495782256126404},"usage_input_tokens":5815,"oracle_file":"records_ref.json"},{"row_id":"own_L03/text/treasurer","fixture_id":"own_L03","mode":"text","question_id":"treasurer","question":{"type":"choice","instructions":"Who was elected treasurer?","criteria":{"odette":"Odette","lewis":"Lewis"}},"type":"choice","qtype":0,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,564,17769,3613,4425,12411,34632,2730,41891,516,31289,7272,550,10877,708,13635,3962,803,779,27845,5525,35590,4908,884,22045,730,1374,4690,730,1718,531,963,730,913,535,1664,797,33864,54898,2176,24579,7329,509,41678,535,730,3745,3967,843,1615,959,1143,13975,12930,518,810,730,528,24753,521,3136,1377,779,29780,12548,50804,777,521,2100,3852,1543,523,1030,779,12377,535,918,54069,523,3617,3962,535,9413,598,19718,843,52709,1126,1635,10412,6650,535,730,1632,3967,3567,1149,1514,6650,523,509,526,523,56975,810,967,24339,708,1098,12377,41975,3967,810,24753,810,14954,791,779,11917,15671,875,779,1517,803,779,5362,523,941,12778,6835,730,1664,3967,811,874,2251,875,768,3177,8324,811,874,967,777,931,523,3877,730,3745,3967,2251,521,779,8324,953,13096,967,777,931,523,509,527,523,3617,3962,803,779,2659,8900,3177,8324,708,1098,5112,803,779,8324,4908,884,730,1532,4690,730,1718,530,1166,12043,906,768,3347,3084,521,7885,968,9213,833,810,2341,791,968,28682,523,2479,1166,1295,14332,35517,896,1166,1014,4001,884,779,26168,523,509,528,523,21640,1090,2312,708,548,54069,17422,2851,779,1423,523,941,10039,9657,779,6844,7405,521,1144,9388,730,527,521,9300,875,779,11917,12562,3549,521,810,35130,1547,27297,45546,3256,779,8008,3509,521,1895,916,1209,1399,11583,20470,523,941,7982,875,768,55097,22212,884,8995,27784,6806,521,810,779,17677,1178,2166,11860,811,6715,1230,768,11013,8153,797,3354,523,49427,5313,14118,988,730,18833,811,730,22989,16671,523,1405,14954,791,779,15671,810,779,20470,1316,5514,779,38666,811,1973,3622,2759,3217,768,1423,523,509,529,523,9312,802,9233,1090,2312,708,1098,1230,9932,3185,802,9233,521,2345,586,521,6892,779,12324,875,779,1423,523,53813,953,730,531,521,21957,535,22398,23023,730,527,521,26547,521,779,6844,7405,730,527,521,9300,810,41634,730,526,521,30542,523,2315,2124,11761,953,730,529,521,25803,521,8787,38666,17236,843,526,521,21405,1109,12235,843,23617,1109,11917,33265,843,7537,518,810,7494,875,779,8008,3509,5158,6688,843,526,521,26668,1191,941,1423,8557,916,768,41938,803,730,526,521,32647,810,768,7493,8386,803,730,534,521,19222,523,941,12324,1320,1381,20425,968,902,7054,1924,5725,521,1316,1833,1295,4662,523,32826,12043,779,12324,968,768,1914,803,6108,521,916,1295,20402,2723,523,509,530,523,32964,988,779,3822,3892,708,26691,2514,843,549,37986,3271,779,2514,21440,730,5411,8267,35304,884,779,1375,977,5329,779,2254,810,50398,1547,803,779,2759,5854,20842,884,779,5253,523,1311,1499,1672,811,8153,779,8008,3509,875,21275,4476,797,779,6844,810,856,5400,875,1547,1209,20470,819,30193,2514,843,545,6310,793,3271,29504,779,6844,7405,521,779,2514,9657,768,30780,4915,797,4843,896,17322,1919,523,835,3316,4915,797,779,11917,856,13573,875,4471,1288,768,35563,1011,874,20560,963,768,16336,7264,819,35798,4977,810,5600,2514,843,549,831,833,3271,10325,988,779,22212,7982,521,779,2514,4468,2204,15073,11513,15489,810,768,26781,15761,884,13878,9690,811,779,17677,536,1547,803,779,15489,1052,1381,50398,523,509,531,523,7329,12562,3549,708,559,4182,4468,875,779,11917,15671,896,779,12562,12940,1178,1381,29878,963,730,1133,521,6019,523,3877,779,730,527,521,9300,988,779,6844,7405,521,779,3549,2166,12973,963,730,1632,521,14671,523,941,15671,1178,6944,811,1547,56302,8810,592,875,779,2280,810,45777,1149,13041,968,779,1764,803,3354,523,6162,4584,2155,2076,4526,1929,30541,1232,1000,797,779,12562,3494,2013,779,6844,521,810,902,8535,8923,1188,3999,779,12562,2076,1523,1336,10309,523,32826,5751,3984,779,10039,1499,3644,1523,3290,8372,536,28682,1845,896,779,15671,1499,2743,1892,811,768,3177,8324,2076,54526,1523,803,779,10039,1090,4636,523,3732,5484,9303,811,49904,768,32673,5043,875,779,3549,797,4963,521,810,2993,3967,12547,3501,811,1801,523,509,532,523,53776,803,12900,708,548,54069,11860,811,2472,2030,906,12377,810,953,840,5508,3800,946,920,4346,523,9413,598,19718,953,840,5508,3800,906,27246,946,920,4346,523,2345,586,13682,2208,906,3185,802,9233,1720,4250,1779,523,8726,3967,1166,33256,875,779,2375,535,10593,3102,810,15642,523,4162,3290,15450,988,2059,16397,521,768,7155,48506,953,4908,523,10593,3102,5068,730,2235,20402,810,15642,730,1443,521,916,730,527,20069,9244,12452,523,10593,3102,953,13096,11510,906,3185,802,9233,523,941,8324,14954,791,2345,586,6248,876,875,1210,1779,803,3714,523,509,533,523,41453,730,526,535,39391,1465,14122,884,779,7314,5253,708,2364,4346,968,1030,774,521,2341,791,968,15821,793,535,997,7464,779,10039,47882,1287,811,730,526,521,1086,988,1352,25511,884,2759,39391,1465,14122,963,779,9028,803,779,7314,5253,521,3910,811,17677,15001,2927,1030,774,10201,896,28285,5419,938,6654,979,3213,811,779,8746,1327,521,1144,30809,779,5799,810,12785,779,3789,8034,875,10415,824,5569,523,3732,5484,5751,3984,779,1465,14122,1499,8524,38883,536,1030,774,25723,896,1186,1499,2472,797,3160,803,779,15565,1090,11404,523,1019,2176,535,730,2999,875,521,730,534,2723,521,730,532,41754,806,1114,523,941,9557,953,7818,523,509,534,523,41453,730,527,535,8900,22398,22819,708,2364,4346,968,779,15671,535,997,7464,779,8900,22819,27870,988,730,1374,811,730,1386,1094,8461,988,730,526,4471,730,1718,531,2927,941,3185,802,9233,59339,1845,896,1033,1423,1090,41938,4201,902,3761,27759,875,2166,523,17857,3967,1845,896,768,3585,22819,1499,3151,1612,1361,16671,521,4155,797,779,1361,1434,1872,884,13878,9690,523,1019,2176,535,730,1133,875,521,730,2117,2723,521,730,528,41754,806,1114,523,941,9557,953,17602,521,810,779,22819,32660,963,730,1374,523,509,1155,523,41453,730,528,535,3681,9041,708,2364,4346,968,38992,1068,521,2341,791,968,43842,535,997,7464,779,10039,12881,768,3681,9041,884,779,52352,13646,5598,779,11917,2927,32826,7951,779,4588,797,11618,1203,5751,1316,1499,2144,1720,779,9041,810,3984,779,11917,15671,40917,779,13646,523,1311,953,11860,811,2021,1287,768,3822,2514,803,38992,1068,521,43842,810,15642,811,2312,811,779,3280,3177,8324,523,3253,11386,953,4281,523,509,1632,523,10972,1312,3436,708,13792,518,7223,4729,884,8995,27784,535,2993,3967,4468,11242,10978,1714,3537,779,5630,963,2254,3217,523,941,27246,1188,5773,811,779,17677,14065,875,779,3354,8153,811,2957,6586,1818,819,10388,518,54894,535,20470,938,4602,811,5514,884,779,9168,3392,803,779,7314,819,10120,518,941,29780,12548,50804,777,1845,896,779,17677,1090,8825,14018,875,3681,7092,21981,797,4963,810,896,779,9041,2724,33460,819,9285,518,1416,48053,535,779,17677,1188,5840,7980,23878,988,5105,536,779,6111,3378,1188,874,16022,797,779,3280,38666,819,11634,518,8528,3711,937,1888,535,9213,833,4468,896,779,1629,3711,937,1888,5329,779,15565,953,20425,797,6060,810,896,1352,46967,2716,1046,797,5277,536,779,15671,1188,2565,8297,1667,523,509,1374,523,13887,803,779,3280,8324,708,1098,3280,3177,8324,1188,874,4908,797,4526,536,779,5562,1188,874,9139,797,779,6844,38666,523,941,8324,8557,963,730,2197,535,1155,523,18,17408,953,11510,3185,802,9233,540,19,16,3352,3102,535,10593,3102,20,19,16,945,596,783,535,15642,20,21],"sequence_length":1440,"marker_positions":[1424,1432],"positions":1440,"window":2048,"window_r10":2048,"temperature_key":"choice:2","temperature":1.7465145587921143,"raw_logits":[-0.2319762408733368,-2.207134246826172],"probabilities_raw":[0.8781641125679016,0.12183595448732376],"probabilities":[0.7560074925422668,0.24399243295192719],"argmax_index":0,"argmax":"odette","top2_margin":0.5120150595903397,"near_tie":false,"gold":"odette","official_answer":{"type":"choice","choice":"odette","confidence":0.7560074925422668,"probabilities":{"odette":0.7560074925422668,"lewis":0.24399243295192719}},"usage_input_tokens":5815,"oracle_file":"records_ref.json"},{"row_id":"own_L03/text/racks_support","fixture_id":"own_L03","mode":"text","question_id":"racks_support","question":{"type":"score","instructions":"How strong was the support for the bicycle racks motion?","criteria":["opposed by most","split","supported by most","unanimous"]},"type":"score","qtype":1,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,564,17769,3613,4425,12411,34632,2730,41891,516,31289,7272,550,10877,708,13635,3962,803,779,27845,5525,35590,4908,884,22045,730,1374,4690,730,1718,531,963,730,913,535,1664,797,33864,54898,2176,24579,7329,509,41678,535,730,3745,3967,843,1615,959,1143,13975,12930,518,810,730,528,24753,521,3136,1377,779,29780,12548,50804,777,521,2100,3852,1543,523,1030,779,12377,535,918,54069,523,3617,3962,535,9413,598,19718,843,52709,1126,1635,10412,6650,535,730,1632,3967,3567,1149,1514,6650,523,509,526,523,56975,810,967,24339,708,1098,12377,41975,3967,810,24753,810,14954,791,779,11917,15671,875,779,1517,803,779,5362,523,941,12778,6835,730,1664,3967,811,874,2251,875,768,3177,8324,811,874,967,777,931,523,3877,730,3745,3967,2251,521,779,8324,953,13096,967,777,931,523,509,527,523,3617,3962,803,779,2659,8900,3177,8324,708,1098,5112,803,779,8324,4908,884,730,1532,4690,730,1718,530,1166,12043,906,768,3347,3084,521,7885,968,9213,833,810,2341,791,968,28682,523,2479,1166,1295,14332,35517,896,1166,1014,4001,884,779,26168,523,509,528,523,21640,1090,2312,708,548,54069,17422,2851,779,1423,523,941,10039,9657,779,6844,7405,521,1144,9388,730,527,521,9300,875,779,11917,12562,3549,521,810,35130,1547,27297,45546,3256,779,8008,3509,521,1895,916,1209,1399,11583,20470,523,941,7982,875,768,55097,22212,884,8995,27784,6806,521,810,779,17677,1178,2166,11860,811,6715,1230,768,11013,8153,797,3354,523,49427,5313,14118,988,730,18833,811,730,22989,16671,523,1405,14954,791,779,15671,810,779,20470,1316,5514,779,38666,811,1973,3622,2759,3217,768,1423,523,509,529,523,9312,802,9233,1090,2312,708,1098,1230,9932,3185,802,9233,521,2345,586,521,6892,779,12324,875,779,1423,523,53813,953,730,531,521,21957,535,22398,23023,730,527,521,26547,521,779,6844,7405,730,527,521,9300,810,41634,730,526,521,30542,523,2315,2124,11761,953,730,529,521,25803,521,8787,38666,17236,843,526,521,21405,1109,12235,843,23617,1109,11917,33265,843,7537,518,810,7494,875,779,8008,3509,5158,6688,843,526,521,26668,1191,941,1423,8557,916,768,41938,803,730,526,521,32647,810,768,7493,8386,803,730,534,521,19222,523,941,12324,1320,1381,20425,968,902,7054,1924,5725,521,1316,1833,1295,4662,523,32826,12043,779,12324,968,768,1914,803,6108,521,916,1295,20402,2723,523,509,530,523,32964,988,779,3822,3892,708,26691,2514,843,549,37986,3271,779,2514,21440,730,5411,8267,35304,884,779,1375,977,5329,779,2254,810,50398,1547,803,779,2759,5854,20842,884,779,5253,523,1311,1499,1672,811,8153,779,8008,3509,875,21275,4476,797,779,6844,810,856,5400,875,1547,1209,20470,819,30193,2514,843,545,6310,793,3271,29504,779,6844,7405,521,779,2514,9657,768,30780,4915,797,4843,896,17322,1919,523,835,3316,4915,797,779,11917,856,13573,875,4471,1288,768,35563,1011,874,20560,963,768,16336,7264,819,35798,4977,810,5600,2514,843,549,831,833,3271,10325,988,779,22212,7982,521,779,2514,4468,2204,15073,11513,15489,810,768,26781,15761,884,13878,9690,811,779,17677,536,1547,803,779,15489,1052,1381,50398,523,509,531,523,7329,12562,3549,708,559,4182,4468,875,779,11917,15671,896,779,12562,12940,1178,1381,29878,963,730,1133,521,6019,523,3877,779,730,527,521,9300,988,779,6844,7405,521,779,3549,2166,12973,963,730,1632,521,14671,523,941,15671,1178,6944,811,1547,56302,8810,592,875,779,2280,810,45777,1149,13041,968,779,1764,803,3354,523,6162,4584,2155,2076,4526,1929,30541,1232,1000,797,779,12562,3494,2013,779,6844,521,810,902,8535,8923,1188,3999,779,12562,2076,1523,1336,10309,523,32826,5751,3984,779,10039,1499,3644,1523,3290,8372,536,28682,1845,896,779,15671,1499,2743,1892,811,768,3177,8324,2076,54526,1523,803,779,10039,1090,4636,523,3732,5484,9303,811,49904,768,32673,5043,875,779,3549,797,4963,521,810,2993,3967,12547,3501,811,1801,523,509,532,523,53776,803,12900,708,548,54069,11860,811,2472,2030,906,12377,810,953,840,5508,3800,946,920,4346,523,9413,598,19718,953,840,5508,3800,906,27246,946,920,4346,523,2345,586,13682,2208,906,3185,802,9233,1720,4250,1779,523,8726,3967,1166,33256,875,779,2375,535,10593,3102,810,15642,523,4162,3290,15450,988,2059,16397,521,768,7155,48506,953,4908,523,10593,3102,5068,730,2235,20402,810,15642,730,1443,521,916,730,527,20069,9244,12452,523,10593,3102,953,13096,11510,906,3185,802,9233,523,941,8324,14954,791,2345,586,6248,876,875,1210,1779,803,3714,523,509,533,523,41453,730,526,535,39391,1465,14122,884,779,7314,5253,708,2364,4346,968,1030,774,521,2341,791,968,15821,793,535,997,7464,779,10039,47882,1287,811,730,526,521,1086,988,1352,25511,884,2759,39391,1465,14122,963,779,9028,803,779,7314,5253,521,3910,811,17677,15001,2927,1030,774,10201,896,28285,5419,938,6654,979,3213,811,779,8746,1327,521,1144,30809,779,5799,810,12785,779,3789,8034,875,10415,824,5569,523,3732,5484,5751,3984,779,1465,14122,1499,8524,38883,536,1030,774,25723,896,1186,1499,2472,797,3160,803,779,15565,1090,11404,523,1019,2176,535,730,2999,875,521,730,534,2723,521,730,532,41754,806,1114,523,941,9557,953,7818,523,509,534,523,41453,730,527,535,8900,22398,22819,708,2364,4346,968,779,15671,535,997,7464,779,8900,22819,27870,988,730,1374,811,730,1386,1094,8461,988,730,526,4471,730,1718,531,2927,941,3185,802,9233,59339,1845,896,1033,1423,1090,41938,4201,902,3761,27759,875,2166,523,17857,3967,1845,896,768,3585,22819,1499,3151,1612,1361,16671,521,4155,797,779,1361,1434,1872,884,13878,9690,523,1019,2176,535,730,1133,875,521,730,2117,2723,521,730,528,41754,806,1114,523,941,9557,953,17602,521,810,779,22819,32660,963,730,1374,523,509,1155,523,41453,730,528,535,3681,9041,708,2364,4346,968,38992,1068,521,2341,791,968,43842,535,997,7464,779,10039,12881,768,3681,9041,884,779,52352,13646,5598,779,11917,2927,32826,7951,779,4588,797,11618,1203,5751,1316,1499,2144,1720,779,9041,810,3984,779,11917,15671,40917,779,13646,523,1311,953,11860,811,2021,1287,768,3822,2514,803,38992,1068,521,43842,810,15642,811,2312,811,779,3280,3177,8324,523,3253,11386,953,4281,523,509,1632,523,10972,1312,3436,708,13792,518,7223,4729,884,8995,27784,535,2993,3967,4468,11242,10978,1714,3537,779,5630,963,2254,3217,523,941,27246,1188,5773,811,779,17677,14065,875,779,3354,8153,811,2957,6586,1818,819,10388,518,54894,535,20470,938,4602,811,5514,884,779,9168,3392,803,779,7314,819,10120,518,941,29780,12548,50804,777,1845,896,779,17677,1090,8825,14018,875,3681,7092,21981,797,4963,810,896,779,9041,2724,33460,819,9285,518,1416,48053,535,779,17677,1188,5840,7980,23878,988,5105,536,779,6111,3378,1188,874,16022,797,779,3280,38666,819,11634,518,8528,3711,937,1888,535,9213,833,4468,896,779,1629,3711,937,1888,5329,779,15565,953,20425,797,6060,810,896,1352,46967,2716,1046,797,5277,536,779,15671,1188,2565,8297,1667,523,509,1374,523,13887,803,779,3280,8324,708,1098,3280,3177,8324,1188,874,4908,797,4526,536,779,5562,1188,874,9139,797,779,6844,38666,523,941,8324,8557,963,730,2197,535,1155,523,18,4023,3636,953,779,2258,875,779,39391,1465,14122,9557,540,19,16,2119,730,525,535,14873,968,1529,20,19,16,2119,730,526,535,13305,20,19,16,2119,730,527,535,7951,968,1529,20,19,16,2119,730,528,535,42869,1069,20,21],"sequence_length":1466,"marker_positions":[1429,1439,1447,1457],"positions":1466,"window":2048,"window_r10":2048,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-4.230611324310303,0.1285986751317978,-1.4657516479492188,-2.841369867324829],"probabilities_raw":[0.010092451237142086,0.7891826033592224,0.1602359414100647,0.04048895090818405],"probabilities":[0.04854496940970421,0.6031103730201721,0.23998364806175232,0.10836095362901688],"argmax_index":1,"argmax":"1","top2_margin":0.3631267249584198,"near_tie":false,"gold":"2","official_answer":{"type":"score","score":1.4081605300307274,"confidence":0.6031103730201721,"probabilities":{"0":0.04854496940970421,"1":0.6031103730201721,"2":0.23998364806175232,"3":0.10836095362901688},"legend":{"0":"opposed by most","1":"split","2":"supported by most","3":"unanimous"}},"usage_input_tokens":5815,"oracle_file":"records_ref.json"},{"row_id":"own_a01/text/first","fixture_id":"own_a01","mode":"text","question_id":"first","question":{"type":"choice","instructions":"Which ticket should be handled first?","criteria":{"HD-101":"Printer jams","HD-102":"Entrance card readers","HD-103":"Second monitor","HD-104":"Burning smell"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,568,38685,593,32909,6881,997,13806,522,5768,1878,997,7820,6881,997,5059,2874,884,8913,730,527,996,2534,884,1973,4426,1878,997,782,29500,6881,997,4905,535,3654,1878,997,3165,1012,592,6881,997,526,3410,511,3205,37080,593,32909,6881,997,13806,522,9270,1878,997,7820,6881,997,29408,13040,963,779,2120,19025,22879,1113,5663,3273,1878,997,782,29500,6881,997,4905,535,1374,1878,997,3165,1012,592,6881,997,2291,1912,4204,511,3205,37080,593,32909,6881,997,13806,522,8884,1878,997,7820,6881,997,7367,875,768,2341,6562,1878,997,782,29500,6881,997,4905,535,1664,1878,997,3165,1012,592,6881,997,526,1811,511,3205,37080,593,32909,6881,997,13806,522,10981,1878,997,7820,6881,997,14100,965,803,15658,8997,3627,779,9397,5362,1878,997,782,29500,6881,997,4905,535,4536,1878,997,3165,1012,592,6881,997,51493,511,20352,18,28828,30437,1753,874,28010,1509,540,19,16,17905,522,5768,535,2343,2874,996,2534,20,19,16,17905,522,9270,535,5719,20391,5060,13040,20,19,16,17905,522,8884,535,8296,6562,20,19,16,17905,522,10981,535,23250,800,19944,20,21],"sequence_length":220,"marker_positions":[179,190,201,210],"positions":220,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.3843047022819519,-2.1585264205932617,-0.326029896736145,0.20786070823669434],"probabilities_raw":[0.4152187407016754,0.03265422582626343,0.20407159626483917,0.34805533289909363],"probabilities":[0.3778960704803467,0.061447251588106155,0.2275114357471466,0.33314526081085205],"argmax_index":0,"argmax":"HD-101","top2_margin":0.04475080966949463,"near_tie":false,"gold":"HD-104","official_answer":{"type":"choice","choice":"HD-101","confidence":0.3778960704803467,"probabilities":{"HD-101":0.3778960704803467,"HD-102":0.061447251588106155,"HD-103":0.2275114357471466,"HD-104":0.33314526081085205}},"usage_input_tokens":618,"oracle_file":"records_ref.json"},{"row_id":"own_a01/text/hazard","fixture_id":"own_a01","mode":"text","question_id":"hazard","question":{"type":"noul","instructions":"Does any ticket describe a possible safety hazard?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,568,38685,593,32909,6881,997,13806,522,5768,1878,997,7820,6881,997,5059,2874,884,8913,730,527,996,2534,884,1973,4426,1878,997,782,29500,6881,997,4905,535,3654,1878,997,3165,1012,592,6881,997,526,3410,511,3205,37080,593,32909,6881,997,13806,522,9270,1878,997,7820,6881,997,29408,13040,963,779,2120,19025,22879,1113,5663,3273,1878,997,782,29500,6881,997,4905,535,1374,1878,997,3165,1012,592,6881,997,2291,1912,4204,511,3205,37080,593,32909,6881,997,13806,522,8884,1878,997,7820,6881,997,7367,875,768,2341,6562,1878,997,782,29500,6881,997,4905,535,1664,1878,997,3165,1012,592,6881,997,526,1811,511,3205,37080,593,32909,6881,997,13806,522,10981,1878,997,7820,6881,997,14100,965,803,15658,8997,3627,779,9397,5362,1878,997,782,29500,6881,997,4905,535,4536,1878,997,3165,1012,592,6881,997,51493,511,20352,18,29589,1523,30437,8487,768,3023,6155,17745,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":203,"marker_positions":[181,193],"positions":203,"window":256,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[2.477010488510132,6.635200023651123],"probabilities_raw":[0.015395121648907661,0.9846048951148987],"probabilities":[0.9238207936286926,0.07617922872304916],"argmax_index":0,"argmax":"true","top2_margin":0.8476415649056435,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.9238207936286926},"usage_input_tokens":618,"oracle_file":"records_ref.json"},{"row_id":"own_a01/text/load","fixture_id":"own_a01","mode":"text","question_id":"load","question":{"type":"score","instructions":"Is the queue overloaded?","criteria":["normal","overloaded"]},"type":"score","qtype":1,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,568,38685,593,32909,6881,997,13806,522,5768,1878,997,7820,6881,997,5059,2874,884,8913,730,527,996,2534,884,1973,4426,1878,997,782,29500,6881,997,4905,535,3654,1878,997,3165,1012,592,6881,997,526,3410,511,3205,37080,593,32909,6881,997,13806,522,9270,1878,997,7820,6881,997,29408,13040,963,779,2120,19025,22879,1113,5663,3273,1878,997,782,29500,6881,997,4905,535,1374,1878,997,3165,1012,592,6881,997,2291,1912,4204,511,3205,37080,593,32909,6881,997,13806,522,8884,1878,997,7820,6881,997,7367,875,768,2341,6562,1878,997,782,29500,6881,997,4905,535,1664,1878,997,3165,1012,592,6881,997,526,1811,511,3205,37080,593,32909,6881,997,13806,522,10981,1878,997,7820,6881,997,14100,965,803,15658,8997,3627,779,9397,5362,1878,997,782,29500,6881,997,4905,535,4536,1878,997,3165,1012,592,6881,997,51493,511,20352,18,4940,779,27838,1391,38596,540,19,16,2119,730,525,535,4381,20,19,16,2119,730,526,535,1391,38596,20,21],"sequence_length":195,"marker_positions":[178,186],"positions":195,"window":256,"window_r10":256,"temperature_key":"score:2","temperature":1.0,"raw_logits":[-2.7285783290863037,2.443589687347412],"probabilities_raw":[0.005640266463160515,0.9943597912788391],"probabilities":[0.005640266463160515,0.9943597912788391],"argmax_index":1,"argmax":"1","top2_margin":0.9887195248156786,"near_tie":false,"gold":"0","official_answer":{"type":"score","score":0.9943597912788391,"confidence":0.9943597912788391,"probabilities":{"0":0.005640266463160515,"1":0.9943597912788391},"legend":{"0":"normal","1":"overloaded"}},"usage_input_tokens":618,"oracle_file":"records_ref.json"},{"row_id":"own_j01/text/overdue","fixture_id":"own_j01","mode":"text","question_id":"overdue","question":{"type":"noul","instructions":"Is the invoice overdue?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,769,47958,6881,997,15807,564,522,1718,531,522,30907,527,1878,997,1271,2466,6881,997,547,1983,893,1174,18119,16643,1878,997,38765,770,6881,997,556,820,799,12890,12519,1157,4847,1878,997,1271,2686,6881,997,1718,531,522,6326,522,5843,1878,997,30717,6881,997,1718,531,522,3654,522,5843,1878,997,19248,6881,997,885,38008,1878,997,2481,53339,4294,806,6881,730,527,521,997,10890,6881,1607,38685,5027,6881,997,2414,29248,47635,835,527,1878,997,590,1937,6881,730,2108,521,997,10829,572,51363,6881,730,526,523,3520,3205,37080,5027,6881,997,19938,1477,2643,10819,1878,997,590,1937,6881,730,7896,521,997,10829,572,51363,6881,730,525,523,5370,3205,37080,5027,6881,997,19589,36603,1878,997,590,1937,6881,730,526,521,997,10829,572,51363,6881,730,1374,523,525,602,5086,997,4093,35377,6881,730,22277,523,525,521,997,14623,572,10233,6881,730,525,523,527,521,997,35377,6881,730,25633,523,527,521,997,27317,957,572,55913,6881,997,18703,6337,2638,730,1664,3378,1878,997,593,53277,6881,997,1718,531,522,3654,522,913,511,602,18,4940,779,60827,1139,1391,30717,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":247,"marker_positions":[225,237],"positions":247,"window":256,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[-0.45539525151252747,1.4913076162338257],"probabilities_raw":[0.12491331994533539,0.8750866651535034],"probabilities":[0.762830913066864,0.237169086933136],"argmax_index":0,"argmax":"true","top2_margin":0.525661826133728,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.762830913066864},"usage_input_tokens":762,"oracle_file":"records_ref.json"},{"row_id":"own_j01/text/method","fixture_id":"own_j01","mode":"text","question_id":"method","question":{"type":"choice","instructions":"How is the customer expected to pay?","criteria":{"card":null,"bank_transfer":null,"cash":null,"cheque":null}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,769,47958,6881,997,15807,564,522,1718,531,522,30907,527,1878,997,1271,2466,6881,997,547,1983,893,1174,18119,16643,1878,997,38765,770,6881,997,556,820,799,12890,12519,1157,4847,1878,997,1271,2686,6881,997,1718,531,522,6326,522,5843,1878,997,30717,6881,997,1718,531,522,3654,522,5843,1878,997,19248,6881,997,885,38008,1878,997,2481,53339,4294,806,6881,730,527,521,997,10890,6881,1607,38685,5027,6881,997,2414,29248,47635,835,527,1878,997,590,1937,6881,730,2108,521,997,10829,572,51363,6881,730,526,523,3520,3205,37080,5027,6881,997,19938,1477,2643,10819,1878,997,590,1937,6881,730,7896,521,997,10829,572,51363,6881,730,525,523,5370,3205,37080,5027,6881,997,19589,36603,1878,997,590,1937,6881,730,526,521,997,10829,572,51363,6881,730,1374,523,525,602,5086,997,4093,35377,6881,730,22277,523,525,521,997,14623,572,10233,6881,730,525,523,527,521,997,35377,6881,730,25633,523,527,521,997,27317,957,572,55913,6881,997,18703,6337,2638,730,1664,3378,1878,997,593,53277,6881,997,1718,531,522,3654,522,913,511,602,18,4023,856,779,13608,5559,811,4025,540,19,16,5060,20,19,16,7493,41953,1100,20,19,16,14184,20,19,16,1636,1357,20,21],"sequence_length":245,"marker_positions":[226,230,236,240],"positions":245,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-9.03740406036377,1.9014228582382202,-6.792163848876953,-6.893122673034668],"probabilities_raw":[1.774932025000453e-05,0.999663233757019,0.00016760120342951268,0.0001515065087005496],"probabilities":[0.00040230932063423097,0.9957361221313477,0.0020003572572022676,0.001861173310317099],"argmax_index":1,"argmax":"bank_transfer","top2_margin":0.9937357648741454,"near_tie":false,"gold":"bank_transfer","official_answer":{"type":"choice","choice":"bank_transfer","confidence":0.9957361221313477,"probabilities":{"card":0.00040230932063423097,"bank_transfer":0.9957361221313477,"cash":0.0020003572572022676,"cheque":0.001861173310317099}},"usage_input_tokens":762,"oracle_file":"records_ref.json"},{"row_id":"own_j01/text/size","fixture_id":"own_j01","mode":"text","question_id":"size","question":{"type":"score","instructions":"How large is the invoice total?","criteria":["under 100","100 to 250","250 to 500","over 500"]},"type":"score","qtype":1,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,769,47958,6881,997,15807,564,522,1718,531,522,30907,527,1878,997,1271,2466,6881,997,547,1983,893,1174,18119,16643,1878,997,38765,770,6881,997,556,820,799,12890,12519,1157,4847,1878,997,1271,2686,6881,997,1718,531,522,6326,522,5843,1878,997,30717,6881,997,1718,531,522,3654,522,5843,1878,997,19248,6881,997,885,38008,1878,997,2481,53339,4294,806,6881,730,527,521,997,10890,6881,1607,38685,5027,6881,997,2414,29248,47635,835,527,1878,997,590,1937,6881,730,2108,521,997,10829,572,51363,6881,730,526,523,3520,3205,37080,5027,6881,997,19938,1477,2643,10819,1878,997,590,1937,6881,730,7896,521,997,10829,572,51363,6881,730,525,523,5370,3205,37080,5027,6881,997,19589,36603,1878,997,590,1937,6881,730,526,521,997,10829,572,51363,6881,730,1374,523,525,602,5086,997,4093,35377,6881,730,22277,523,525,521,997,14623,572,10233,6881,730,525,523,527,521,997,35377,6881,730,25633,523,527,521,997,27317,957,572,55913,6881,997,18703,6337,2638,730,1664,3378,1878,997,593,53277,6881,997,1718,531,522,3654,522,913,511,602,18,4023,2611,856,779,60827,1139,3426,540,19,16,2119,730,525,535,1490,730,2096,20,19,16,2119,730,526,535,730,2096,811,730,7896,20,19,16,2119,730,527,535,730,7896,811,730,4291,20,19,16,2119,730,528,535,1391,730,4291,20,21],"sequence_length":270,"marker_positions":[226,236,248,260],"positions":270,"window":512,"window_r10":512,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-15.045595169067383,-6.663366317749023,-6.0419511795043945,-7.225985527038574],"probabilities_raw":[6.670520815532655e-05,0.2914173901081085,0.5424911379814148,0.1660248339176178],"probabilities":[0.002488102298229933,0.316214919090271,0.4528668224811554,0.22843007743358612],"argmax_index":2,"argmax":"2","top2_margin":0.1366519033908844,"near_tie":false,"gold":"2","official_answer":{"type":"score","score":1.9072387963533401,"confidence":0.4528668224811554,"probabilities":{"0":0.002488102298229933,"1":0.316214919090271,"2":0.4528668224811554,"3":0.22843007743358612},"legend":{"0":"under 100","1":"100 to 250","2":"250 to 500","3":"over 500"}},"usage_input_tokens":762,"oracle_file":"records_ref.json"},{"row_id":"own_j02/text/next_action","fixture_id":"own_j02","mode":"text","question_id":"next_action","question":{"type":"choice","instructions":"What should the shop do next?","criteria":{"refund_missing_item":"Refund the item that was not sent","resend_order":"Send the whole order again","track_parcel":"Ask the courier where the parcel is","cancel_order":"Cancel the order","change_address":"Update the delivery address","discount_code":"Give a discount code","replace_damaged":"Replace damaged goods","confirm_delivery":"Tell the customer the parcel was delivered","fraud_review":"Flag the order for fraud review","no_action":"Nothing needs to be done"}},"type":"choice","qtype":0,"K":10,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,3667,6590,6881,997,42468,522,26260,1532,1878,997,15162,6881,997,557,1983,834,917,38281,1878,997,57267,6881,997,1718,531,522,6468,522,6135,730,1133,535,2019,1878,997,26207,6881,1607,38685,38017,6881,997,556,2508,522,526,552,548,1878,997,3055,6881,997,46553,791,63437,730,526,14895,1878,997,590,1937,6881,730,527,521,997,19248,6881,997,2354,9487,511,3205,37080,38017,6881,997,549,41555,522,14819,1878,997,3055,6881,997,41388,32849,17125,730,14819,836,1878,997,590,1937,6881,730,526,521,997,19248,6881,997,1066,803,9051,511,3205,37080,38017,6881,997,12411,542,4894,17727,1878,997,3055,6881,997,26818,13192,521,730,1838,22173,1878,997,590,1937,6881,730,528,521,997,19248,6881,997,2354,9487,511,602,5086,997,2354,11468,6881,37080,8618,6265,6881,997,2482,1930,2286,1145,1878,997,19132,800,36619,6881,997,14402,1991,791,1878,997,14402,1991,791,40040,6881,997,1718,531,522,6468,522,3654,730,1632,535,2108,511,3205,997,27317,957,6881,37080,14164,6881,997,19563,1878,997,576,2922,3107,6881,730,2142,523,529,521,997,1891,1356,791,6881,730,525,523,525,3205,997,38765,770,5205,2176,6881,997,15090,856,1727,17125,540,859,953,15418,875,936,2927,602,18,3493,1753,779,15565,1112,3280,540,19,16,53467,5377,45556,43840,535,12620,1356,779,8688,896,953,1014,3567,20,19,16,1039,959,572,3667,535,26995,779,4394,2565,2030,20,19,16,5254,572,1608,5072,535,18388,779,2286,1145,1695,779,61026,856,20,19,16,23496,572,3667,535,2577,5072,779,2565,20,19,16,2785,53220,535,23361,779,11746,4332,20,19,16,25668,30024,535,27908,768,25668,5214,20,19,16,8297,4370,831,5278,535,1416,4128,15073,11391,20,19,16,10547,4370,808,36603,535,27589,779,13608,779,61026,953,14465,20,19,16,24744,15623,2265,535,32655,779,2565,875,24744,4202,20,19,16,1295,55692,535,28657,3988,811,874,4007,20,21],"sequence_length":378,"marker_positions":[251,267,280,296,307,317,327,339,354,367],"positions":378,"window":512,"window_r10":512,"temperature_key":"choice:6-10","temperature":1.1751071214675903,"raw_logits":[-1.7629752159118652,-6.255130290985107,0.7563826441764832,-5.840479373931885,-10.807558059692383,-8.814050674438477,-13.214742660522461,-4.54431676864624,-5.7599897384643555,-2.928925037384033],"probabilities_raw":[0.07224540412425995,0.0008088949834927917,0.8973326086997986,0.001224539359100163,8.527027603122406e-06,6.259893416427076e-05,7.6801677550975e-07,0.004475989378988743,0.0013271773932501674,0.022513583302497864],"probabilities":[0.09913931041955948,0.0021679105702787638,0.8459538817405701,0.0030852300114929676,4.503615491557866e-05,0.0002456512884236872,5.806538865726907e-06,0.009296542964875698,0.0033039606641978025,0.03675662726163864],"argmax_index":2,"argmax":"track_parcel","top2_margin":0.7468145713210106,"near_tie":false,"gold":"refund_missing_item","official_answer":{"type":"choice","choice":"track_parcel","confidence":0.8459538817405701,"probabilities":{"refund_missing_item":0.09913931041955948,"resend_order":0.0021679105702787638,"track_parcel":0.8459538817405701,"cancel_order":0.0030852300114929676,"change_address":4.503615491557866e-05,"discount_code":0.0002456512884236872,"replace_damaged":5.806538865726907e-06,"confirm_delivery":0.009296542964875698,"fraud_review":0.0033039606641978025,"no_action":0.03675662726163864}},"usage_input_tokens":931,"oracle_file":"records_ref.json"},{"row_id":"own_j02/text/partial","fixture_id":"own_j02","mode":"text","question_id":"partial","question":{"type":"noul","instructions":"Was only part of the order shipped?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,3667,6590,6881,997,42468,522,26260,1532,1878,997,15162,6881,997,557,1983,834,917,38281,1878,997,57267,6881,997,1718,531,522,6468,522,6135,730,1133,535,2019,1878,997,26207,6881,1607,38685,38017,6881,997,556,2508,522,526,552,548,1878,997,3055,6881,997,46553,791,63437,730,526,14895,1878,997,590,1937,6881,730,527,521,997,19248,6881,997,2354,9487,511,3205,37080,38017,6881,997,549,41555,522,14819,1878,997,3055,6881,997,41388,32849,17125,730,14819,836,1878,997,590,1937,6881,730,526,521,997,19248,6881,997,1066,803,9051,511,3205,37080,38017,6881,997,12411,542,4894,17727,1878,997,3055,6881,997,26818,13192,521,730,1838,22173,1878,997,590,1937,6881,730,528,521,997,19248,6881,997,2354,9487,511,602,5086,997,2354,11468,6881,37080,8618,6265,6881,997,2482,1930,2286,1145,1878,997,19132,800,36619,6881,997,14402,1991,791,1878,997,14402,1991,791,40040,6881,997,1718,531,522,6468,522,3654,730,1632,535,2108,511,3205,997,27317,957,6881,37080,14164,6881,997,19563,1878,997,576,2922,3107,6881,730,2142,523,529,521,997,1891,1356,791,6881,730,525,523,525,3205,997,38765,770,5205,2176,6881,997,15090,856,1727,17125,540,859,953,15418,875,936,2927,602,18,28519,1550,1192,803,779,2565,39993,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":274,"marker_positions":[252,264],"positions":274,"window":512,"window_r10":512,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[1.2194080352783203,5.291305065155029],"probabilities_raw":[0.0167593602091074,0.9832406640052795],"probabilities":[0.9200953841209412,0.07990466803312302],"argmax_index":0,"argmax":"true","top2_margin":0.8401907160878181,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.9200953841209412},"usage_input_tokens":931,"oracle_file":"records_ref.json"},{"row_id":"own_j02/text/charged_claim","fixture_id":"own_j02","mode":"text","question_id":"charged_claim","question":{"type":"noul","instructions":"Does the customer say they paid for something they did not receive?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,3667,6590,6881,997,42468,522,26260,1532,1878,997,15162,6881,997,557,1983,834,917,38281,1878,997,57267,6881,997,1718,531,522,6468,522,6135,730,1133,535,2019,1878,997,26207,6881,1607,38685,38017,6881,997,556,2508,522,526,552,548,1878,997,3055,6881,997,46553,791,63437,730,526,14895,1878,997,590,1937,6881,730,527,521,997,19248,6881,997,2354,9487,511,3205,37080,38017,6881,997,549,41555,522,14819,1878,997,3055,6881,997,41388,32849,17125,730,14819,836,1878,997,590,1937,6881,730,526,521,997,19248,6881,997,1066,803,9051,511,3205,37080,38017,6881,997,12411,542,4894,17727,1878,997,3055,6881,997,26818,13192,521,730,1838,22173,1878,997,590,1937,6881,730,528,521,997,19248,6881,997,2354,9487,511,602,5086,997,2354,11468,6881,37080,8618,6265,6881,997,2482,1930,2286,1145,1878,997,19132,800,36619,6881,997,14402,1991,791,1878,997,14402,1991,791,40040,6881,997,1718,531,522,6468,522,3654,730,1632,535,2108,511,3205,997,27317,957,6881,37080,14164,6881,997,19563,1878,997,576,2922,3107,6881,730,2142,523,529,521,997,1891,1356,791,6881,730,525,523,525,3205,997,38765,770,5205,2176,6881,997,15090,856,1727,17125,540,859,953,15418,875,936,2927,602,18,29589,779,13608,2992,1186,10274,875,3503,1186,2172,1014,6984,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":279,"marker_positions":[257,269],"positions":279,"window":512,"window_r10":512,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[2.150092363357544,-0.286288321018219],"probabilities_raw":[0.9195597767829895,0.0804402306675911],"probabilities":[0.18814170360565186,0.8118582963943481],"argmax_index":1,"argmax":"false","top2_margin":0.6237165927886963,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.18814170360565186},"usage_input_tokens":931,"oracle_file":"records_ref.json"},{"row_id":"own_m01/text/bedrooms","fixture_id":"own_m01","mode":"text","question_id":"bedrooms","question":{"type":"choice","instructions":"How many bedrooms does the flat have?","criteria":{"one":null,"two":null,"three":null}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,12832,776,811,2944,1141,2642,1050,799,40315,9690,521,15398,799,12890,708,15780,1985,791,5652,9645,884,779,4976,8913,803,768,4204,916,768,12372,523,55657,730,526,521,5411,1094,5805,521,28553,7449,811,4009,6272,516,16429,521,9447,3676,5699,730,1374,4493,523,20448,988,730,526,4526,523,879,3113,938,1014,4557,6794,1883,523,873,1808,4192,968,10449,843,15344,9889,810,15622,1550,521,1295,1312,5165,1191,941,9645,856,9104,1250,2075,10325,988,768,56926,810,768,24910,7073,523,879,1445,8784,797,779,39932,536,1295,23171,3494,523,13483,1327,884,826,4306,24463,1685,730,1155,535,1056,810,730,1532,535,1056,523,22194,779,11380,521,873,3085,521,811,2643,768,25647,523,18,4023,1697,6100,14488,2177,779,9645,1052,540,19,16,1235,20,19,16,1547,20,19,16,2204,20,21],"sequence_length":164,"marker_positions":[152,156,160],"positions":164,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.658261299133301,-0.02049952745437622,-5.027771472930908],"probabilities_raw":[0.06633322685956955,0.9274629354476929,0.006203918252140284],"probabilities":[0.1287752240896225,0.8475256562232971,0.02369912900030613],"argmax_index":1,"argmax":"two","top2_margin":0.7187504321336746,"near_tie":false,"gold":"two","official_answer":{"type":"choice","choice":"two","confidence":0.8475256562232971,"probabilities":{"one":0.1287752240896225,"two":0.8475256562232971,"three":0.02369912900030613}},"usage_input_tokens":1408,"oracle_file":"records_ref.json"},{"row_id":"own_m01/text/lift","fixture_id":"own_m01","mode":"text","question_id":"lift","question":{"type":"noul","instructions":"Is there a lift?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,12832,776,811,2944,1141,2642,1050,799,40315,9690,521,15398,799,12890,708,15780,1985,791,5652,9645,884,779,4976,8913,803,768,4204,916,768,12372,523,55657,730,526,521,5411,1094,5805,521,28553,7449,811,4009,6272,516,16429,521,9447,3676,5699,730,1374,4493,523,20448,988,730,526,4526,523,879,3113,938,1014,4557,6794,1883,523,873,1808,4192,968,10449,843,15344,9889,810,15622,1550,521,1295,1312,5165,1191,941,9645,856,9104,1250,2075,10325,988,768,56926,810,768,24910,7073,523,879,1445,8784,797,779,39932,536,1295,23171,3494,523,13483,1327,884,826,4306,24463,1685,730,1155,535,1056,810,730,1532,535,1056,523,22194,779,11380,521,873,3085,521,811,2643,768,25647,523,18,4940,1413,768,12372,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":170,"marker_positions":[148,160],"positions":170,"window":256,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[0.049930721521377563,9.483137130737305],"probabilities_raw":[8.001578680705279e-05,0.9999200105667114],"probabilities":[0.9965332746505737,0.003466650377959013],"argmax_index":0,"argmax":"true","top2_margin":0.9930666242726147,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.9965332746505737},"usage_input_tokens":1408,"oracle_file":"records_ref.json"},{"row_id":"own_m01/text/furnished","fixture_id":"own_m01","mode":"text","question_id":"furnished","question":{"type":"score","instructions":"How furnished is the flat?","criteria":["unfurnished","partly furnished","fully furnished"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,12832,776,811,2944,1141,2642,1050,799,40315,9690,521,15398,799,12890,708,15780,1985,791,5652,9645,884,779,4976,8913,803,768,4204,916,768,12372,523,55657,730,526,521,5411,1094,5805,521,28553,7449,811,4009,6272,516,16429,521,9447,3676,5699,730,1374,4493,523,20448,988,730,526,4526,523,879,3113,938,1014,4557,6794,1883,523,873,1808,4192,968,10449,843,15344,9889,810,15622,1550,521,1295,1312,5165,1191,941,9645,856,9104,1250,2075,10325,988,768,56926,810,768,24910,7073,523,879,1445,8784,797,779,39932,536,1295,23171,3494,523,13483,1327,884,826,4306,24463,1685,730,1155,535,1056,810,730,1532,535,1056,523,22194,779,11380,521,873,3085,521,811,2643,768,25647,523,18,4023,60603,856,779,9645,540,19,16,2119,730,525,535,9104,1250,2075,20,19,16,2119,730,526,535,20618,60603,20,19,16,2119,730,527,535,7408,60603,20,21],"sequence_length":177,"marker_positions":[149,159,168],"positions":177,"window":256,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-0.9351052641868591,-1.015489101409912,-0.2348705232143402],"probabilities_raw":[0.25400131940841675,0.23438279330730438,0.5116159319877625],"probabilities":[0.2895609736442566,0.27641525864601135,0.43402379751205444],"argmax_index":2,"argmax":"2","top2_margin":0.14446282386779785,"near_tie":false,"gold":"0","official_answer":{"type":"score","score":1.1444628536701202,"confidence":0.43402379751205444,"probabilities":{"0":0.2895609736442566,"1":0.27641525864601135,"2":0.43402379751205444},"legend":{"0":"unfurnished","1":"partly furnished","2":"fully furnished"}},"usage_input_tokens":1408,"oracle_file":"records_ref.json"},{"row_id":"own_m01/text/rabbit","fixture_id":"own_m01","mode":"text","question_id":"rabbit","question":{"type":"choice","instructions":"Could a tenant keep a rabbit?","criteria":{"yes":"Allowed","by_agreement":"Only with the landlord's agreement","no":"Not allowed"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,12832,776,811,2944,1141,2642,1050,799,40315,9690,521,15398,799,12890,708,15780,1985,791,5652,9645,884,779,4976,8913,803,768,4204,916,768,12372,523,55657,730,526,521,5411,1094,5805,521,28553,7449,811,4009,6272,516,16429,521,9447,3676,5699,730,1374,4493,523,20448,988,730,526,4526,523,879,3113,938,1014,4557,6794,1883,523,873,1808,4192,968,10449,843,15344,9889,810,15622,1550,521,1295,1312,5165,1191,941,9645,856,9104,1250,2075,10325,988,768,56926,810,768,24910,7073,523,879,1445,8784,797,779,39932,536,1295,23171,3494,523,13483,1327,884,826,4306,24463,1685,730,1155,535,1056,810,730,1532,535,1056,523,22194,779,11380,521,873,3085,521,811,2643,768,25647,523,18,45313,768,44416,3089,768,36802,540,19,16,18672,535,2595,19226,20,19,16,968,572,890,22748,535,11005,916,779,2496,46130,1090,10449,20,19,16,1295,535,4524,6767,20,21],"sequence_length":179,"marker_positions":[150,157,172],"positions":179,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.355165719985962,1.5525476932525635,0.4870193600654602],"probabilities_raw":[0.03902411088347435,0.7147216200828552,0.24625429511070251],"probabilities":[0.07868150621652603,0.627973198890686,0.29334530234336853],"argmax_index":1,"argmax":"by_agreement","top2_margin":0.3346278965473175,"near_tie":false,"gold":"no","official_answer":{"type":"choice","choice":"by_agreement","confidence":0.627973198890686,"probabilities":{"yes":0.07868150621652603,"by_agreement":0.627973198890686,"no":0.29334530234336853}},"usage_input_tokens":1408,"oracle_file":"records_ref.json"},{"row_id":"own_m01/text/parking","fixture_id":"own_m01","mode":"text","question_id":"parking","question":{"type":"noul","instructions":"Does the flat come with a parking space?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,12832,776,811,2944,1141,2642,1050,799,40315,9690,521,15398,799,12890,708,15780,1985,791,5652,9645,884,779,4976,8913,803,768,4204,916,768,12372,523,55657,730,526,521,5411,1094,5805,521,28553,7449,811,4009,6272,516,16429,521,9447,3676,5699,730,1374,4493,523,20448,988,730,526,4526,523,879,3113,938,1014,4557,6794,1883,523,873,1808,4192,968,10449,843,15344,9889,810,15622,1550,521,1295,1312,5165,1191,941,9645,856,9104,1250,2075,10325,988,768,56926,810,768,24910,7073,523,879,1445,8784,797,779,39932,536,1295,23171,3494,523,13483,1327,884,826,4306,24463,1685,730,1155,535,1056,810,730,1532,535,1056,523,22194,779,11380,521,873,3085,521,811,2643,768,25647,523,18,29589,779,9645,2743,916,768,23171,3494,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":174,"marker_positions":[152,164],"positions":174,"window":256,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[1.846391201019287,-2.831726312637329],"probabilities_raw":[0.9907891154289246,0.009210866875946522],"probabilities":[0.056922901421785355,0.9430771470069885],"argmax_index":1,"argmax":"false","top2_margin":0.8861542455852032,"near_tie":false,"gold":"false","official_answer":{"type":"noul","noul":0.056922901421785355},"usage_input_tokens":1408,"oracle_file":"records_ref.json"},{"row_id":"own_m01/text/bill","fixture_id":"own_m01","mode":"text","question_id":"bill","question":{"type":"choice","instructions":"Which bill is included in the rent?","criteria":{"electricity":null,"gas":null,"water":null,"internet":null,"none":"No bill is included"}},"type":"choice","qtype":0,"K":5,"media":[],"prefix_rows":0,"sequence_ids":[1,17,12832,776,811,2944,1141,2642,1050,799,40315,9690,521,15398,799,12890,708,15780,1985,791,5652,9645,884,779,4976,8913,803,768,4204,916,768,12372,523,55657,730,526,521,5411,1094,5805,521,28553,7449,811,4009,6272,516,16429,521,9447,3676,5699,730,1374,4493,523,20448,988,730,526,4526,523,879,3113,938,1014,4557,6794,1883,523,873,1808,4192,968,10449,843,15344,9889,810,15622,1550,521,1295,1312,5165,1191,941,9645,856,9104,1250,2075,10325,988,768,56926,810,768,24910,7073,523,879,1445,8784,797,779,39932,536,1295,23171,3494,523,13483,1327,884,826,4306,24463,1685,730,1155,535,1056,810,730,1532,535,1056,523,22194,779,11380,521,873,3085,521,811,2643,768,25647,523,18,28828,5363,856,4557,797,779,16429,540,19,16,10366,20,19,16,4916,20,19,16,1883,20,19,16,9399,20,19,16,10735,535,3253,5363,856,4557,20,21],"sequence_length":176,"marker_positions":[151,155,159,163,167],"positions":176,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-6.876915454864502,-14.46700668334961,-6.447972774505615,-9.973302841186523,1.3173890113830566],"probabilities_raw":[0.0002760255301836878,1.3951301980341668e-07,0.00042387392022646964,1.2479739780246746e-05,0.9992875456809998],"probabilities":[0.0028496168088167906,1.2591870472533628e-05,0.0038713153917342424,0.0003120219334959984,0.9929545521736145],"argmax_index":4,"argmax":"none","top2_margin":0.9890832367818803,"near_tie":false,"gold":"water","official_answer":{"type":"choice","choice":"none","confidence":0.9929545521736145,"probabilities":{"electricity":0.0028496168088167906,"gas":1.2591870472533628e-05,"water":0.0038713153917342424,"internet":0.0003120219334959984,"none":0.9929545521736145}},"usage_input_tokens":1408,"oracle_file":"records_ref.json"},{"row_id":"own_m01/text/weekday_viewing","fixture_id":"own_m01","mode":"text","question_id":"weekday_viewing","question":{"type":"noul","instructions":"Can the flat be viewed on a weekday?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,12832,776,811,2944,1141,2642,1050,799,40315,9690,521,15398,799,12890,708,15780,1985,791,5652,9645,884,779,4976,8913,803,768,4204,916,768,12372,523,55657,730,526,521,5411,1094,5805,521,28553,7449,811,4009,6272,516,16429,521,9447,3676,5699,730,1374,4493,523,20448,988,730,526,4526,523,879,3113,938,1014,4557,6794,1883,523,873,1808,4192,968,10449,843,15344,9889,810,15622,1550,521,1295,1312,5165,1191,941,9645,856,9104,1250,2075,10325,988,768,56926,810,768,24910,7073,523,879,1445,8784,797,779,39932,536,1295,23171,3494,523,13483,1327,884,826,4306,24463,1685,730,1155,535,1056,810,730,1532,535,1056,523,22194,779,11380,521,873,3085,521,811,2643,768,25647,523,18,9036,779,9645,874,16521,884,768,3025,2367,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":175,"marker_positions":[153,165],"positions":175,"window":256,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[1.393991470336914,2.843672752380371],"probabilities_raw":[0.1900506168603897,0.8099493384361267],"probabilities":[0.7047432661056519,0.29525667428970337],"argmax_index":0,"argmax":"true","top2_margin":0.4094865918159485,"near_tie":false,"gold":"false","official_answer":{"type":"noul","noul":0.7047432661056519},"usage_input_tokens":1408,"oracle_file":"records_ref.json"},{"row_id":"own_m01/text/term","fixture_id":"own_m01","mode":"text","question_id":"term","question":{"type":"score","instructions":"How long is the minimum tenancy?","criteria":["under 6 months","6 months","12 months","over 12 months"]},"type":"score","qtype":1,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,12832,776,811,2944,1141,2642,1050,799,40315,9690,521,15398,799,12890,708,15780,1985,791,5652,9645,884,779,4976,8913,803,768,4204,916,768,12372,523,55657,730,526,521,5411,1094,5805,521,28553,7449,811,4009,6272,516,16429,521,9447,3676,5699,730,1374,4493,523,20448,988,730,526,4526,523,879,3113,938,1014,4557,6794,1883,523,873,1808,4192,968,10449,843,15344,9889,810,15622,1550,521,1295,1312,5165,1191,941,9645,856,9104,1250,2075,10325,988,768,56926,810,768,24910,7073,523,879,1445,8784,797,779,39932,536,1295,23171,3494,523,13483,1327,884,826,4306,24463,1685,730,1155,535,1056,810,730,1532,535,1056,523,22194,779,11380,521,873,3085,521,811,2643,768,25647,523,18,4023,1787,856,779,9447,3676,5699,540,19,16,2119,730,525,535,1490,730,531,4493,20,19,16,2119,730,526,535,730,531,4493,20,19,16,2119,730,527,535,730,1374,4493,20,19,16,2119,730,528,535,1391,730,1374,4493,20,21],"sequence_length":193,"marker_positions":[151,162,172,182],"positions":193,"window":256,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-3.809436559677124,-12.830514907836914,5.591951847076416,0.9847365617752075],"probabilities_raw":[8.178635471267626e-05,9.882718643439148e-09,0.9900380373001099,0.00988015253096819],"probabilities":[0.004064416512846947,2.2106480173533782e-05,0.9309849143028259,0.06492854654788971],"argmax_index":2,"argmax":"2","top2_margin":0.8660563677549362,"near_tie":false,"gold":"2","official_answer":{"type":"score","score":2.0567775747294945,"confidence":0.9309849143028259,"probabilities":{"0":0.004064416512846947,"1":2.2106480173533782e-05,"2":0.9309849143028259,"3":0.06492854654788971},"legend":{"0":"under 6 months","1":"6 months","2":"12 months","3":"over 12 months"}},"usage_input_tokens":1408,"oracle_file":"records_ref.json"},{"row_id":"own_t01/text/team","fixture_id":"own_t01","mode":"text","question_id":"team","question":{"type":"choice","instructions":"Which team should handle this ticket first?","criteria":{"billing":"Charges, invoices and refunds","deliveries":"Boxes, couriers and delivery dates","account":"Sign-in, plan changes and pauses","technical":"App or website faults","other":null}},"type":"choice","qtype":0,"K":5,"media":[],"prefix_rows":0,"sequence_ids":[1,17,561,32909,2679,31464,526,1141,8414,730,1718,531,522,6135,522,1536,730,4905,535,1374,708,6314,535,50642,843,38765,770,2908,730,1718,528,1015,41518,535,61958,791,11126,875,4690,509,42120,521,859,6217,884,779,9855,834,917,3016,786,984,1512,19771,2984,523,4411,5060,8294,4427,1547,16620,803,730,2752,523,2475,884,4690,730,526,521,1203,859,1550,5068,1235,6377,523,859,4001,45942,1727,7493,810,1186,5689,1123,811,3452,1010,1509,523,13831,53467,779,49006,7786,523,859,6351,1236,1672,811,36298,5514,969,875,1547,6272,2158,859,6217,35754,521,7929,4690,730,898,523,20018,521,50642,18,28828,3410,1753,9936,1033,30437,1509,540,19,16,5363,800,535,61958,774,521,60827,1864,810,53467,592,20,19,16,5514,969,535,16208,774,521,2286,3670,810,11746,14721,20,19,16,3863,535,11309,4086,521,2984,3807,810,784,20231,20,19,16,10017,535,5135,933,5647,50022,20,19,16,1312,20,21],"sequence_length":187,"marker_positions":[133,147,161,174,183],"positions":187,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[5.459228515625,-0.8426796197891235,1.1409907341003418,-3.7750244140625,-2.193328619003296],"probabilities_raw":[0.9845149517059326,0.0018044234020635486,0.013117029331624508,9.612519352231175e-05,0.00046747553278692067],"probabilities":[0.9412418603897095,0.010438592173159122,0.04305680841207504,0.0012850860366597772,0.0039776465855538845],"argmax_index":0,"argmax":"billing","top2_margin":0.8981850519776344,"near_tie":false,"gold":"billing","official_answer":{"type":"choice","choice":"billing","confidence":0.9412418603897095,"probabilities":{"billing":0.9412418603897095,"deliveries":0.010438592173159122,"account":0.04305680841207504,"technical":0.0012850860366597772,"other":0.0039776465855538845}},"usage_input_tokens":660,"oracle_file":"records_ref.json"},{"row_id":"own_t01/text/several_actions","fixture_id":"own_t01","mode":"text","question_id":"several_actions","question":{"type":"noul","instructions":"Does the ticket ask for more than one action?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,561,32909,2679,31464,526,1141,8414,730,1718,531,522,6135,522,1536,730,4905,535,1374,708,6314,535,50642,843,38765,770,2908,730,1718,528,1015,41518,535,61958,791,11126,875,4690,509,42120,521,859,6217,884,779,9855,834,917,3016,786,984,1512,19771,2984,523,4411,5060,8294,4427,1547,16620,803,730,2752,523,2475,884,4690,730,526,521,1203,859,1550,5068,1235,6377,523,859,4001,45942,1727,7493,810,1186,5689,1123,811,3452,1010,1509,523,13831,53467,779,49006,7786,523,859,6351,1236,1672,811,36298,5514,969,875,1547,6272,2158,859,6217,35754,521,7929,4690,730,898,523,20018,521,50642,18,29589,779,30437,3452,875,1209,1399,1235,4272,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":157,"marker_positions":[135,147],"positions":157,"window":256,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[-1.5088059902191162,2.8384313583374023],"probabilities_raw":[0.012777145951986313,0.9872227907180786],"probabilities":[0.931430995464325,0.06856900453567505],"argmax_index":0,"argmax":"true","top2_margin":0.8628619909286499,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.931430995464325},"usage_input_tokens":660,"oracle_file":"records_ref.json"},{"row_id":"own_t01/text/upset","fixture_id":"own_t01","mode":"text","question_id":"upset","question":{"type":"score","instructions":"How upset is the customer?","criteria":["calm","mildly annoyed","frustrated","angry"]},"type":"score","qtype":1,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,561,32909,2679,31464,526,1141,8414,730,1718,531,522,6135,522,1536,730,4905,535,1374,708,6314,535,50642,843,38765,770,2908,730,1718,528,1015,41518,535,61958,791,11126,875,4690,509,42120,521,859,6217,884,779,9855,834,917,3016,786,984,1512,19771,2984,523,4411,5060,8294,4427,1547,16620,803,730,2752,523,2475,884,4690,730,526,521,1203,859,1550,5068,1235,6377,523,859,4001,45942,1727,7493,810,1186,5689,1123,811,3452,1010,1509,523,13831,53467,779,49006,7786,523,859,6351,1236,1672,811,36298,5514,969,875,1547,6272,2158,859,6217,35754,521,7929,4690,730,898,523,20018,521,50642,18,4023,30133,856,779,13608,540,19,16,2119,730,525,535,19243,20,19,16,2119,730,526,535,13545,876,35740,791,20,19,16,2119,730,527,535,44658,20,19,16,2119,730,528,535,23732,20,21],"sequence_length":166,"marker_positions":[131,139,150,158],"positions":166,"window":256,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-0.5534851551055908,-2.147355318069458,-0.1437389999628067,-2.5713460445404053],"probabilities_raw":[0.35180142521858215,0.07146420329809189,0.5299661159515381,0.046768318861722946],"probabilities":[0.3359356224536896,0.13370925188064575,0.4257071018218994,0.10464800149202347],"argmax_index":2,"argmax":"2","top2_margin":0.08977147936820984,"near_tie":false,"gold":"1","official_answer":{"type":"score","score":1.299067460000515,"confidence":0.4257071018218994,"probabilities":{"0":0.3359356224536896,"1":0.13370925188064575,"2":0.4257071018218994,"3":0.10464800149202347},"legend":{"0":"calm","1":"mildly annoyed","2":"frustrated","3":"angry"}},"usage_input_tokens":660,"oracle_file":"records_ref.json"},{"row_id":"own_t01/text/refund","fixture_id":"own_t01","mode":"text","question_id":"refund","question":{"type":"noul","instructions":"Refund request?","criteria":{"true":"The customer explicitly asks for money back","false":"No refund is requested"}},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,561,32909,2679,31464,526,1141,8414,730,1718,531,522,6135,522,1536,730,4905,535,1374,708,6314,535,50642,843,38765,770,2908,730,1718,528,1015,41518,535,61958,791,11126,875,4690,509,42120,521,859,6217,884,779,9855,834,917,3016,786,984,1512,19771,2984,523,4411,5060,8294,4427,1547,16620,803,730,2752,523,2475,884,4690,730,526,521,1203,859,1550,5068,1235,6377,523,859,4001,45942,1727,7493,810,1186,5689,1123,811,3452,1010,1509,523,13831,53467,779,49006,7786,523,859,6351,1236,1672,811,36298,5514,969,875,1547,6272,2158,859,6217,35754,521,7929,4690,730,898,523,20018,521,50642,18,9906,1356,5940,540,19,16,4831,535,3253,53467,856,21968,20,19,16,3347,535,941,13608,22922,21090,875,4636,1892,20,21],"sequence_length":150,"marker_positions":[129,138],"positions":150,"window":256,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[-5.6721014976501465,12.44875717163086],"probabilities_raw":[1.3496180173433459e-08,1.0],"probabilities":[0.9999810457229614,1.8929660654976033e-05],"argmax_index":0,"argmax":"true","top2_margin":0.9999621160623064,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.9999810457229614},"usage_input_tokens":660,"oracle_file":"records_ref.json"},{"row_id":"own_t04/text/crash_owner","fixture_id":"own_t04","mode":"text","question_id":"crash_owner","question":{"type":"choice","instructions":"Who owns the login crash?","criteria":{"aiko":"Aiko","ben":"Ben","chloe":"Chloé","femi":"Femi"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,7491,1983,1395,57394,1141,9573,3410,2472,6688,521,22153,708,41678,535,835,14007,521,6623,521,1157,1837,860,521,911,14851,708,542,14007,535,10414,779,49550,39198,8660,536,13668,884,2289,4202,2076,61590,819,30819,535,779,21064,884,33990,875,18204,730,1374,7039,856,2510,2714,536,950,1011,30961,936,1550,884,1235,1927,9161,523,18578,791,3032,779,6625,5756,12116,779,2341,9161,884,16012,819,2435,1837,860,535,6872,8788,875,730,529,523,527,43360,536,3988,5743,988,911,14851,819,547,14851,535,41669,13793,9167,536,1188,8685,1157,1837,860,779,19771,5592,5196,5743,4008,819,15888,6015,535,6872,730,529,523,527,14967,988,22045,811,17980,1929,803,779,33990,21064,523,6623,40917,779,21064,536,835,14007,1188,9256,916,1889,27968,523,18,17408,40917,779,33990,21064,540,19,16,768,14007,535,835,14007,20,19,16,7572,535,6623,20,19,16,979,47222,535,1157,1837,860,20,19,16,3783,582,535,911,14851,20,21],"sequence_length":191,"marker_positions":[160,168,174,183],"positions":191,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.1562695503234863,-0.11443820595741272,-2.6990933418273926,-4.24807596206665],"probabilities_raw":[0.2442837953567505,0.692398726940155,0.05222216248512268,0.011095328256487846],"probabilities":[0.2819421887397766,0.593431293964386,0.09365379065275192,0.030972765758633614],"argmax_index":1,"argmax":"ben","top2_margin":0.3114891052246094,"near_tie":false,"gold":"ben","official_answer":{"type":"choice","choice":"ben","confidence":0.593431293964386,"probabilities":{"aiko":0.2819421887397766,"ben":0.593431293964386,"chloe":0.09365379065275192,"femi":0.030972765758633614}},"usage_input_tokens":753,"oracle_file":"records_ref.json"},{"row_id":"own_t04/text/blocked","fixture_id":"own_t04","mode":"text","question_id":"blocked","question":{"type":"noul","instructions":"Is any team member blocked?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,7491,1983,1395,57394,1141,9573,3410,2472,6688,521,22153,708,41678,535,835,14007,521,6623,521,1157,1837,860,521,911,14851,708,542,14007,535,10414,779,49550,39198,8660,536,13668,884,2289,4202,2076,61590,819,30819,535,779,21064,884,33990,875,18204,730,1374,7039,856,2510,2714,536,950,1011,30961,936,1550,884,1235,1927,9161,523,18578,791,3032,779,6625,5756,12116,779,2341,9161,884,16012,819,2435,1837,860,535,6872,8788,875,730,529,523,527,43360,536,3988,5743,988,911,14851,819,547,14851,535,41669,13793,9167,536,1188,8685,1157,1837,860,779,19771,5592,5196,5743,4008,819,15888,6015,535,6872,730,529,523,527,14967,988,22045,811,17980,1929,803,779,33990,21064,523,6623,40917,779,21064,536,835,14007,1188,9256,916,1889,27968,523,18,4940,1523,3410,5484,26781,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":182,"marker_positions":[160,172],"positions":182,"window":256,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[2.0561749935150146,-0.3539355993270874],"probabilities_raw":[0.9175950288772583,0.0824049636721611],"probabilities":[0.19056159257888794,0.8094383478164673],"argmax_index":1,"argmax":"false","top2_margin":0.6188767552375793,"near_tie":false,"gold":"true","official_answer":{"type":"noul","noul":0.19056159257888794},"usage_input_tokens":753,"oracle_file":"records_ref.json"},{"row_id":"own_t04/text/release_day","fixture_id":"own_t04","mode":"text","question_id":"release_day","question":{"type":"choice","instructions":"When is release 4.2 now planned?","criteria":{"thursday":"Thursday","friday":"Friday","monday":"Monday","unknown":"Not decided"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,7491,1983,1395,57394,1141,9573,3410,2472,6688,521,22153,708,41678,535,835,14007,521,6623,521,1157,1837,860,521,911,14851,708,542,14007,535,10414,779,49550,39198,8660,536,13668,884,2289,4202,2076,61590,819,30819,535,779,21064,884,33990,875,18204,730,1374,7039,856,2510,2714,536,950,1011,30961,936,1550,884,1235,1927,9161,523,18578,791,3032,779,6625,5756,12116,779,2341,9161,884,16012,819,2435,1837,860,535,6872,8788,875,730,529,523,527,43360,536,3988,5743,988,911,14851,819,547,14851,535,41669,13793,9167,536,1188,8685,1157,1837,860,779,19771,5592,5196,5743,4008,819,15888,6015,535,6872,730,529,523,527,14967,988,22045,811,17980,1929,803,779,33990,21064,523,6623,40917,779,21064,536,835,14007,1188,9256,916,1889,27968,523,18,4453,856,6872,730,529,523,527,2166,13573,540,19,16,823,18908,535,22045,20,19,16,2261,9008,535,16012,20,19,16,10954,892,535,17980,20,19,16,10654,535,4524,8091,20,21],"sequence_length":192,"marker_positions":[164,171,178,185],"positions":192,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.7694559693336487,-1.497985601425171,4.049992561340332,-2.367877960205078],"probabilities_raw":[0.007962951436638832,0.0038430634886026382,0.9865837693214417,0.0016102311201393604],"probabilities":[0.030133897438645363,0.01790776662528515,0.9423384070396423,0.009619934484362602],"argmax_index":2,"argmax":"monday","top2_margin":0.912204509600997,"near_tie":false,"gold":"monday","official_answer":{"type":"choice","choice":"monday","confidence":0.9423384070396423,"probabilities":{"thursday":0.030133897438645363,"friday":0.01790776662528515,"monday":0.9423384070396423,"unknown":0.009619934484362602}},"usage_input_tokens":753,"oracle_file":"records_ref.json"},{"row_id":"own_t04/text/blockers","fixture_id":"own_t04","mode":"text","question_id":"blockers","question":{"type":"score","instructions":"How many people report a blocker?","criteria":["none","one","two or more"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,7491,1983,1395,57394,1141,9573,3410,2472,6688,521,22153,708,41678,535,835,14007,521,6623,521,1157,1837,860,521,911,14851,708,542,14007,535,10414,779,49550,39198,8660,536,13668,884,2289,4202,2076,61590,819,30819,535,779,21064,884,33990,875,18204,730,1374,7039,856,2510,2714,536,950,1011,30961,936,1550,884,1235,1927,9161,523,18578,791,3032,779,6625,5756,12116,779,2341,9161,884,16012,819,2435,1837,860,535,6872,8788,875,730,529,523,527,43360,536,3988,5743,988,911,14851,819,547,14851,535,41669,13793,9167,536,1188,8685,1157,1837,860,779,19771,5592,5196,5743,4008,819,15888,6015,535,6872,730,529,523,527,14967,988,22045,811,17980,1929,803,779,33990,21064,523,6623,40917,779,21064,536,835,14007,1188,9256,916,1889,27968,523,18,4023,1697,1519,2312,768,4409,770,540,19,16,2119,730,525,535,10735,20,19,16,2119,730,526,535,1235,20,19,16,2119,730,527,535,1547,933,1209,20,21],"sequence_length":188,"marker_positions":[162,170,178],"positions":188,"window":256,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-2.957319974899292,-4.160699367523193,-2.986135482788086],"probabilities_raw":[0.44018465280532837,0.13213378190994263,0.42768150568008423],"probabilities":[0.40285491943359375,0.20094417035579681,0.39620086550712585],"argmax_index":0,"argmax":"0","top2_margin":0.0066540539264678955,"near_tie":true,"gold":"1","official_answer":{"type":"score","score":0.9933459013700485,"confidence":0.40285491943359375,"probabilities":{"0":0.40285491943359375,"1":0.20094417035579681,"2":0.39620086550712585},"legend":{"0":"none","1":"one","2":"two or more"}},"usage_input_tokens":753,"oracle_file":"records_ref.json"},{"row_id":"own_t06/text/first_request","fixture_id":"own_t06","mode":"text","question_id":"first_request","question":{"type":"choice","instructions":"What did the guest ask for first?","criteria":{"room_service":"Food or drinks to the room","late_checkout":"Leave the room later than usual","towels":"Extra towels or linen","maintenance":"Something in the room is broken","billing":"A question about the bill","booking_change":"Change the dates or room type","lost_item":"Something was lost","noise":"A noise complaint","transport":"Help with a taxi or train","other":null}},"type":"choice","qtype":0,"K":10,"media":[],"prefix_rows":0,"sequence_ids":[1,17,568,913,535,5843,570,54053,843,5652,730,25454,3271,15012,521,779,16542,797,1727,5362,7404,1901,3822,810,936,1090,34517,708,568,913,535,6135,570,13660,26862,843,564,959,54898,2176,13901,3271,859,6217,29640,811,8498,896,510,859,6016,8685,5791,988,12003,2084,3599,819,568,913,535,6326,570,54053,535,14165,523,1236,1011,859,3089,779,5362,3032,730,527,5806,27968,540,1727,8842,856,963,730,528,708,568,913,535,6468,570,13660,26862,535,25957,64210,3032,730,1536,535,1056,856,3023,875,730,898,14076,521,6283,859,1614,936,1784,568,913,535,6468,570,54053,535,18672,8694,708,568,913,535,1698,570,54053,535,2510,31731,4010,810,936,1090,5629,52125,708,568,913,535,2117,570,13660,26862,535,4998,6650,521,1494,59063,856,884,2772,1868,523,1405,1188,874,916,1010,797,730,1155,5112,521,810,1023,1011,4876,1010,811,5362,730,26740,1288,1010,6863,523,18,3493,2172,779,21522,3452,875,1509,540,19,16,5362,52799,535,9267,933,20056,811,779,5362,20,19,16,5019,45625,1066,535,41072,779,5362,2919,1399,13376,20,19,16,25583,1706,535,38332,25583,1706,933,56043,20,19,16,12003,535,39781,797,779,5362,856,11694,20,19,16,5363,800,535,835,3952,1337,779,5363,20,19,16,2643,800,572,6386,535,15015,779,14721,933,5362,2432,20,19,16,5807,43840,535,39781,953,5807,20,19,16,10925,535,835,10925,38203,20,19,16,5600,535,18525,916,768,47502,933,8842,20,19,16,1312,20,21],"sequence_length":294,"marker_positions":[190,202,215,226,237,248,262,271,279,290],"positions":294,"window":512,"window_r10":512,"temperature_key":"choice:6-10","temperature":1.1751071214675903,"raw_logits":[-3.301970958709717,-0.8134485483169556,-5.719726085662842,1.0345563888549805,-6.834534168243408,-2.957244634628296,-8.278870582580566,-11.795267105102539,-2.055696725845337,-1.509829044342041],"probabilities_raw":[0.009950069710612297,0.11983334273099899,0.000886764726601541,0.7605994939804077,0.00029083952540531754,0.014045554213225842,6.860979192424566e-05,2.0381423837534385e-06,0.03460000455379486,0.05972325801849365],"probabilities":[0.01712716370820999,0.14236073195934296,0.0021884406451135874,0.6860777735710144,0.000847472867462784,0.022966191172599792,0.0002479304384905845,1.2437837540346663e-05,0.04946316033601761,0.0787087231874466],"argmax_index":3,"argmax":"maintenance","top2_margin":0.5437170416116714,"near_tie":false,"gold":"maintenance","official_answer":{"type":"choice","choice":"maintenance","confidence":0.6860777735710144,"probabilities":{"room_service":0.01712716370820999,"late_checkout":0.14236073195934296,"towels":0.0021884406451135874,"maintenance":0.6860777735710144,"billing":0.000847472867462784,"booking_change":0.022966191172599792,"lost_item":0.0002479304384905845,"noise":1.2437837540346663e-05,"transport":0.04946316033601761,"other":0.0787087231874466}},"usage_input_tokens":727,"oracle_file":"records_ref.json"},{"row_id":"own_t06/text/fixed","fixture_id":"own_t06","mode":"text","question_id":"fixed","question":{"type":"noul","instructions":"Has the heating problem been fixed by the end of the chat?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,568,913,535,5843,570,54053,843,5652,730,25454,3271,15012,521,779,16542,797,1727,5362,7404,1901,3822,810,936,1090,34517,708,568,913,535,6135,570,13660,26862,843,564,959,54898,2176,13901,3271,859,6217,29640,811,8498,896,510,859,6016,8685,5791,988,12003,2084,3599,819,568,913,535,6326,570,54053,535,14165,523,1236,1011,859,3089,779,5362,3032,730,527,5806,27968,540,1727,8842,856,963,730,528,708,568,913,535,6468,570,13660,26862,535,25957,64210,3032,730,1536,535,1056,856,3023,875,730,898,14076,521,6283,859,1614,936,1784,568,913,535,6468,570,54053,535,18672,8694,708,568,913,535,1698,570,54053,535,2510,31731,4010,810,936,1090,5629,52125,708,568,913,535,2117,570,13660,26862,535,4998,6650,521,1494,59063,856,884,2772,1868,523,1405,1188,874,916,1010,797,730,1155,5112,521,810,1023,1011,4876,1010,811,5362,730,26740,1288,1010,6863,523,18,24860,779,16542,3220,1381,9167,968,779,1764,803,779,22537,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":217,"marker_positions":[195,207],"positions":217,"window":256,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[2.7867722511291504,-0.9277684688568115],"probabilities_raw":[0.9762129783630371,0.02378701977431774],"probabilities":[0.09716012328863144,0.9028398990631104],"argmax_index":1,"argmax":"false","top2_margin":0.8056797757744789,"near_tie":false,"gold":"false","official_answer":{"type":"noul","noul":0.09716012328863144},"usage_input_tokens":727,"oracle_file":"records_ref.json"},{"row_id":"own_t06/text/patience","fixture_id":"own_t06","mode":"text","question_id":"patience","question":{"type":"score","instructions":"How patient is the guest by the end?","criteria":["patient","impatient","angry"]},"type":"score","qtype":1,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,568,913,535,5843,570,54053,843,5652,730,25454,3271,15012,521,779,16542,797,1727,5362,7404,1901,3822,810,936,1090,34517,708,568,913,535,6135,570,13660,26862,843,564,959,54898,2176,13901,3271,859,6217,29640,811,8498,896,510,859,6016,8685,5791,988,12003,2084,3599,819,568,913,535,6326,570,54053,535,14165,523,1236,1011,859,3089,779,5362,3032,730,527,5806,27968,540,1727,8842,856,963,730,528,708,568,913,535,6468,570,13660,26862,535,25957,64210,3032,730,1536,535,1056,856,3023,875,730,898,14076,521,6283,859,1614,936,1784,568,913,535,6468,570,54053,535,18672,8694,708,568,913,535,1698,570,54053,535,2510,31731,4010,810,936,1090,5629,52125,708,568,913,535,2117,570,13660,26862,535,4998,6650,521,1494,59063,856,884,2772,1868,523,1405,1188,874,916,1010,797,730,1155,5112,521,810,1023,1011,4876,1010,811,5362,730,26740,1288,1010,6863,523,18,4023,5087,856,779,21522,968,779,1764,540,19,16,2119,730,525,535,5087,20,19,16,2119,730,526,535,2269,51908,20,19,16,2119,730,527,535,23732,20,21],"sequence_length":216,"marker_positions":[191,199,208],"positions":216,"window":256,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-0.20327231287956238,-2.439265489578247,-3.508890390396118],"probabilities_raw":[0.8744603395462036,0.09346750378608704,0.03207217901945114],"probabilities":[0.7029393911361694,0.19303540885448456,0.10402525961399078],"argmax_index":0,"argmax":"0","top2_margin":0.5099039822816849,"near_tie":false,"gold":"1","official_answer":{"type":"score","score":0.4010859280824661,"confidence":0.7029393911361694,"probabilities":{"0":0.7029393911361694,"1":0.19303540885448456,"2":0.10402525961399078},"legend":{"0":"patient","1":"impatient","2":"angry"}},"usage_input_tokens":727,"oracle_file":"records_ref.json"},{"row_id":"own_t07/text/slot","fixture_id":"own_t07","mode":"text","question_id":"slot","question":{"type":"choice","instructions":"Which of these free slots fits the caller?","criteria":{"tue_1100":"Tuesday 11:00","wed_1530":"Wednesday 15:30","thu_1230":"Thursday 12:30","none":"None of them fits"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,556,820,799,12890,33496,1141,2692,793,18284,19094,521,16012,730,1443,535,3167,708,13694,5052,521,1033,856,54017,523,859,1052,768,16084,20328,884,22092,963,730,534,535,1664,916,3583,2412,786,523,39781,4010,1287,963,1336,521,1101,859,1595,811,4876,936,523,859,1011,1112,22153,1720,730,1386,535,1056,933,1523,1394,884,22045,8532,523,13831,1868,1123,1892,811,10547,523,859,1011,1901,3999,12645,2013,779,2285,521,1101,1720,730,1133,535,1056,856,2238,523,28296,1010,12588,22935,42474,3280,3025,535,837,1037,730,1632,535,1056,521,16666,730,1386,535,1664,521,16666,730,1417,535,1056,521,1097,594,730,4528,535,1664,521,1097,594,730,1374,535,1664,523,18,28828,803,1518,3262,42474,25113,779,59850,540,19,16,766,1037,572,7984,525,535,22092,730,1632,535,1056,20,19,16,28782,572,11610,525,535,22153,730,1386,535,1664,20,19,16,823,594,572,10293,525,535,22045,730,1374,535,1664,20,19,16,10735,535,11357,803,1377,25113,20,21],"sequence_length":199,"marker_positions":[149,163,176,190],"positions":199,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.19124549627304077,-1.5787943601608276,1.2088134288787842,-0.8737671971321106],"probabilities_raw":[0.23356255888938904,0.039781827479600906,0.6461410522460938,0.08051449805498123],"probabilities":[0.2618938982486725,0.07396066188812256,0.5417618751525879,0.12238364666700363],"argmax_index":2,"argmax":"thu_1230","top2_margin":0.2798679769039154,"near_tie":false,"gold":"wed_1530","official_answer":{"type":"choice","choice":"thu_1230","confidence":0.5417618751525879,"probabilities":{"tue_1100":0.2618938982486725,"wed_1530":0.07396066188812256,"thu_1230":0.5417618751525879,"none":0.12238364666700363}},"usage_input_tokens":199,"oracle_file":"records_ref.json"},{"row_id":"own_t10/text/topic","fixture_id":"own_t10","mode":"text","question_id":"topic","question":{"type":"choice","instructions":"What is the letter about?","criteria":{"noise":"Noise","repairs":"Repairs","rent":"Rent or charges","parking":"Parking","pets":"Pets"}},"type":"choice","qtype":0,"K":5,"media":[],"prefix_rows":0,"sequence_ids":[1,17,3097,779,17612,11380,521,9855,834,917,7920,509,550,4238,797,9645,730,534,810,859,1283,4862,521,875,779,4976,1394,1033,5805,521,1337,779,10925,988,779,21780,7494,797,779,39932,523,941,12719,938,1593,3032,1720,44396,521,810,779,823,51143,4834,1287,1626,779,8913,803,1727,32874,523,859,1052,5751,875,779,10170,4934,811,874,6235,811,730,4972,535,1056,522,2019,535,1056,521,906,779,3622,7360,2992,523,2182,6300,3807,968,779,1764,803,3280,3025,521,859,1188,9920,936,916,779,10698,516,15671,523,509,566,2670,521,2161,887,3309,843,40723,730,534,518,18,3493,856,779,7627,1337,540,19,16,10925,535,63144,20,19,16,38183,535,4274,5569,20,19,16,16429,535,55657,933,16620,20,19,16,23171,535,4514,800,20,19,16,22906,535,873,1808,20,21],"sequence_length":163,"marker_positions":[128,134,141,149,156],"positions":163,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[6.030937671661377,-7.381817817687988,-4.890927314758301,-10.31625747680664,-10.759885787963867],"probabilities_raw":[0.9999804496765137,1.495910510129761e-06,1.805866304493975e-05,7.952342429007331e-08,5.103049005583671e-08],"probabilities":[0.9995075464248657,6.897762068547308e-05,0.00040875584818422794,8.47906540002441e-06,6.176179795147618e-06],"argmax_index":0,"argmax":"noise","top2_margin":0.9990987905766815,"near_tie":false,"gold":"noise","official_answer":{"type":"choice","choice":"noise","confidence":0.9995075464248657,"probabilities":{"noise":0.9995075464248657,"repairs":6.897762068547308e-05,"rent":0.00040875584818422794,"parking":8.47906540002441e-06,"pets":6.176179795147618e-06}},"usage_input_tokens":490,"oracle_file":"records_ref.json"},{"row_id":"own_t10/text/first_time","fixture_id":"own_t10","mode":"text","question_id":"first_time","question":{"type":"noul","instructions":"Is this the first time the writer complains?"},"type":"noul","qtype":2,"K":2,"media":[],"prefix_rows":0,"sequence_ids":[1,17,3097,779,17612,11380,521,9855,834,917,7920,509,550,4238,797,9645,730,534,810,859,1283,4862,521,875,779,4976,1394,1033,5805,521,1337,779,10925,988,779,21780,7494,797,779,39932,523,941,12719,938,1593,3032,1720,44396,521,810,779,823,51143,4834,1287,1626,779,8913,803,1727,32874,523,859,1052,5751,875,779,10170,4934,811,874,6235,811,730,4972,535,1056,522,2019,535,1056,521,906,779,3622,7360,2992,523,2182,6300,3807,968,779,1764,803,3280,3025,521,859,1188,9920,936,916,779,10698,516,15671,523,509,566,2670,521,2161,887,3309,843,40723,730,534,518,18,4940,1033,779,1509,1394,779,9978,6537,2190,540,19,16,4831,535,1295,521,779,8294,2177,1014,3965,20,19,16,3347,535,18672,521,779,8294,10995,20,21],"sequence_length":154,"marker_positions":[132,144],"positions":154,"window":256,"window_r10":256,"temperature_key":"noul:2","temperature":1.6663223505020142,"raw_logits":[-0.21387678384780884,4.600820541381836],"probabilities_raw":[0.008044439367949963,0.991955578327179],"probabilities":[0.9473206996917725,0.05267929285764694],"argmax_index":0,"argmax":"true","top2_margin":0.8946414068341255,"near_tie":false,"gold":"false","official_answer":{"type":"noul","noul":0.9473206996917725},"usage_input_tokens":490,"oracle_file":"records_ref.json"},{"row_id":"own_t10/text/escalation","fixture_id":"own_t10","mode":"text","question_id":"escalation","question":{"type":"score","instructions":"How far is the writer prepared to take this?","criteria":["no further step","a reminder","the residents' committee","a lawyer or court"]},"type":"score","qtype":1,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,3097,779,17612,11380,521,9855,834,917,7920,509,550,4238,797,9645,730,534,810,859,1283,4862,521,875,779,4976,1394,1033,5805,521,1337,779,10925,988,779,21780,7494,797,779,39932,523,941,12719,938,1593,3032,1720,44396,521,810,779,823,51143,4834,1287,1626,779,8913,803,1727,32874,523,859,1052,5751,875,779,10170,4934,811,874,6235,811,730,4972,535,1056,522,2019,535,1056,521,906,779,3622,7360,2992,523,2182,6300,3807,968,779,1764,803,3280,3025,521,859,1188,9920,936,916,779,10698,516,15671,523,509,566,2670,521,2161,887,3309,843,40723,730,534,518,18,4023,3414,856,779,9978,9217,811,2337,1033,540,19,16,2119,730,525,535,1295,3759,3801,20,19,16,2119,730,526,535,768,36685,20,19,16,2119,730,527,535,779,10698,516,15671,20,19,16,2119,730,528,535,768,23749,933,7326,20,21],"sequence_length":173,"marker_positions":[132,142,151,162],"positions":173,"window":256,"window_r10":256,"temperature_key":"score:3-5","temperature":1.7301132678985596,"raw_logits":[-2.038007974624634,-1.6411914825439453,4.013549327850342,-8.809741973876953],"probabilities_raw":[0.0023404823150485754,0.003480492625385523,0.9941763877868652,2.681505065993406e-06],"probabilities":[0.02831273339688778,0.035611558705568314,0.9355105757713318,0.000565113325137645],"argmax_index":2,"argmax":"2","top2_margin":0.8998990170657635,"near_tie":false,"gold":"2","official_answer":{"type":"score","score":1.9083280502236448,"confidence":0.9355105757713318,"probabilities":{"0":0.02831273339688778,"1":0.035611558705568314,"2":0.9355105757713318,"3":0.000565113325137645},"legend":{"0":"no further step","1":"a reminder","2":"the residents' committee","3":"a lawyer or court"}},"usage_input_tokens":490,"oracle_file":"records_ref.json"},{"row_id":"semif_005dcf6d2c0f21d64796/text/decision","fixture_id":"semif_005dcf6d2c0f21d64796","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Use the homeowner's latest permission decision; a withdrawal cancels the earlier grant. May the surveyor enter the attic?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,2681,28117,1509,6767,768,8153,777,811,7157,779,1530,793,523,13629,779,5089,1689,22922,46159,896,15001,523,3253,2919,6177,9647,523,18,14875,779,2681,28117,1090,10500,15001,6177,536,768,28687,32491,1706,779,7045,8825,523,3354,779,8153,777,7157,779,1530,793,540,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":98,"marker_positions":[58,69,81],"positions":98,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.0582985877990723,1.225986361503601,2.4099020957946777],"probabilities_raw":[0.3500926196575165,0.15230506658554077,0.49760234355926514],"probabilities":[0.35244399309158325,0.19448202848434448,0.4530739486217499],"argmax_index":2,"argmax":"insufficient","top2_margin":0.10062995553016663,"near_tie":false,"gold":"prohibited","official_answer":{"type":"choice","choice":"insufficient","confidence":0.4530739486217499,"probabilities":{"permitted":0.35244399309158325,"prohibited":0.19448202848434448,"insufficient":0.4530739486217499}},"usage_input_tokens":98,"oracle_file":"records_ref.json"},{"row_id":"semif_00bb4cc043f7d2b288d4/text/decision","fixture_id":"semif_00bb4cc043f7d2b288d4","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the recovery email be changed?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,12153,4943,876,22558,9365,779,11256,8355,523,18,18693,902,3863,2853,1550,1403,779,12153,3575,6229,9365,896,2853,523,3354,779,11256,8355,874,7093,540,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,21],"sequence_length":75,"marker_positions":[35,52,64],"positions":75,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.306075096130371,0.6486483812332153,4.865217208862305],"probabilities_raw":[0.07084484398365021,0.013505072332918644,0.9156500697135925],"probabilities":[0.1328362673521042,0.04065634310245514,0.8265073895454407],"argmax_index":2,"argmax":"permitted","top2_margin":0.6936711221933365,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"permitted","confidence":0.8265073895454407,"probabilities":{"insufficient":0.1328362673521042,"prohibited":0.04065634310245514,"permitted":0.8265073895454407}},"usage_input_tokens":75,"oracle_file":"records_ref.json"},{"row_id":"semif_02f2b95a5c5a4777a45c/text/decision","fixture_id":"semif_02f2b95a5c5a4777a45c","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose A whenever A is able; choose B only when A is unable and B is able.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,856,779,16495,57490,810,856,11649,3222,811,9936,1033,3869,523,46050,931,879,856,779,3981,3321,57490,810,856,1236,3222,811,9936,936,523,18,52739,835,19148,835,856,3222,536,7056,879,1550,1403,835,856,12359,810,879,856,3222,523,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,21],"sequence_length":82,"marker_positions":[55,63,71],"positions":82,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.056809425354004,1.9211969375610352,0.8836579918861389],"probabilities_raw":[0.4581752121448517,0.40006986260414124,0.14175499975681305],"probabilities":[0.4273075461387634,0.38785475492477417,0.18483762443065643],"argmax_index":0,"argmax":"B","top2_margin":0.03945279121398926,"near_tie":false,"gold":"A","official_answer":{"type":"choice","choice":"B","confidence":0.4273075461387634,"probabilities":{"B":0.4273075461387634,"A":0.38785475492477417,"insufficient":0.18483762443065643}},"usage_input_tokens":82,"oracle_file":"records_ref.json"},{"row_id":"semif_03d919fd0e9b3a7cd1e6/text/decision","fixture_id":"semif_03d919fd0e9b3a7cd1e6","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the corrected date is in July.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,40348,35283,768,5562,23819,1203,6602,1295,35101,5562,523,18,9886,894,779,4673,535,779,35101,5562,856,797,5105,523,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":60,"marker_positions":[28,39,49],"positions":60,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.0700466632843018,-3.3887336254119873,-0.8867999315261841],"probabilities_raw":[0.9467533826828003,0.004031983204185963,0.0492146797478199],"probabilities":[0.8762499690055847,0.01774740032851696,0.10600268840789795],"argmax_index":0,"argmax":"insufficient","top2_margin":0.7702472805976868,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.8762499690055847,"probabilities":{"insufficient":0.8762499690055847,"supported":0.01774740032851696,"contradicted":0.10600268840789795}},"usage_input_tokens":60,"oracle_file":"records_ref.json"},{"row_id":"semif_07ee70cfe4b01330ce89/text/decision","fixture_id":"semif_07ee70cfe4b01330ce89","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the parcel has been weighed.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,3059,14350,768,61026,32634,1203,1295,9100,933,12552,2203,523,18,9886,894,779,4673,535,779,61026,1178,1381,45943,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,21],"sequence_length":60,"marker_positions":[28,38,49],"positions":60,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-10.376697540283203,-5.948400974273682,1.1245343685150146],"probabilities_raw":[1.0108947208209429e-05,0.0008470146567560732,0.9991428852081299],"probabilities":[0.0002685684012249112,0.006351233460009098,0.9933801889419556],"argmax_index":2,"argmax":"insufficient","top2_margin":0.9870289554819465,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.9933801889419556,"probabilities":{"supported":0.0002685684012249112,"contradicted":0.006351233460009098,"insufficient":0.9933801889419556}},"usage_input_tokens":60,"oracle_file":"records_ref.json"},{"row_id":"semif_0aa1447680e4f25862bb/text/decision","fixture_id":"semif_0aa1447680e4f25862bb","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; in conflicts, the signed ledger overrides the desk clerk. May it be released?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,26862,53076,7944,779,7494,28553,906,10274,523,941,10423,4462,1856,7944,936,906,946,38008,523,9758,7944,938,2835,523,18,35516,7494,8544,1403,779,28553,856,8112,10274,536,797,19980,521,779,10423,4462,1856,1391,58340,779,26862,53076,523,3354,936,874,5789,540,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,21],"sequence_length":97,"marker_positions":[57,74,86],"positions":97,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.831890106201172,1.8205256462097168,2.5837559700012207],"probabilities_raw":[0.46642276644706726,0.16964839398860931,0.36392882466316223],"probabilities":[0.4304542541503906,0.20901064574718475,0.3605351746082306],"argmax_index":0,"argmax":"insufficient","top2_margin":0.06991907954216003,"near_tie":false,"gold":"prohibited","official_answer":{"type":"choice","choice":"insufficient","confidence":0.4304542541503906,"probabilities":{"insufficient":0.4304542541503906,"prohibited":0.20901064574718475,"permitted":0.3605351746082306}},"usage_input_tokens":97,"oracle_file":"records_ref.json"},{"row_id":"semif_0aa24a6701ac9c56c4a0/text/decision","fixture_id":"semif_0aa24a6701ac9c56c4a0","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the candidate with currently valid streaming permission; a withdrawal cancels an earlier grant.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,21009,30403,32340,521,1757,22922,46159,779,52847,2076,779,3308,523,46050,931,879,535,941,21009,30403,32340,810,779,5156,3084,44987,936,6728,797,5323,523,18,15534,779,13719,916,6654,6612,32340,15001,536,768,28687,32491,1706,902,7045,8825,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,21],"sequence_length":84,"marker_positions":[57,68,76],"positions":84,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.172837495803833,2.860612392425537,1.9425041675567627],"probabilities_raw":[0.2643018066883087,0.52577143907547,0.20992669463157654],"probabilities":[0.28713083267211914,0.46929967403411865,0.24356946349143982],"argmax_index":1,"argmax":"B","top2_margin":0.1821688413619995,"near_tie":false,"gold":"B","official_answer":{"type":"choice","choice":"B","confidence":0.46929967403411865,"probabilities":{"insufficient":0.28713083267211914,"B":0.46929967403411865,"A":0.24356946349143982}},"usage_input_tokens":84,"oracle_file":"records_ref.json"},{"row_id":"semif_0b43ea8e24e74d621f1b/text/decision","fixture_id":"semif_0b43ea8e24e74d621f1b","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the replacement lenses have been fitted.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,23120,44987,936,25539,779,16400,31286,811,779,13608,1090,28623,523,18,9886,894,779,4673,535,779,16400,31286,1052,1381,25539,523,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":62,"marker_positions":[30,41,51],"positions":62,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.1856169700622559,6.04213809967041,-3.6982507705688477],"probabilities_raw":[0.0007255796226672828,0.9992156028747559,5.881147808395326e-05],"probabilities":[0.005686212796717882,0.9933690428733826,0.0009447598131373525],"argmax_index":1,"argmax":"supported","top2_margin":0.9876828300766647,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"supported","confidence":0.9933690428733826,"probabilities":{"insufficient":0.005686212796717882,"supported":0.9933690428733826,"contradicted":0.0009447598131373525}},"usage_input_tokens":62,"oracle_file":"records_ref.json"},{"row_id":"semif_0ebd299739302d8b057b/text/decision","fixture_id":"semif_0ebd299739302d8b057b","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: under the supplied protocol, the crate is currently in storage.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,26862,3446,3498,779,2391,931,1178,3191,8784,536,779,51746,30174,3498,936,856,797,8784,523,9758,7944,938,2835,521,810,779,30174,12199,19980,1490,779,13061,523,18,9886,894,779,4673,535,1490,779,21706,13061,521,779,2391,931,856,6654,797,8784,523,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":88,"marker_positions":[56,67,78],"positions":88,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.0599265098571777,0.8353641033172607,0.7449114322662354],"probabilities_raw":[0.6400587558746338,0.18810448050498962,0.1718367636203766],"probabilities":[0.5531457662582397,0.23064285516738892,0.2162114381790161],"argmax_index":0,"argmax":"contradicted","top2_margin":0.32250291109085083,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"contradicted","confidence":0.5531457662582397,"probabilities":{"contradicted":0.5531457662582397,"insufficient":0.23064285516738892,"supported":0.2162114381790161}},"usage_input_tokens":88,"oracle_file":"records_ref.json"},{"row_id":"semif_1105577da4c8dad4609d/text/decision","fixture_id":"semif_1105577da4c8dad4609d","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the desk log reports the crate in storage.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,26862,3446,3498,779,2391,931,856,797,8784,523,941,51746,30174,3498,936,1178,3191,8784,523,9758,7944,938,2835,523,941,21706,13061,3498,779,51746,30174,12199,6249,9448,19148,1518,1547,5617,9171,523,18,9886,894,779,4673,535,779,26862,3446,7944,779,2391,931,797,8784,523,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":92,"marker_positions":[60,71,82],"positions":92,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.914824366569519,1.321368932723999,-0.17958185076713562],"probabilities_raw":[0.5968159437179565,0.329690158367157,0.07349391281604767],"probabilities":[0.5323480367660522,0.34840649366378784,0.11924546957015991],"argmax_index":0,"argmax":"contradicted","top2_margin":0.1839415431022644,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"contradicted","confidence":0.5323480367660522,"probabilities":{"contradicted":0.5323480367660522,"insufficient":0.34840649366378784,"supported":0.11924546957015991}},"usage_input_tokens":92,"oracle_file":"records_ref.json"},{"row_id":"semif_19d3d8d0bf773e88be7b/text/decision","fixture_id":"semif_19d3d8d0bf773e88be7b","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate reporting an awarded grant.","criteria":{"A":"Candidate A","B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,8825,953,14708,523,46050,931,879,535,941,5771,953,5068,536,779,9170,6728,1044,2017,3019,523,18,52739,779,13719,15603,902,14708,8825,523,19,16,835,535,46050,931,835,20,19,16,879,535,46050,931,879,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,21],"sequence_length":64,"marker_positions":[37,45,53],"positions":64,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.8596923351287842,-1.9263551235198975,-0.09066939353942871],"probabilities_raw":[0.6904811263084412,0.04257814958691597,0.26694080233573914],"probabilities":[0.608325719833374,0.08314049988985062,0.3085337281227112],"argmax_index":0,"argmax":"A","top2_margin":0.29979199171066284,"near_tie":false,"gold":"A","official_answer":{"type":"choice","choice":"A","confidence":0.608325719833374,"probabilities":{"A":0.608325719833374,"B":0.08314049988985062,"insufficient":0.3085337281227112}},"usage_input_tokens":64,"oracle_file":"records_ref.json"},{"row_id":"semif_19f37d5a97034bd7d04f/text/decision","fixture_id":"semif_19f37d5a97034bd7d04f","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the rooftop terrace?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,33648,1090,37181,6175,13546,779,30122,9041,810,4943,876,13748,774,779,1232,33388,920,2543,10074,523,18,542,37181,6175,22558,9544,8544,811,779,3508,936,13546,523,3354,779,33648,7157,779,1232,33388,920,2543,10074,540,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,21],"sequence_length":88,"marker_positions":[48,65,77],"positions":88,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.548405170440674,3.762322187423706,-0.5323378443717957],"probabilities_raw":[0.2266252338886261,0.7629669308662415,0.010407790541648865],"probabilities":[0.28646430373191833,0.6818165183067322,0.031719230115413666],"argmax_index":1,"argmax":"prohibited","top2_margin":0.39535221457481384,"near_tie":false,"gold":"prohibited","official_answer":{"type":"choice","choice":"prohibited","confidence":0.6818165183067322,"probabilities":{"insufficient":0.28646430373191833,"prohibited":0.6818165183067322,"permitted":0.031719230115413666}},"usage_input_tokens":88,"oracle_file":"records_ref.json"},{"row_id":"semif_1a81ead69e93e5d0c86a/text/decision","fixture_id":"semif_1a81ead69e93e5d0c86a","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the recovery email be changed?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,12153,21968,902,3863,10711,521,1203,779,7387,811,874,7093,938,49033,523,18,18693,902,3863,2853,1550,1403,779,12153,3575,6229,9365,896,2853,523,3354,779,11256,8355,874,7093,540,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":81,"marker_positions":[41,52,64],"positions":81,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-5.296720027923584,-2.989729642868042,2.016939878463745],"probabilities_raw":[0.0006615055026486516,0.006644258741289377,0.9926942586898804],"probabilities":[0.005209682509303093,0.027071693912148476,0.9677186608314514],"argmax_index":2,"argmax":"insufficient","top2_margin":0.9406469669193029,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.9677186608314514,"probabilities":{"permitted":0.005209682509303093,"prohibited":0.027071693912148476,"insufficient":0.9677186608314514}},"usage_input_tokens":81,"oracle_file":"records_ref.json"},{"row_id":"semif_1df851a202994a07f123/text/decision","fixture_id":"semif_1df851a202994a07f123","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the agreed handover is complete.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,11860,2090,1321,6835,2059,779,3956,8649,810,779,5196,8170,523,941,11746,3084,44987,779,3956,8649,11423,810,22922,3498,779,8170,953,1014,14465,523,18,9886,894,779,4673,535,779,11860,2090,1321,856,5156,523,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":79,"marker_positions":[47,58,69],"positions":79,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.49528419971466064,1.0892655849456787,-9.397531509399414],"probabilities_raw":[0.35571545362472534,0.6442665457725525,1.7976592062041163e-05],"probabilities":[0.39535412192344666,0.6043087244033813,0.0003372168866917491],"argmax_index":1,"argmax":"insufficient","top2_margin":0.2089546024799347,"near_tie":false,"gold":"contradicted","official_answer":{"type":"choice","choice":"insufficient","confidence":0.6043087244033813,"probabilities":{"contradicted":0.39535412192344666,"insufficient":0.6043087244033813,"supported":0.0003372168866917491}},"usage_input_tokens":79,"oracle_file":"records_ref.json"},{"row_id":"semif_2175379f4c69e6207626/text/decision","fixture_id":"semif_2175379f4c69e6207626","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate reporting an observed test result.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,6567,938,33037,29067,7441,536,1295,1927,953,2685,523,46050,931,879,535,941,7850,33037,1927,7410,523,18,52739,779,13719,15603,902,6023,1927,1725,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,21],"sequence_length":65,"marker_positions":[38,49,57],"positions":65,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.35558879375457764,-0.2648470997810364,0.7199549674987793],"probabilities_raw":[0.3358735144138336,0.1806025356054306,0.4835239052772522],"probabilities":[0.34022027254104614,0.2184140384197235,0.44136565923690796],"argmax_index":2,"argmax":"A","top2_margin":0.10114538669586182,"near_tie":false,"gold":"B","official_answer":{"type":"choice","choice":"A","confidence":0.44136565923690796,"probabilities":{"insufficient":0.34022027254104614,"B":0.2184140384197235,"A":0.44136565923690796}},"usage_input_tokens":65,"oracle_file":"records_ref.json"},{"row_id":"semif_21ca7a8b970e36b8e20f/text/decision","fixture_id":"semif_21ca7a8b970e36b8e20f","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"The stage manager may approve props but may not delegate that authority. Only authorized approvers may approve. May the assistant approve a prop?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,16396,3575,6229,779,5937,11430,811,51132,41955,523,941,5937,11430,17498,779,4889,811,902,16701,521,1316,1178,1295,1312,17784,10562,523,18,1098,5937,11430,1400,51132,41955,1203,1400,1014,40500,896,10562,523,11005,30403,2396,2053,1400,51132,523,3354,779,16701,51132,768,3346,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,21],"sequence_length":99,"marker_positions":[59,71,88],"positions":99,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.8783601522445679,1.3168089389801025,3.038339614868164],"probabilities_raw":[0.08911655843257904,0.1381574273109436,0.7727259397506714],"probabilities":[0.14192068576812744,0.19411849975585938,0.6639607548713684],"argmax_index":2,"argmax":"permitted","top2_margin":0.46984225511550903,"near_tie":false,"gold":"prohibited","official_answer":{"type":"choice","choice":"permitted","confidence":0.6639607548713684,"probabilities":{"prohibited":0.14192068576812744,"insufficient":0.19411849975585938,"permitted":0.6639607548713684}},"usage_input_tokens":99,"oracle_file":"records_ref.json"},{"row_id":"semif_22ec57822db50ec121cf/text/decision","fixture_id":"semif_22ec57822db50ec121cf","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate supporting a paywall.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,859,64078,26677,768,4025,17686,523,46050,931,879,535,859,22879,26677,768,4025,17686,523,18,52739,779,13719,12469,768,4025,17686,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,835,535,46050,931,835,20,19,16,879,535,46050,931,879,20,21],"sequence_length":61,"marker_positions":[34,45,53],"positions":61,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.028011880815029144,0.252865195274353,1.7705678939819336],"probabilities_raw":[0.12556403875350952,0.157223641872406,0.7172123193740845],"probabilities":[0.17710436880588531,0.20796306431293488,0.6149325966835022],"argmax_index":2,"argmax":"B","top2_margin":0.4069695323705673,"near_tie":false,"gold":"A","official_answer":{"type":"choice","choice":"B","confidence":0.6149325966835022,"probabilities":{"insufficient":0.17710436880588531,"A":0.20796306431293488,"B":0.6149325966835022}},"usage_input_tokens":61,"oracle_file":"records_ref.json"},{"row_id":"semif_25172cd6ddc49ab0ca70/text/decision","fixture_id":"semif_25172cd6ddc49ab0ca70","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Mass production is allowed only after mass-production approval. May it begin?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,22773,22922,2396,2240,3507,3893,803,779,59534,523,18,31678,3893,856,6767,1550,1720,3507,25252,980,17784,523,3354,936,6305,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,21],"sequence_length":70,"marker_positions":[30,42,59],"positions":70,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.136638641357422,3.2041687965393066,0.9671251773834229],"probabilities_raw":[0.2370395064353943,0.6893553733825684,0.0736050233244896],"probabilities":[0.27952614426612854,0.5992463827133179,0.1212274357676506],"argmax_index":1,"argmax":"insufficient","top2_margin":0.31972023844718933,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"insufficient","confidence":0.5992463827133179,"probabilities":{"prohibited":0.27952614426612854,"insufficient":0.5992463827133179,"permitted":0.1212274357676506}},"usage_input_tokens":70,"oracle_file":"records_ref.json"},{"row_id":"semif_2608370d01783735c3ea/text/decision","fixture_id":"semif_2608370d01783735c3ea","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the corrected date is in June.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,40348,10234,26956,779,7627,811,4963,523,8874,3575,12800,768,23819,27809,5105,953,5405,810,4963,953,36072,523,18,9886,894,779,4673,535,779,35101,5562,856,797,4963,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":71,"marker_positions":[39,49,60],"positions":71,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-6.859125137329102,0.526771068572998,-1.9133896827697754],"probabilities_raw":[0.0005699152243323624,0.9193147420883179,0.08011537790298462],"probabilities":[0.004332481883466244,0.8473929166793823,0.14827457070350647],"argmax_index":1,"argmax":"insufficient","top2_margin":0.6991183459758759,"near_tie":false,"gold":"contradicted","official_answer":{"type":"choice","choice":"insufficient","confidence":0.8473929166793823,"probabilities":{"supported":0.004332481883466244,"insufficient":0.8473929166793823,"contradicted":0.14827457070350647}},"usage_input_tokens":71,"oracle_file":"records_ref.json"},{"row_id":"semif_2708193212d8a4a523c7/text/decision","fixture_id":"semif_2708193212d8a4a523c7","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Book the school only as a fallback when the town hall is unavailable or unsuitable; otherwise book the town hall. May the school be booked under this policy?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,4887,11917,810,2254,2059,10547,1186,938,3161,810,11500,875,779,8324,523,18,21509,779,2254,1550,906,768,3981,3321,1403,779,4887,11917,856,58754,933,4994,3972,1108,536,9141,2643,779,4887,11917,523,3354,779,2254,874,2643,791,1490,1033,5792,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":95,"marker_positions":[55,67,78],"positions":95,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.818483293056488,3.5373549461364746,2.113175868988037],"probabilities_raw":[0.0504717193543911,0.765312910079956,0.18421539664268494],"probabilities":[0.09527885168790817,0.6644770503044128,0.24024412035942078],"argmax_index":1,"argmax":"permitted","top2_margin":0.42423292994499207,"near_tie":false,"gold":"prohibited","official_answer":{"type":"choice","choice":"permitted","confidence":0.6644770503044128,"probabilities":{"prohibited":0.09527885168790817,"permitted":0.6644770503044128,"insufficient":0.24024412035942078}},"usage_input_tokens":95,"oracle_file":"records_ref.json"},{"row_id":"semif_2bf540d515a51755b25c/text/decision","fixture_id":"semif_2bf540d515a51755b25c","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate authorizing engraving.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,13831,41071,1124,779,11019,523,46050,931,879,535,13085,1123,1531,936,1499,2144,521,1203,1112,1014,41071,1124,936,4763,523,18,52739,779,13719,3575,4651,41071,3665,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,21],"sequence_length":68,"marker_positions":[41,52,60],"positions":68,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.8004128932952881,0.6302740573883057,0.15096712112426758],"probabilities_raw":[0.4226751923561096,0.35654670000076294,0.22077812254428864],"probabilities":[0.39771389961242676,0.352198988199234,0.25008711218833923],"argmax_index":0,"argmax":"insufficient","top2_margin":0.04551491141319275,"near_tie":false,"gold":"A","official_answer":{"type":"choice","choice":"insufficient","confidence":0.39771389961242676,"probabilities":{"insufficient":0.39771389961242676,"B":0.352198988199234,"A":0.25008711218833923}},"usage_input_tokens":68,"oracle_file":"records_ref.json"},{"row_id":"semif_31eb4533d2c89959ec46/text/decision","fixture_id":"semif_31eb4533d2c89959ec46","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Use the homeowner's latest permission decision; a withdrawal cancels the earlier grant. May the surveyor enter the attic?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,2681,28117,6516,902,2960,15001,6177,521,1203,13834,1352,4140,4689,1523,2919,9448,938,3161,523,18,14875,779,2681,28117,1090,10500,15001,6177,536,768,28687,32491,1706,779,7045,8825,523,3354,779,8153,777,7157,779,1530,793,540,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,21],"sequence_length":90,"marker_positions":[50,67,79],"positions":90,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.566591262817383,0.23663032054901123,0.29303932189941406],"probabilities_raw":[0.8331629037857056,0.08106638491153717,0.08577068150043488],"probabilities":[0.7212923765182495,0.1365465521812439,0.1421610563993454],"argmax_index":0,"argmax":"insufficient","top2_margin":0.5791313201189041,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.7212923765182495,"probabilities":{"insufficient":0.7212923765182495,"prohibited":0.1365465521812439,"permitted":0.1421610563993454}},"usage_input_tokens":90,"oracle_file":"records_ref.json"},{"row_id":"semif_3226202ba1ada2570e9f/text/decision","fixture_id":"semif_3226202ba1ada2570e9f","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Under the alternative rule, a pass requires working in the valley; residence alone is insufficient. May it be issued?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,3434,919,5609,797,779,20610,1203,4117,5329,936,523,18,15246,779,8530,7330,521,768,2230,6835,3822,797,779,20610,536,22115,6955,856,28441,523,3354,936,874,12800,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":79,"marker_positions":[39,51,62],"positions":79,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.931678295135498,1.6135848760604858,2.03139328956604],"probabilities_raw":[0.3530586063861847,0.25686243176460266,0.39007890224456787],"probabilities":[0.3483629524707794,0.2775549590587616,0.3740820288658142],"argmax_index":2,"argmax":"insufficient","top2_margin":0.02571907639503479,"near_tie":false,"gold":"prohibited","official_answer":{"type":"choice","choice":"insufficient","confidence":0.3740820288658142,"probabilities":{"prohibited":0.3483629524707794,"permitted":0.2775549590587616,"insufficient":0.3740820288658142}},"usage_input_tokens":79,"oracle_file":"records_ref.json"},{"row_id":"semif_32f36cd0fbbfb3604586/text/decision","fixture_id":"semif_32f36cd0fbbfb3604586","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the agreed handover is complete.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,11860,2090,1321,6835,779,3956,8649,810,5196,8170,523,941,11746,3084,44987,896,2059,11423,523,18,9886,894,779,4673,535,779,11860,2090,1321,856,5156,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":68,"marker_positions":[36,46,57],"positions":68,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.0280404090881348,0.17128825187683105,-1.9770114421844482],"probabilities_raw":[0.21253859996795654,0.7051793932914734,0.08228201419115067],"probabilities":[0.258857786655426,0.6097228527069092,0.1314193308353424],"argmax_index":1,"argmax":"insufficient","top2_margin":0.35086506605148315,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"insufficient","confidence":0.6097228527069092,"probabilities":{"supported":0.258857786655426,"insufficient":0.6097228527069092,"contradicted":0.1314193308353424}},"usage_input_tokens":68,"oracle_file":"records_ref.json"},{"row_id":"semif_33d6e5da58f0fee2490d/text/decision","fixture_id":"semif_33d6e5da58f0fee2490d","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"The first-aid badge requires only first-aid training. May this guide receive it?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,8170,7850,11820,37414,9007,4611,1203,1014,1509,9764,833,4611,523,18,1098,1509,9764,833,59534,6835,1550,1509,9764,833,4611,523,3354,1033,8170,6984,936,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,21],"sequence_length":76,"marker_positions":[36,48,65],"positions":76,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.6971306800842285,2.78682541847229,-2.3960108757019043],"probabilities_raw":[0.004112706519663334,0.9903295040130615,0.0055577936582267284],"probabilities":[0.019044214859604836,0.9573412537574768,0.023614587262272835],"argmax_index":1,"argmax":"insufficient","top2_margin":0.933726666495204,"near_tie":false,"gold":"prohibited","official_answer":{"type":"choice","choice":"insufficient","confidence":0.9573412537574768,"probabilities":{"prohibited":0.019044214859604836,"insufficient":0.9573412537574768,"permitted":0.023614587262272835}},"usage_input_tokens":76,"oracle_file":"records_ref.json"},{"row_id":"semif_36273dd4777876a26338/text/decision","fixture_id":"semif_36273dd4777876a26338","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the parcel has been weighed.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,3059,5813,779,45853,810,8112,779,61026,1090,8744,4623,523,18,9886,894,779,4673,535,779,61026,1178,1381,45943,523,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,7951,535,941,4589,48308,779,4673,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,21],"sequence_length":60,"marker_positions":[28,39,49],"positions":60,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.4476537704467773,8.073257446289062,-0.6171743869781494],"probabilities_raw":[2.696134652069304e-05,0.9998049139976501,0.00016815461276564747],"probabilities":[0.000543197151273489,0.9974485039710999,0.002008317969739437],"argmax_index":1,"argmax":"supported","top2_margin":0.9954401860013604,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"supported","confidence":0.9974485039710999,"probabilities":{"contradicted":0.000543197151273489,"supported":0.9974485039710999,"insufficient":0.002008317969739437}},"usage_input_tokens":60,"oracle_file":"records_ref.json"},{"row_id":"semif_36c71bb649b7d5eccb1f/text/decision","fixture_id":"semif_36c71bb649b7d5eccb1f","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: terminal Maple is online.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,59063,44987,19459,60468,856,4175,810,19459,42074,788,856,49550,523,18,9886,894,779,4673,535,19459,60468,856,4175,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":60,"marker_positions":[28,38,49],"positions":60,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[7.094854354858398,0.14647667109966278,-2.7314815521240234],"probabilities_raw":[0.9989867806434631,0.0009592188289389014,5.395556945586577e-05],"probabilities":[0.9921786785125732,0.0069338493049144745,0.0008874361519701779],"argmax_index":0,"argmax":"supported","top2_margin":0.9852448292076588,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"supported","confidence":0.9921786785125732,"probabilities":{"supported":0.9921786785125732,"insufficient":0.0069338493049144745,"contradicted":0.0008874361519701779}},"usage_input_tokens":60,"oracle_file":"records_ref.json"},{"row_id":"semif_37cebc3792e98049439f/text/decision","fixture_id":"semif_37cebc3792e98049439f","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the outside workshop can perform this repair.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,45609,7330,9938,592,38183,811,779,797,41722,23120,1403,936,1011,2463,779,1336,536,1517,779,5329,23120,1550,1403,779,797,41722,23120,4584,523,9758,29142,10547,1186,1011,2463,1033,12940,523,18,9886,894,779,4673,535,779,5329,23120,1011,2463,1033,12940,523,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":88,"marker_positions":[56,67,77],"positions":88,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.0944440364837646,2.0053186416625977,-0.7762060165405273],"probabilities_raw":[0.2746860384941101,0.6830058097839355,0.042308107018470764],"probabilities":[0.3144998848438263,0.6028423309326172,0.08265769481658936],"argmax_index":1,"argmax":"supported","top2_margin":0.2883424460887909,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"supported","confidence":0.6028423309326172,"probabilities":{"insufficient":0.3144998848438263,"supported":0.6028423309326172,"contradicted":0.08265769481658936}},"usage_input_tokens":88,"oracle_file":"records_ref.json"},{"row_id":"semif_386e10b1c1ce906473a4/text/decision","fixture_id":"semif_386e10b1c1ce906473a4","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the submission qualifying under the stated exception and exclusion.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,49290,5019,11528,6261,875,20155,1230,1798,521,6794,896,10144,39208,3469,1134,15419,2563,523,46050,931,835,535,4411,38120,953,5019,1929,803,768,20155,1230,1047,810,5260,779,4225,19214,523,46050,931,879,535,29207,953,5019,1929,803,768,20155,1230,1047,1203,39273,779,19214,523,18,15534,779,38120,46102,1490,779,9519,11287,810,32294,523,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,21],"sequence_length":98,"marker_positions":[71,79,87],"positions":98,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.2154874801635742,3.1739749908447266,1.4671909809112549],"probabilities_raw":[0.10666880756616592,0.7561322450637817,0.13719898462295532],"probabilities":[0.16004574298858643,0.6483826637268066,0.19157147407531738],"argmax_index":1,"argmax":"A","top2_margin":0.45681118965148926,"near_tie":false,"gold":"A","official_answer":{"type":"choice","choice":"A","confidence":0.6483826637268066,"probabilities":{"B":0.16004574298858643,"A":0.6483826637268066,"insufficient":0.19157147407531738}},"usage_input_tokens":98,"oracle_file":"records_ref.json"},{"row_id":"semif_3b0958cabc339005cf66/text/decision","fixture_id":"semif_3b0958cabc339005cf66","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the egg discovery was fictional.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,542,6634,35283,902,7267,2630,19736,768,12037,933,3984,779,3863,856,37640,523,18,9886,894,779,4673,535,779,7267,12037,953,37640,523,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":63,"marker_positions":[31,42,52],"positions":63,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.8897602558135986,-6.432289123535156,-0.8964612483978271],"probabilities_raw":[0.8559790253639221,0.000565636670216918,0.14345543086528778],"probabilities":[0.7785053253173828,0.004166017286479473,0.21732863783836365],"argmax_index":0,"argmax":"insufficient","top2_margin":0.5611766874790192,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.7785053253173828,"probabilities":{"insufficient":0.7785053253173828,"supported":0.004166017286479473,"contradicted":0.21732863783836365}},"usage_input_tokens":63,"oracle_file":"records_ref.json"},{"row_id":"semif_3dd9c3ac98f8ddf7bdd3/text/decision","fixture_id":"semif_3dd9c3ac98f8ddf7bdd3","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the document contains weighing instructions.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,3059,12930,12874,875,45853,779,61026,810,22922,3498,1295,45853,1178,1381,5813,523,18,9886,894,779,4673,535,779,4479,6034,45853,12874,523,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,7951,535,941,4589,48308,779,4673,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,21],"sequence_length":64,"marker_positions":[32,43,53],"positions":64,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.29899075627326965,-2.115078926086426,-0.10533981025218964],"probabilities_raw":[0.4208163917064667,0.06845032423734665,0.5107333064079285],"probabilities":[0.4129457473754883,0.11284510046243668,0.47420912981033325],"argmax_index":2,"argmax":"insufficient","top2_margin":0.06126338243484497,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"insufficient","confidence":0.47420912981033325,"probabilities":{"contradicted":0.4129457473754883,"supported":0.11284510046243668,"insufficient":0.47420912981033325}},"usage_input_tokens":64,"oracle_file":"records_ref.json"},{"row_id":"semif_3e75f2b84e8e08e3d3f2/text/decision","fixture_id":"semif_3e75f2b84e8e08e3d3f2","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate supporting a paywall.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,10046,6289,779,13732,523,46050,931,879,535,941,5647,1178,768,7043,20453,523,18,52739,779,13719,12469,768,4025,17686,523,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,21],"sequence_length":60,"marker_positions":[33,41,49],"positions":60,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.2606961727142334,-3.224853515625,0.07640176266431808],"probabilities_raw":[0.5369623899459839,0.016450868919491768,0.4465867280960083],"probabilities":[0.5103158354759216,0.04231620952486992,0.44736793637275696],"argmax_index":0,"argmax":"B","top2_margin":0.06294789910316467,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"B","confidence":0.5103158354759216,"probabilities":{"B":0.5103158354759216,"A":0.04231620952486992,"insufficient":0.44736793637275696}},"usage_input_tokens":60,"oracle_file":"records_ref.json"},{"row_id":"semif_3f783eec5c90812479bc/text/decision","fixture_id":"semif_3f783eec5c90812479bc","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"A resident's pass requires living in the valley and has no workplace restriction. May it be issued?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,3434,919,5609,5329,779,20610,810,4117,5662,936,523,18,542,12518,1090,2230,6835,4624,797,779,20610,810,1178,1295,21701,27434,523,3354,936,874,12800,540,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":76,"marker_positions":[36,47,59],"positions":76,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.9592502117156982,1.7731610536575317,2.0034730434417725],"probabilities_raw":[0.591741681098938,0.1807258129119873,0.2275324910879135],"probabilities":[0.5171136260032654,0.22162669897079468,0.26125967502593994],"argmax_index":0,"argmax":"permitted","top2_margin":0.25585395097732544,"near_tie":false,"gold":"prohibited","official_answer":{"type":"choice","choice":"permitted","confidence":0.5171136260032654,"probabilities":{"permitted":0.5171136260032654,"prohibited":0.22162669897079468,"insufficient":0.26125967502593994}},"usage_input_tokens":76,"oracle_file":"records_ref.json"},{"row_id":"semif_4089c199b8b796ec3ed8/text/decision","fixture_id":"semif_4089c199b8b796ec3ed8","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Book the school only as a fallback when the town hall is unavailable or unsuitable; otherwise book the town hall. May the school be booked under this policy?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,1547,40111,938,5668,521,1203,13834,16425,4689,14503,2583,856,4617,523,18,21509,779,2254,1550,906,768,3981,3321,1403,779,4887,11917,856,58754,933,4994,3972,1108,536,9141,2643,779,4887,11917,523,3354,779,2254,874,2643,791,1490,1033,5792,540,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,21],"sequence_length":94,"marker_positions":[54,71,83],"positions":94,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.3879141807556152,0.4206157922744751,2.479020833969116],"probabilities_raw":[0.44738325476646423,0.06255944818258286,0.4900572597980499],"probabilities":[0.432425856590271,0.10606959462165833,0.4615044593811035],"argmax_index":2,"argmax":"permitted","top2_margin":0.02907860279083252,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"permitted","confidence":0.4615044593811035,"probabilities":{"insufficient":0.432425856590271,"prohibited":0.10606959462165833,"permitted":0.4615044593811035}},"usage_input_tokens":94,"oracle_file":"records_ref.json"},{"row_id":"semif_42384af9ae56beb37ff5/text/decision","fixture_id":"semif_42384af9ae56beb37ff5","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the candidate whose claimed authority comes only through a prohibited second delegation.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,7330,22558,2314,53859,968,779,7576,1203,23369,2796,1523,3759,53859,523,46050,931,835,535,859,5068,3346,17784,10562,5828,988,779,7576,523,46050,931,879,535,859,5068,936,1550,988,835,521,810,1052,1295,1312,8825,523,18,15534,779,13719,5977,13289,10562,4834,1550,1626,768,33776,2341,53859,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,835,535,46050,931,835,20,19,16,879,535,46050,931,879,20,21],"sequence_length":91,"marker_positions":[64,75,83],"positions":91,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.6472855806350708,2.0896692276000977,2.033372163772583],"probabilities_raw":[0.24828527867794037,0.3864343762397766,0.36528030037879944],"probabilities":[0.27105966210365295,0.3717978298664093,0.3571425676345825],"argmax_index":1,"argmax":"A","top2_margin":0.014655262231826782,"near_tie":true,"gold":"B","official_answer":{"type":"choice","choice":"A","confidence":0.3717978298664093,"probabilities":{"insufficient":0.27105966210365295,"A":0.3717978298664093,"B":0.3571425676345825}},"usage_input_tokens":91,"oracle_file":"records_ref.json"},{"row_id":"semif_42c54fa2ad45743e24d8/text/decision","fixture_id":"semif_42c54fa2ad45743e24d8","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the submission qualifying under the stated exception and exclusion.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,49290,5019,11528,6261,875,20155,1230,1798,521,6794,896,10144,39208,3469,1134,15419,2563,523,46050,931,835,810,46050,931,879,2312,5019,11528,6261,1203,3176,13834,1230,1047,20306,4689,19214,2203,523,18,15534,779,38120,46102,1490,779,9519,11287,810,32294,523,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,879,535,46050,931,879,20,21],"sequence_length":81,"marker_positions":[54,62,73],"positions":81,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.7830364108085632,2.160759449005127,1.3004415035247803],"probabilities_raw":[0.1505223959684372,0.596950888633728,0.2525267004966736],"probabilities":[0.19520875811576843,0.5222931504249573,0.2824980318546295],"argmax_index":1,"argmax":"insufficient","top2_margin":0.23979511857032776,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.5222931504249573,"probabilities":{"A":0.19520875811576843,"insufficient":0.5222931504249573,"B":0.2824980318546295}},"usage_input_tokens":81,"oracle_file":"records_ref.json"},{"row_id":"semif_4444dad25e438f128673/text/decision","fixture_id":"semif_4444dad25e438f128673","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"A resident's pass requires living in the valley and has no workplace restriction. May it be issued?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,3434,919,5609,797,779,20610,1203,4117,5329,936,523,18,542,12518,1090,2230,6835,4624,797,779,20610,810,1178,1295,21701,27434,523,3354,936,874,12800,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":76,"marker_positions":[36,48,59],"positions":76,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.421286106109619,4.4906415939331055,2.1723551750183105],"probabilities_raw":[0.10309968143701553,0.8165203332901001,0.08038000017404556],"probabilities":[0.16071435809135437,0.7047528624534607,0.13453277945518494],"argmax_index":1,"argmax":"permitted","top2_margin":0.5440385043621063,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"permitted","confidence":0.7047528624534607,"probabilities":{"prohibited":0.16071435809135437,"permitted":0.7047528624534607,"insufficient":0.13453277945518494}},"usage_input_tokens":76,"oracle_file":"records_ref.json"},{"row_id":"semif_44574c44e243ee1b2f52/text/decision","fixture_id":"semif_44574c44e243ee1b2f52","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the corrected date is in July.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,40348,10234,26956,779,7627,811,4963,523,8874,3575,12800,768,23819,27809,5105,953,5405,810,4963,953,36072,523,18,9886,894,779,4673,535,779,35101,5562,856,797,5105,523,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":71,"marker_positions":[39,50,61],"positions":71,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.28942108154296875,-3.0245723724365234,2.4573111534118652],"probabilities_raw":[0.06003668159246445,0.003895431524142623,0.9360678791999817],"probabilities":[0.12112487852573395,0.017167197540402412,0.8617079257965088],"argmax_index":2,"argmax":"supported","top2_margin":0.7405830472707748,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"supported","confidence":0.8617079257965088,"probabilities":{"insufficient":0.12112487852573395,"contradicted":0.017167197540402412,"supported":0.8617079257965088}},"usage_input_tokens":71,"oracle_file":"records_ref.json"},{"row_id":"semif_44bae0c8a1615a096c43/text/decision","fixture_id":"semif_44bae0c8a1615a096c43","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate containing only an unexecuted test procedure.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,6634,8269,896,779,7850,33037,1927,7410,523,46050,931,879,535,941,6634,15994,33037,29067,7441,1203,3498,1295,1927,953,2685,523,18,52739,779,13719,8960,1550,902,946,1513,58009,1927,9100,523,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,21],"sequence_length":74,"marker_positions":[47,55,63],"positions":74,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.737524390220642,0.3417292535305023,0.8451249003410339],"probabilities_raw":[0.04498295858502388,0.35979601740837097,0.5952211022377014],"probabilities":[0.08515271544456482,0.3760550320148468,0.5387921929359436],"argmax_index":2,"argmax":"insufficient","top2_margin":0.1627371609210968,"near_tie":false,"gold":"B","official_answer":{"type":"choice","choice":"insufficient","confidence":0.5387921929359436,"probabilities":{"B":0.08515271544456482,"A":0.3760550320148468,"insufficient":0.5387921929359436}},"usage_input_tokens":74,"oracle_file":"records_ref.json"},{"row_id":"semif_4533dd91267b4d815690/text/decision","fixture_id":"semif_4533dd91267b4d815690","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Normally reject opened paint. Accept it for a manufacturing defect, except that added solvent always makes it ineligible. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,8414,5799,22654,1178,768,12803,13789,810,1236,6034,5231,41095,523,9758,11445,938,11649,523,18,41914,1587,22879,8414,5799,523,45681,936,875,768,12803,13789,521,6794,896,5231,41095,3469,4201,936,797,808,16979,523,3354,1033,1789,874,11086,540,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,21],"sequence_length":93,"marker_positions":[53,64,81],"positions":93,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[3.4331963062286377,2.286992311477661,1.4278408288955688],"probabilities_raw":[0.6884902119636536,0.218830406665802,0.09267935901880264],"probabilities":[0.5953508019447327,0.2625322937965393,0.14211688935756683],"argmax_index":0,"argmax":"permitted","top2_margin":0.33281850814819336,"near_tie":false,"gold":"prohibited","official_answer":{"type":"choice","choice":"permitted","confidence":0.5953508019447327,"probabilities":{"permitted":0.5953508019447327,"insufficient":0.2625322937965393,"prohibited":0.14211688935756683}},"usage_input_tokens":93,"oracle_file":"records_ref.json"},{"row_id":"semif_46b7029b9a704138b77a/text/decision","fixture_id":"semif_46b7029b9a704138b77a","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the owner gave permission on Monday.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,3312,17980,779,12153,30403,33300,1450,52518,523,2357,22092,1689,22922,46159,896,52847,523,941,3084,3498,17688,1486,23496,7045,15001,521,810,1295,2919,15001,953,2997,523,18,9886,894,779,4673,535,779,12153,6516,15001,884,17980,523,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":81,"marker_positions":[49,60,71],"positions":81,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.612383246421814,0.810704231262207,2.3589706420898438],"probabilities_raw":[0.12571735680103302,0.1532939225435257,0.7209887504577637],"probabilities":[0.17748259007930756,0.2044944167137146,0.6180230379104614],"argmax_index":2,"argmax":"supported","top2_margin":0.4135286211967468,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"supported","confidence":0.6180230379104614,"probabilities":{"insufficient":0.17748259007930756,"contradicted":0.2044944167137146,"supported":0.6180230379104614}},"usage_input_tokens":81,"oracle_file":"records_ref.json"},{"row_id":"semif_46d6a0731bd288c370f0/text/decision","fixture_id":"semif_46d6a0731bd288c370f0","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: Mo has authority to approve exhibit labels.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,30729,856,5668,797,31804,1337,18577,521,1203,1295,10562,25274,933,53859,4450,938,21706,523,18,9886,894,779,4673,535,8376,1178,10562,811,51132,16993,18577,523,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":67,"marker_positions":[35,46,57],"positions":67,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.5726563930511475,-0.306399405002594,-4.652129173278809],"probabilities_raw":[0.9461489915847778,0.053162023425102234,0.0006890883669257164],"probabilities":[0.882128119468689,0.1128116250038147,0.005060174502432346],"argmax_index":0,"argmax":"insufficient","top2_margin":0.7693164944648743,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.882128119468689,"probabilities":{"insufficient":0.882128119468689,"contradicted":0.1128116250038147,"supported":0.005060174502432346}},"usage_input_tokens":67,"oracle_file":"records_ref.json"},{"row_id":"semif_4aa83af95bb69e90bc2d/text/decision","fixture_id":"semif_4aa83af95bb69e90bc2d","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"The stage manager may delegate prop approval to an assistant, and doing so grants the assistant approval authority. May this assistant approve?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,16396,3575,6229,779,5937,11430,811,51132,41955,523,941,5937,11430,17498,779,4889,811,902,16701,521,1316,1178,1295,1312,17784,10562,523,18,1098,5937,11430,1400,40500,3346,17784,811,902,16701,521,810,5296,1101,25274,779,16701,17784,10562,523,3354,1033,16701,51132,540,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":97,"marker_positions":[57,68,80],"positions":97,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.168987989425659,0.28088173270225525,1.7221136093139648],"probabilities_raw":[0.5583527684211731,0.08451125770807266,0.35713595151901245],"probabilities":[0.5034531950950623,0.1306791752576828,0.36586761474609375],"argmax_index":0,"argmax":"permitted","top2_margin":0.1375855803489685,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"permitted","confidence":0.5034531950950623,"probabilities":{"permitted":0.5034531950950623,"prohibited":0.1306791752576828,"insufficient":0.36586761474609375}},"usage_input_tokens":97,"oracle_file":"records_ref.json"},{"row_id":"semif_4aee37a6a155747766aa/text/decision","fixture_id":"semif_4aee37a6a155747766aa","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; in conflicts, the signed ledger overrides the desk clerk. May it be released?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,26862,53076,7944,779,28553,946,38008,521,2158,779,10423,4462,1856,7944,936,10274,523,9758,7944,938,2835,523,18,35516,7494,8544,1403,779,28553,856,8112,10274,536,797,19980,521,779,10423,4462,1856,1391,58340,779,26862,53076,523,3354,936,874,5789,540,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":95,"marker_positions":[55,66,78],"positions":95,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.37359619140625,1.4480979442596436,2.7218360900878906],"probabilities_raw":[0.16869127750396729,0.18173906207084656,0.6495696902275085],"probabilities":[0.21392832696437836,0.22562186419963837,0.5604497790336609],"argmax_index":2,"argmax":"insufficient","top2_margin":0.3348279148340225,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"insufficient","confidence":0.5604497790336609,"probabilities":{"permitted":0.21392832696437836,"prohibited":0.22562186419963837,"insufficient":0.5604497790336609}},"usage_input_tokens":95,"oracle_file":"records_ref.json"},{"row_id":"semif_4b3d794c004566610181/text/decision","fixture_id":"semif_4b3d794c004566610181","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Sell copies only with the architect's sales permission. May copies be sold?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,6765,4943,876,22558,18109,16711,803,779,2984,523,18,560,965,16711,1550,916,779,6765,1090,11599,15001,523,3354,16711,874,4997,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,21],"sequence_length":71,"marker_positions":[31,43,60],"positions":71,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.2797635793685913,1.3045634031295776,4.427176475524902],"probabilities_raw":[0.008577145636081696,0.04182225093245506,0.9496005773544312],"probabilities":[0.030341804027557373,0.09409179538488388,0.8755663633346558],"argmax_index":2,"argmax":"permitted","top2_margin":0.7814745679497719,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"permitted","confidence":0.8755663633346558,"probabilities":{"prohibited":0.030341804027557373,"insufficient":0.09409179538488388,"permitted":0.8755663633346558}},"usage_input_tokens":71,"oracle_file":"records_ref.json"},{"row_id":"semif_4d7a003342db5b0a17e3/text/decision","fixture_id":"semif_4d7a003342db5b0a17e3","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate reporting receipt with the award still undecided.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,42271,44987,43152,803,768,8825,5771,1203,22922,3498,1295,9170,6177,1178,1381,2047,523,46050,931,879,535,941,42271,44987,896,779,8825,953,14708,523,18,52739,779,13719,15603,43152,916,779,9170,2510,1044,2017,3019,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,835,535,46050,931,835,20,19,16,879,535,46050,931,879,20,21],"sequence_length":79,"marker_positions":[52,63,71],"positions":79,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.0214812755584717,-1.9063987731933594,-0.2571205794811249],"probabilities_raw":[0.2808697521686554,0.11592849344015121,0.6032017469406128],"probabilities":[0.30695444345474243,0.16313357651233673,0.529911994934082],"argmax_index":2,"argmax":"B","top2_margin":0.2229575514793396,"near_tie":false,"gold":"A","official_answer":{"type":"choice","choice":"B","confidence":0.529911994934082,"probabilities":{"insufficient":0.30695444345474243,"A":0.16313357651233673,"B":0.529911994934082}},"usage_input_tokens":79,"oracle_file":"records_ref.json"},{"row_id":"semif_4da6aedc1139224eeff2/text/decision","fixture_id":"semif_4da6aedc1139224eeff2","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the submission disqualified specifically by the signature exclusion despite the outage exception.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,49290,5019,11528,6261,875,20155,1230,1798,521,6794,896,10144,39208,3469,1134,15419,2563,523,46050,931,835,535,4411,38120,953,5019,1929,803,768,20155,1230,1047,810,5260,779,4225,19214,523,46050,931,879,535,29207,953,5019,1929,803,768,20155,1230,1047,1203,39273,779,19214,523,18,15534,779,38120,1134,15419,2441,10285,968,779,19214,32294,9618,779,1230,1047,11287,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,21],"sequence_length":104,"marker_positions":[77,88,96],"positions":104,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.825056552886963,0.6948303580284119,2.1762826442718506],"probabilities_raw":[0.3644622266292572,0.117706798017025,0.517831027507782],"probabilities":[0.3661385178565979,0.1633097380399704,0.4705516993999481],"argmax_index":2,"argmax":"A","top2_margin":0.10441318154335022,"near_tie":false,"gold":"B","official_answer":{"type":"choice","choice":"A","confidence":0.4705516993999481,"probabilities":{"insufficient":0.3661385178565979,"B":0.1633097380399704,"A":0.4705516993999481}},"usage_input_tokens":104,"oracle_file":"records_ref.json"},{"row_id":"semif_51ceab3ce622c7c759f1/text/decision","fixture_id":"semif_51ceab3ce622c7c759f1","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the egg discovery was fictional.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,10720,768,25473,813,1914,521,768,3338,7061,1165,779,12037,803,768,33390,7267,523,941,3893,6634,4478,1033,856,37640,810,1295,2151,7267,953,7935,523,18,9886,894,779,4673,535,779,7267,12037,953,37640,523,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":78,"marker_positions":[46,57,68],"positions":78,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.9429095387458801,0.016189217567443848,6.5538153648376465],"probabilities_raw":[0.0005537894903682172,0.0014450271846726537,0.9980011582374573],"probabilities":[0.004657903220504522,0.009241331368684769,0.9861007332801819],"argmax_index":2,"argmax":"supported","top2_margin":0.9768594019114971,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"supported","confidence":0.9861007332801819,"probabilities":{"contradicted":0.004657903220504522,"insufficient":0.009241331368684769,"supported":0.9861007332801819}},"usage_input_tokens":78,"oracle_file":"records_ref.json"},{"row_id":"semif_533d4423d311de82b27d/text/decision","fixture_id":"semif_533d4423d311de82b27d","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate asking only for information while withholding authorization.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,13608,21090,1531,902,41071,3665,1499,2144,1203,22922,45818,3785,936,41071,1975,4763,523,46050,931,879,535,941,13608,13765,592,779,15565,811,41071,1124,779,11019,523,18,52739,779,13719,14065,1550,875,2203,2158,916,4289,800,52847,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,835,535,46050,931,835,20,19,16,879,535,46050,931,879,20,21],"sequence_length":82,"marker_positions":[55,66,74],"positions":82,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.1153349876403809,0.07922947406768799,-2.398470163345337],"probabilities_raw":[0.7222212553024292,0.25626859068870544,0.021510181948542595],"probabilities":[0.6417179703712463,0.3061330020427704,0.052149027585983276],"argmax_index":0,"argmax":"insufficient","top2_margin":0.33558496832847595,"near_tie":false,"gold":"A","official_answer":{"type":"choice","choice":"insufficient","confidence":0.6417179703712463,"probabilities":{"insufficient":0.6417179703712463,"A":0.3061330020427704,"B":0.052149027585983276}},"usage_input_tokens":82,"oracle_file":"records_ref.json"},{"row_id":"semif_56c419b12cd37cbe790c/text/decision","fixture_id":"semif_56c419b12cd37cbe790c","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the guest endorses the listing's towel statement.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,21522,16461,521,997,1098,21008,3498,25583,1706,938,21706,521,1203,10735,1166,4617,523,859,25202,896,8294,2927,18,9886,894,779,4673,535,779,21522,30261,774,779,21008,1090,50624,8294,523,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":74,"marker_positions":[42,53,64],"positions":74,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.183104008436203,-0.15960577130317688,-3.6311588287353516],"probabilities_raw":[0.5774011611938477,0.4098648726940155,0.012733984738588333],"probabilities":[0.5410023927688599,0.42352527379989624,0.03547229990363121],"argmax_index":0,"argmax":"contradicted","top2_margin":0.11747711896896362,"near_tie":false,"gold":"contradicted","official_answer":{"type":"choice","choice":"contradicted","confidence":0.5410023927688599,"probabilities":{"contradicted":0.5410023927688599,"insufficient":0.42352527379989624,"supported":0.03547229990363121}},"usage_input_tokens":74,"oracle_file":"records_ref.json"},{"row_id":"semif_579beed184ffc1104aa9/text/decision","fixture_id":"semif_579beed184ffc1104aa9","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the complete incident package.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,542,5156,13443,6835,2059,10621,523,46050,931,835,535,941,39037,856,12930,1203,779,33407,2312,856,26117,523,46050,931,879,535,9758,779,39037,810,33407,2312,938,12930,523,18,15534,779,5156,13509,13443,523,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,21],"sequence_length":72,"marker_positions":[45,53,61],"positions":72,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[3.4579966068267822,1.1308937072753906,2.0269265174865723],"probabilities_raw":[0.7481496334075928,0.07300297170877457,0.17884740233421326],"probabilities":[0.6453805565834045,0.12242553383111954,0.2321939319372177],"argmax_index":0,"argmax":"B","top2_margin":0.41318662464618683,"near_tie":false,"gold":"B","official_answer":{"type":"choice","choice":"B","confidence":0.6453805565834045,"probabilities":{"B":0.6453805565834045,"A":0.12242553383111954,"insufficient":0.2321939319372177}},"usage_input_tokens":72,"oracle_file":"records_ref.json"},{"row_id":"semif_587ced261c6092cd1325/text/decision","fixture_id":"semif_587ced261c6092cd1325","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate authorizing engraving.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,13608,21090,1531,902,41071,3665,1499,2144,1203,22922,45818,3785,936,41071,1975,4763,523,46050,931,879,535,941,13608,13765,592,779,15565,811,41071,1124,779,11019,523,18,52739,779,13719,3575,4651,41071,3665,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,21],"sequence_length":77,"marker_positions":[50,61,69],"positions":77,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.8004045486450195,-1.860771656036377,-0.186965674161911],"probabilities_raw":[0.8600244522094727,0.022104576230049133,0.11787096410989761],"probabilities":[0.7604895234107971,0.05562574788928032,0.18388469517230988],"argmax_index":0,"argmax":"insufficient","top2_margin":0.5766048282384872,"near_tie":false,"gold":"B","official_answer":{"type":"choice","choice":"insufficient","confidence":0.7604895234107971,"probabilities":{"insufficient":0.7604895234107971,"B":0.05562574788928032,"A":0.18388469517230988}},"usage_input_tokens":77,"oracle_file":"records_ref.json"},{"row_id":"semif_599b3092b6fd18312d23/text/decision","fixture_id":"semif_599b3092b6fd18312d23","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the candidate with currently valid streaming permission; a withdrawal cancels an earlier grant.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,29117,800,953,30403,810,779,5156,3084,3498,15001,6728,797,5323,523,46050,931,879,535,29117,800,953,30403,521,1757,22922,47927,523,18,15534,779,13719,916,6654,6612,32340,15001,536,768,28687,32491,1706,902,7045,8825,523,19,16,879,535,46050,931,879,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,835,535,46050,931,835,20,21],"sequence_length":79,"marker_positions":[52,60,71],"positions":79,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[3.349609851837158,2.186221122741699,2.6548283100128174],"probabilities_raw":[0.5519954562187195,0.17245754599571228,0.27554699778556824],"probabilities":[0.4891495704650879,0.21306899189949036,0.29778143763542175],"argmax_index":0,"argmax":"B","top2_margin":0.19136813282966614,"near_tie":false,"gold":"A","official_answer":{"type":"choice","choice":"B","confidence":0.4891495704650879,"probabilities":{"B":0.4891495704650879,"insufficient":0.21306899189949036,"A":0.29778143763542175}},"usage_input_tokens":79,"oracle_file":"records_ref.json"},{"row_id":"semif_5a37d1dfcad20810a76a/text/decision","fixture_id":"semif_5a37d1dfcad20810a76a","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Under the alternative policy, either suitable available venue may be booked without priority. May the school be booked?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,4887,11917,810,2254,2059,10547,1186,938,3161,810,11500,875,779,8324,523,18,15246,779,8530,5792,521,4021,11500,3161,35908,1400,874,2643,791,2630,17150,523,3354,779,2254,874,2643,791,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":83,"marker_positions":[43,55,66],"positions":83,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.7450401186943054,2.489084005355835,2.3256301879882812],"probabilities_raw":[0.02085919864475727,0.5294926166534424,0.4496481716632843],"probabilities":[0.049891456961631775,0.5027567744255066,0.4473517835140228],"argmax_index":1,"argmax":"permitted","top2_margin":0.055404990911483765,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"permitted","confidence":0.5027567744255066,"probabilities":{"prohibited":0.049891456961631775,"permitted":0.5027567744255066,"insufficient":0.4473517835140228}},"usage_input_tokens":83,"oracle_file":"records_ref.json"},{"row_id":"semif_5d31b4bf2d0fc193f53b/text/decision","fixture_id":"semif_5d31b4bf2d0fc193f53b","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Approve final handover exactly when both the translated manual and glossary have been supplied. May final handover be approved?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,49104,7944,3822,884,779,2090,1321,1203,14350,1295,11746,3084,875,4021,8688,523,18,8475,32144,2524,2090,1321,8544,1403,2059,779,19089,16687,810,37610,1126,1052,1381,21706,523,3354,2524,2090,1321,874,12043,540,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,21],"sequence_length":86,"marker_positions":[46,57,74],"positions":86,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.4837925434112549,2.482841968536377,-2.960299491882324],"probabilities_raw":[0.26828014850616455,0.7285681962966919,0.0031516929157078266],"probabilities":[0.32433220744132996,0.6621069312095642,0.013560839928686619],"argmax_index":1,"argmax":"insufficient","top2_margin":0.33777472376823425,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.6621069312095642,"probabilities":{"permitted":0.32433220744132996,"insufficient":0.6621069312095642,"prohibited":0.013560839928686619}},"usage_input_tokens":86,"oracle_file":"records_ref.json"},{"row_id":"semif_6149a17bc154f9c5b4a0/text/decision","fixture_id":"semif_6149a17bc154f9c5b4a0","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the lamp has a fault.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,1789,5792,44117,60706,43675,9618,8414,25885,521,1203,22922,13748,774,1883,5896,1919,1403,779,29114,856,60706,523,1470,8414,29114,856,60706,1929,1883,11851,936,523,3253,1312,3360,856,2251,523,18,9886,894,779,4673,535,779,29114,1178,768,15868,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":86,"marker_positions":[54,64,75],"positions":86,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.638434648513794,1.2378493547439575,1.305659532546997],"probabilities_raw":[0.0688871443271637,0.4497777223587036,0.4813351035118103],"probabilities":[0.1132504865527153,0.4326384365558624,0.4541110396385193],"argmax_index":2,"argmax":"contradicted","top2_margin":0.02147260308265686,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"contradicted","confidence":0.4541110396385193,"probabilities":{"supported":0.1132504865527153,"insufficient":0.4326384365558624,"contradicted":0.4541110396385193}},"usage_input_tokens":86,"oracle_file":"records_ref.json"},{"row_id":"semif_61ac20b07a5189231e15/text/decision","fixture_id":"semif_61ac20b07a5189231e15","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Internal training reproduction is allowed when the architect permits that use. May the plan appear in the training booklet?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,6765,22558,10004,2782,768,2984,797,902,7477,4611,62248,810,4943,876,23369,2796,18109,16711,523,18,32135,4611,19494,856,6767,1403,779,6765,22558,896,1517,523,3354,779,2984,4650,797,779,4611,62248,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":85,"marker_positions":[45,57,68],"positions":85,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.741729736328125,2.7703092098236084,1.8944990634918213],"probabilities_raw":[0.2015257179737091,0.5636855363845825,0.23478880524635315],"probabilities":[0.23807977139949799,0.49638843536376953,0.2655317783355713],"argmax_index":1,"argmax":"permitted","top2_margin":0.23085665702819824,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"permitted","confidence":0.49638843536376953,"probabilities":{"prohibited":0.23807977139949799,"permitted":0.49638843536376953,"insufficient":0.2655317783355713}},"usage_input_tokens":85,"oracle_file":"records_ref.json"},{"row_id":"semif_64d63f86e328aff1481c/text/decision","fixture_id":"semif_64d63f86e328aff1481c","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the routing rule assigns this repair to the outside workshop.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,45609,7330,9938,592,38183,811,779,797,41722,23120,1403,936,1011,2463,779,1336,536,1517,779,5329,23120,1550,1403,779,797,41722,23120,4584,523,941,797,41722,23120,4584,2463,1033,12940,536,779,5329,23120,1011,523,18,9886,894,779,4673,535,779,45609,7330,9938,592,1033,12940,811,779,5329,23120,523,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":97,"marker_positions":[65,76,87],"positions":97,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.448075771331787,0.11779351532459259,1.8871092796325684],"probabilities_raw":[0.3551628291606903,0.0939059853553772,0.5509311556816101],"probabilities":[0.36297598481178284,0.1403396874666214,0.49668434262275696],"argmax_index":2,"argmax":"supported","top2_margin":0.13370835781097412,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"supported","confidence":0.49668434262275696,"probabilities":{"contradicted":0.36297598481178284,"insufficient":0.1403396874666214,"supported":0.49668434262275696}},"usage_input_tokens":97,"oracle_file":"records_ref.json"},{"row_id":"semif_674ca4e9346216255466/text/decision","fixture_id":"semif_674ca4e9346216255466","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the candidate ready according to the controlling signed checklist.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,10423,57119,12199,19980,521,1203,46050,931,835,810,46050,931,879,3176,1550,2826,6289,2630,57119,933,22537,5742,523,18,15534,779,13719,8177,4262,811,779,19670,10423,57119,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,21],"sequence_length":66,"marker_positions":[39,50,58],"positions":66,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[3.3179497718811035,1.9383889436721802,1.3750910758972168],"probabilities_raw":[0.7168546915054321,0.1804244965314865,0.10272078961133957],"probabilities":[0.6161913871765137,0.23000140488147736,0.15380728244781494],"argmax_index":0,"argmax":"insufficient","top2_margin":0.3861899822950363,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.6161913871765137,"probabilities":{"insufficient":0.6161913871765137,"B":0.23000140488147736,"A":0.15380728244781494}},"usage_input_tokens":66,"oracle_file":"records_ref.json"},{"row_id":"semif_695dd61195ca025c4f4c/text/decision","fixture_id":"semif_695dd61195ca025c4f4c","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate offering a future delivery.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,28356,893,44987,896,779,52414,1079,953,14465,523,46050,931,879,535,941,28356,893,7637,811,5514,768,52414,1079,27968,523,18,52739,779,13719,13591,768,3866,11746,523,19,16,879,535,46050,931,879,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,835,535,46050,931,835,20,21],"sequence_length":69,"marker_positions":[42,50,61],"positions":69,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.545011043548584,-1.2503870725631714,-0.6067150831222534],"probabilities_raw":[0.41083216667175293,0.2029193788766861,0.3862484395503998],"probabilities":[0.3904627561569214,0.2359113097190857,0.3736259341239929],"argmax_index":0,"argmax":"B","top2_margin":0.016836822032928467,"near_tie":true,"gold":"B","official_answer":{"type":"choice","choice":"B","confidence":0.3904627561569214,"probabilities":{"B":0.3904627561569214,"insufficient":0.2359113097190857,"A":0.3736259341239929}},"usage_input_tokens":69,"oracle_file":"records_ref.json"},{"row_id":"semif_6d64a8459f64eac7f4cc/text/decision","fixture_id":"semif_6d64a8459f64eac7f4cc","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the routing rule assigns this repair to the outside workshop.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,45609,7330,9938,592,38183,811,779,797,41722,23120,1403,936,1011,2463,779,1336,536,1517,779,5329,23120,1550,1403,779,797,41722,23120,4584,523,42749,23120,1090,4978,811,2463,1033,12940,856,21706,523,18,9886,894,779,4673,535,779,45609,7330,9938,592,1033,12940,811,779,5329,23120,523,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":94,"marker_positions":[62,73,84],"positions":94,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.3540732264518738,1.012641191482544,-0.8345171213150024],"probabilities_raw":[0.30895915627479553,0.5969163775444031,0.09412453323602676],"probabilities":[0.3301945626735687,0.5285418629646301,0.14126355946063995],"argmax_index":1,"argmax":"contradicted","top2_margin":0.1983473002910614,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"contradicted","confidence":0.5285418629646301,"probabilities":{"insufficient":0.3301945626735687,"contradicted":0.5285418629646301,"supported":0.14126355946063995}},"usage_input_tokens":94,"oracle_file":"records_ref.json"},{"row_id":"semif_6db17cc22bd656d78558/text/decision","fixture_id":"semif_6db17cc22bd656d78558","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; in conflicts, the signed ledger overrides the desk clerk. May it be released?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,3084,35283,779,26862,53076,810,4462,1856,1203,14350,13834,28553,5742,523,18,35516,7494,8544,1403,779,28553,856,8112,10274,536,797,19980,521,779,10423,4462,1856,1391,58340,779,26862,53076,523,3354,936,874,5789,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":87,"marker_positions":[47,59,70],"positions":87,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.8385608196258545,2.026137590408325,2.54236102104187],"probabilities_raw":[0.23653285205364227,0.285334974527359,0.4781322181224823],"probabilities":[0.2633902430534363,0.3011564314365387,0.4354533553123474],"argmax_index":2,"argmax":"insufficient","top2_margin":0.13429692387580872,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.4354533553123474,"probabilities":{"prohibited":0.2633902430534363,"permitted":0.3011564314365387,"insufficient":0.4354533553123474}},"usage_input_tokens":87,"oracle_file":"records_ref.json"},{"row_id":"semif_712f6ffb763eef7c9a12/text/decision","fixture_id":"semif_712f6ffb763eef7c9a12","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate supporting a paywall.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,10046,31494,768,18958,875,768,4025,17686,1550,811,17575,2723,936,523,46050,931,879,535,941,10046,30261,774,26677,768,4025,17686,523,18,52739,779,13719,12469,768,4025,17686,523,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,21],"sequence_length":71,"marker_positions":[44,52,60],"positions":71,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.45648908615112305,0.1176786869764328,0.13668951392173767],"probabilities_raw":[0.21808426082134247,0.38724181056022644,0.39467400312423706],"probabilities":[0.24785038828849792,0.37352126836776733,0.37862834334373474],"argmax_index":2,"argmax":"insufficient","top2_margin":0.005107074975967407,"near_tie":true,"gold":"B","official_answer":{"type":"choice","choice":"insufficient","confidence":0.37862834334373474,"probabilities":{"B":0.24785038828849792,"A":0.37352126836776733,"insufficient":0.37862834334373474}},"usage_input_tokens":71,"oracle_file":"records_ref.json"},{"row_id":"semif_75b470586834df8c568d/text/decision","fixture_id":"semif_75b470586834df8c568d","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the guest endorses the listing's towel statement.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,21522,52075,779,21008,1203,4852,1295,7082,1337,25583,1706,933,779,5776,523,18,9886,894,779,4673,535,779,21522,30261,774,779,21008,1090,50624,8294,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,21],"sequence_length":67,"marker_positions":[35,45,56],"positions":67,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-4.853909492492676,0.4020563066005707,0.866544246673584],"probabilities_raw":[0.0020090413745492697,0.3851463794708252,0.6128445863723755],"probabilities":[0.009686571545898914,0.4137553572654724,0.5765581130981445],"argmax_index":2,"argmax":"insufficient","top2_margin":0.16280275583267212,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.5765581130981445,"probabilities":{"supported":0.009686571545898914,"contradicted":0.4137553572654724,"insufficient":0.5765581130981445}},"usage_input_tokens":67,"oracle_file":"records_ref.json"},{"row_id":"semif_76c7a79b1ff2363a60c5/text/decision","fixture_id":"semif_76c7a79b1ff2363a60c5","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: terminal Cedar is online.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,59063,3498,19459,42074,788,856,4175,523,50432,60468,856,22922,49550,523,18,9886,894,779,4673,535,19459,42074,788,856,4175,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":62,"marker_positions":[30,40,51],"positions":62,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[3.787590980529785,1.6854655742645264,-0.33135542273521423],"probabilities_raw":[0.878381073474884,0.10733503848314285,0.014283912256360054],"probabilities":[0.7839997410774231,0.17464955151081085,0.04135073721408844],"argmax_index":0,"argmax":"supported","top2_margin":0.6093501895666122,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"supported","confidence":0.7839997410774231,"probabilities":{"supported":0.7839997410774231,"insufficient":0.17464955151081085,"contradicted":0.04135073721408844}},"usage_input_tokens":62,"oracle_file":"records_ref.json"},{"row_id":"semif_774f7f9245f83c796d44/text/decision","fixture_id":"semif_774f7f9245f83c796d44","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the candidate holding valid prop approval authority under the stated delegation rule.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,7576,1090,53859,7330,856,21706,521,1203,46050,931,835,810,46050,931,879,3176,1550,1149,6289,810,1295,10562,37078,523,18,15534,779,13719,12467,6612,3346,17784,10562,1490,779,9519,53859,7330,523,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,21],"sequence_length":70,"marker_positions":[43,51,59],"positions":70,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.467068910598755,1.1629825830459595,1.8013712167739868],"probabilities_raw":[0.5601188540458679,0.1520276963710785,0.2878533899784088],"probabilities":[0.4961564838886261,0.1954556256532669,0.3083878755569458],"argmax_index":0,"argmax":"B","top2_margin":0.1877686083316803,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"B","confidence":0.4961564838886261,"probabilities":{"B":0.4961564838886261,"A":0.1954556256532669,"insufficient":0.3083878755569458}},"usage_input_tokens":70,"oracle_file":"records_ref.json"},{"row_id":"semif_796d5c0da6eaadae6a3f/text/decision","fixture_id":"semif_796d5c0da6eaadae6a3f","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the candidate documenting an authorization that was later withdrawn.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,21009,30403,32340,521,1757,22922,46159,779,52847,2076,779,3308,523,46050,931,879,535,941,21009,30403,32340,810,779,5156,3084,44987,936,6728,797,5323,523,18,15534,779,13719,62439,902,52847,896,953,2919,47927,523,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,21],"sequence_length":78,"marker_positions":[51,59,67],"positions":78,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[3.8280439376831055,1.9527827501296997,1.7718702554702759],"probabilities_raw":[0.7804832458496094,0.11965972185134888,0.09985702484846115],"probabilities":[0.6701697111129761,0.17555658519268036,0.15427377820014954],"argmax_index":0,"argmax":"B","top2_margin":0.4946131259202957,"near_tie":false,"gold":"A","official_answer":{"type":"choice","choice":"B","confidence":0.6701697111129761,"probabilities":{"B":0.6701697111129761,"A":0.17555658519268036,"insufficient":0.15427377820014954}},"usage_input_tokens":78,"oracle_file":"records_ref.json"},{"row_id":"semif_7ada97b6f99f4788e5c7/text/decision","fixture_id":"semif_7ada97b6f99f4788e5c7","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate reporting an awarded grant.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,42271,44987,43152,803,768,8825,5771,1203,22922,3498,1295,9170,6177,1178,1381,2047,523,46050,931,879,535,941,42271,44987,896,779,8825,953,14708,523,18,52739,779,13719,15603,902,14708,8825,523,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,21],"sequence_length":74,"marker_positions":[47,55,63],"positions":74,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.2991751730442047,-2.6162571907043457,-0.412447452545166],"probabilities_raw":[0.6472375988960266,0.03506771847605705,0.31769469380378723],"probabilities":[0.579340934753418,0.07218901813030243,0.3484700620174408],"argmax_index":0,"argmax":"B","top2_margin":0.23087087273597717,"near_tie":false,"gold":"B","official_answer":{"type":"choice","choice":"B","confidence":0.579340934753418,"probabilities":{"B":0.579340934753418,"A":0.07218901813030243,"insufficient":0.3484700620174408}},"usage_input_tokens":74,"oracle_file":"records_ref.json"},{"row_id":"semif_7ed75d854d99b3a81916/text/decision","fixture_id":"semif_7ed75d854d99b3a81916","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose A whenever A is able; choose B only when A is unable and B is able.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,856,16495,810,46050,931,879,856,779,3981,3321,521,1203,13834,13719,1090,4978,811,9936,779,3869,856,21706,523,18,52739,835,19148,835,856,3222,536,7056,879,1550,1403,835,856,12359,810,879,856,3222,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,21],"sequence_length":76,"marker_positions":[49,60,68],"positions":76,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.1733789443969727,1.3448199033737183,1.0819072723388672],"probabilities_raw":[0.5642066597938538,0.2463766634464264,0.18941661715507507],"probabilities":[0.49705639481544495,0.2750169038772583,0.22792671620845795],"argmax_index":0,"argmax":"insufficient","top2_margin":0.22203949093818665,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.49705639481544495,"probabilities":{"insufficient":0.49705639481544495,"B":0.2750169038772583,"A":0.22792671620845795}},"usage_input_tokens":76,"oracle_file":"records_ref.json"},{"row_id":"semif_823bf736b92e9d8fa66d/text/decision","fixture_id":"semif_823bf736b92e9d8fa66d","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"The first-aid badge requires only first-aid training. May this guide receive it?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,8170,15791,875,4611,521,1203,18724,2203,856,1014,21706,523,18,1098,1509,9764,833,59534,6835,1550,1509,9764,833,4611,523,3354,1033,8170,6984,936,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,21],"sequence_length":75,"marker_positions":[35,47,64],"positions":75,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.5380306243896484,2.647414207458496,-0.08526135236024857],"probabilities_raw":[0.014084970578551292,0.9257027506828308,0.06021236255764961],"probabilities":[0.042185135185718536,0.8387300372123718,0.11908479779958725],"argmax_index":1,"argmax":"insufficient","top2_margin":0.7196452394127846,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.8387300372123718,"probabilities":{"prohibited":0.042185135185718536,"insufficient":0.8387300372123718,"permitted":0.11908479779958725}},"usage_input_tokens":75,"oracle_file":"records_ref.json"},{"row_id":"semif_82c2cd97c160ad27951b/text/decision","fixture_id":"semif_82c2cd97c160ad27951b","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate asking to repair the existing window.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,8449,16057,768,60621,523,46050,931,879,535,941,12153,6602,768,2375,3955,523,18,52739,779,13719,14065,811,12940,779,7605,8449,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,21],"sequence_length":62,"marker_positions":[35,46,54],"positions":62,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.8707722425460815,-2.1442296504974365,0.39768466353416443],"probabilities_raw":[0.5980427265167236,0.029331453144550323,0.37262576818466187],"probabilities":[0.5466614961624146,0.06344036757946014,0.3898981213569641],"argmax_index":0,"argmax":"insufficient","top2_margin":0.15676337480545044,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.5466614961624146,"probabilities":{"insufficient":0.5466614961624146,"B":0.06344036757946014,"A":0.3898981213569641}},"usage_input_tokens":62,"oracle_file":"records_ref.json"},{"row_id":"semif_834f3b3e10cc61b33d25/text/decision","fixture_id":"semif_834f3b3e10cc61b33d25","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: terminal Maple is online.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,59063,3498,19459,42074,788,856,4175,523,50432,60468,856,22922,49550,523,18,9886,894,779,4673,535,19459,60468,856,4175,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":61,"marker_positions":[29,39,50],"positions":61,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.509235382080078,1.4758033752441406,1.9639065265655518],"probabilities_raw":[0.00702153705060482,0.37766996026039124,0.6153085231781006],"probabilities":[0.023447468876838684,0.4040045142173767,0.572547972202301],"argmax_index":2,"argmax":"contradicted","top2_margin":0.16854345798492432,"near_tie":false,"gold":"contradicted","official_answer":{"type":"choice","choice":"contradicted","confidence":0.572547972202301,"probabilities":{"supported":0.023447468876838684,"insufficient":0.4040045142173767,"contradicted":0.572547972202301}},"usage_input_tokens":61,"oracle_file":"records_ref.json"},{"row_id":"semif_843185706adff0c60f8b/text/decision","fixture_id":"semif_843185706adff0c60f8b","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the agreed handover is complete.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,10449,6835,3956,8649,810,768,5196,8170,521,1203,779,11746,3084,6602,1295,2203,1337,4021,8688,523,18,9886,894,779,4673,535,779,11860,2090,1321,856,5156,523,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,7951,535,941,4589,48308,779,4673,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,21],"sequence_length":69,"marker_positions":[37,48,58],"positions":69,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.3888823986053467,-8.769230842590332,1.5510286092758179],"probabilities_raw":[0.2382715493440628,2.5104656742769293e-05,0.761703372001648],"probabilities":[0.30347737669944763,0.00043750001350417733,0.6960850954055786],"argmax_index":2,"argmax":"insufficient","top2_margin":0.392607718706131,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.6960850954055786,"probabilities":{"contradicted":0.30347737669944763,"supported":0.00043750001350417733,"insufficient":0.6960850954055786}},"usage_input_tokens":69,"oracle_file":"records_ref.json"},{"row_id":"semif_86a15618ba4cb90a307e/text/decision","fixture_id":"semif_86a15618ba4cb90a307e","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the parcel has been weighed.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,3059,12930,12874,875,45853,779,61026,810,22922,3498,1295,45853,1178,1381,5813,523,18,9886,894,779,4673,535,779,61026,1178,1381,45943,523,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":64,"marker_positions":[32,43,54],"positions":64,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.7928204536437988,1.682106852531433,-5.491433143615723],"probabilities_raw":[0.5274591445922852,0.4721788465976715,0.0003619742637965828],"probabilities":[0.5182804465293884,0.4788702130317688,0.002849374897778034],"argmax_index":0,"argmax":"contradicted","top2_margin":0.03941023349761963,"near_tie":false,"gold":"contradicted","official_answer":{"type":"choice","choice":"contradicted","confidence":0.5182804465293884,"probabilities":{"contradicted":0.5182804465293884,"insufficient":0.4788702130317688,"supported":0.002849374897778034}},"usage_input_tokens":64,"oracle_file":"records_ref.json"},{"row_id":"semif_8a4c3de28be95c105ca1/text/decision","fixture_id":"semif_8a4c3de28be95c105ca1","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Mass production is allowed only after mass-production approval. May it begin?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,22773,2396,2240,3525,768,6643,59534,1203,4943,876,916,4289,592,17784,875,3507,3893,523,18,31678,3893,856,6767,1550,1720,3507,25252,980,17784,523,3354,936,6305,540,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,21],"sequence_length":78,"marker_positions":[38,49,66],"positions":78,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.75178062915802,2.2357535362243652,0.6280560493469238],"probabilities_raw":[0.015214934945106506,0.8204160332679749,0.16436903178691864],"probabilities":[0.04213201254606247,0.7272379398345947,0.230630025267601],"argmax_index":1,"argmax":"insufficient","top2_margin":0.4966079145669937,"near_tie":false,"gold":"prohibited","official_answer":{"type":"choice","choice":"insufficient","confidence":0.7272379398345947,"probabilities":{"permitted":0.04213201254606247,"insufficient":0.7272379398345947,"prohibited":0.230630025267601}},"usage_input_tokens":78,"oracle_file":"records_ref.json"},{"row_id":"semif_8c992e17e4fb54b044f5/text/decision","fixture_id":"semif_8c992e17e4fb54b044f5","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate reporting completed delivery.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,28356,893,44987,896,779,52414,1079,953,14465,523,46050,931,879,535,941,28356,893,7637,811,5514,768,52414,1079,27968,523,18,52739,779,13719,15603,7850,11746,523,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,21],"sequence_length":68,"marker_positions":[41,49,57],"positions":68,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.3221626281738281,0.9677733778953552,-2.26184344291687],"probabilities_raw":[0.5782465934753418,0.40569886565208435,0.016054581850767136],"probabilities":[0.5394794940948486,0.41882413625717163,0.041696369647979736],"argmax_index":0,"argmax":"B","top2_margin":0.120655357837677,"near_tie":false,"gold":"A","official_answer":{"type":"choice","choice":"B","confidence":0.5394794940948486,"probabilities":{"B":0.5394794940948486,"A":0.41882413625717163,"insufficient":0.041696369647979736}},"usage_input_tokens":68,"oracle_file":"records_ref.json"},{"row_id":"semif_8e8c3804a3c15ebb31e3/text/decision","fixture_id":"semif_8e8c3804a3c15ebb31e3","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the guest disputes the towel statement.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,21522,16461,521,997,1098,21008,3498,25583,1706,938,21706,523,859,11667,535,7448,25583,1706,1166,4617,2013,1727,5776,2927,18,9886,894,779,4673,535,779,21522,38419,779,50624,8294,523,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":73,"marker_positions":[41,52,62],"positions":73,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.09096686542034149,-3.5231728553771973,1.6575649976730347],"probabilities_raw":[0.14752578735351562,0.0047673797234892845,0.8477067947387695],"probabilities":[0.21866793930530548,0.018836576491594315,0.7624955177307129],"argmax_index":2,"argmax":"contradicted","top2_margin":0.5438275784254074,"near_tie":false,"gold":"contradicted","official_answer":{"type":"choice","choice":"contradicted","confidence":0.7624955177307129,"probabilities":{"insufficient":0.21866793930530548,"supported":0.018836576491594315,"contradicted":0.7624955177307129}},"usage_input_tokens":73,"oracle_file":"records_ref.json"},{"row_id":"semif_90494c76078ffc3c717a/text/decision","fixture_id":"semif_90494c76078ffc3c717a","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Normally reject opened paint. Accept it for a manufacturing defect, except that added solvent always makes it ineligible. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,8414,22654,1178,768,12803,13789,810,779,28332,44987,1295,41095,953,5231,523,18,41914,1587,22879,8414,5799,523,45681,936,875,768,12803,13789,521,6794,896,5231,41095,3469,4201,936,797,808,16979,523,3354,1033,1789,874,11086,540,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,21],"sequence_length":90,"marker_positions":[50,61,78],"positions":90,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[3.778726100921631,1.7921221256256104,0.7049117684364319],"probabilities_raw":[0.8450194001197815,0.11590322852134705,0.039077434688806534],"probabilities":[0.7389857172966003,0.17878293991088867,0.08223139494657516],"argmax_index":0,"argmax":"permitted","top2_margin":0.5602027773857117,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"permitted","confidence":0.7389857172966003,"probabilities":{"permitted":0.7389857172966003,"insufficient":0.17878293991088867,"prohibited":0.08223139494657516}},"usage_input_tokens":90,"oracle_file":"records_ref.json"},{"row_id":"semif_92ec2ffa3dd46ada1335/text/decision","fixture_id":"semif_92ec2ffa3dd46ada1335","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: this lamp qualifies for return under the stated policy.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,1789,5792,44117,60706,43675,9618,8414,25885,521,1203,22922,13748,774,1883,5896,1919,1403,779,29114,856,60706,523,1470,8414,29114,856,60706,1929,1883,11851,936,523,3253,1312,3360,856,2251,523,18,9886,894,779,4673,535,1033,29114,4357,10812,875,1789,1490,779,9519,5792,523,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":91,"marker_positions":[59,70,81],"positions":91,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.858945608139038,1.736254096031189,0.3094581365585327],"probabilities_raw":[0.4768959879875183,0.4218319058418274,0.1012720987200737],"probabilities":[0.4451003968715668,0.40775102376937866,0.14714854955673218],"argmax_index":0,"argmax":"contradicted","top2_margin":0.03734937310218811,"near_tie":false,"gold":"contradicted","official_answer":{"type":"choice","choice":"contradicted","confidence":0.4451003968715668,"probabilities":{"contradicted":0.4451003968715668,"insufficient":0.40775102376937866,"supported":0.14714854955673218}},"usage_input_tokens":91,"oracle_file":"records_ref.json"},{"row_id":"semif_942ec65dac0bfb9874f9/text/decision","fixture_id":"semif_942ec65dac0bfb9874f9","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the replacement lenses have been fitted.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,2525,7867,8269,768,28623,22820,1203,1295,2565,933,27742,5742,523,18,9886,894,779,4673,535,779,16400,31286,1052,1381,25539,523,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":62,"marker_positions":[30,41,51],"positions":62,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[3.149710178375244,-5.195099830627441,-0.9152504205703735],"probabilities_raw":[0.9828963875770569,0.00023356221208814532,0.016870107501745224],"probabilities":[0.945720911026001,0.0024374090135097504,0.051841605454683304],"argmax_index":0,"argmax":"insufficient","top2_margin":0.8938793055713177,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.945720911026001,"probabilities":{"insufficient":0.945720911026001,"supported":0.0024374090135097504,"contradicted":0.051841605454683304}},"usage_input_tokens":62,"oracle_file":"records_ref.json"},{"row_id":"semif_9473603b44af63beab79/text/decision","fixture_id":"semif_9473603b44af63beab79","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"The first-aid badge requires only first-aid training. May this guide receive it?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,8170,7850,1509,9764,833,4611,1203,1014,11820,37414,9007,4611,523,18,1098,1509,9764,833,59534,6835,1550,1509,9764,833,4611,523,3354,1033,8170,6984,936,540,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,21],"sequence_length":76,"marker_positions":[36,47,64],"positions":76,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.09622183442115784,2.969804525375366,-1.9030358791351318],"probabilities_raw":[0.053090523928403854,0.9397191405296326,0.007190356031060219],"probabilities":[0.11075764894485474,0.8626878261566162,0.026554502546787262],"argmax_index":1,"argmax":"insufficient","top2_margin":0.7519301772117615,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"insufficient","confidence":0.8626878261566162,"probabilities":{"permitted":0.11075764894485474,"insufficient":0.8626878261566162,"prohibited":0.026554502546787262}},"usage_input_tokens":76,"oracle_file":"records_ref.json"},{"row_id":"semif_960a4bcd27363e6db4b9/text/decision","fixture_id":"semif_960a4bcd27363e6db4b9","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the candidate with currently valid streaming permission; a withdrawal cancels an earlier grant.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,835,15001,1534,856,8815,1203,1352,4450,810,2919,3807,938,10144,523,46050,931,879,535,941,3308,2154,856,21706,2630,15001,3129,523,18,15534,779,13719,916,6654,6612,32340,15001,536,768,28687,32491,1706,902,7045,8825,523,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,21],"sequence_length":79,"marker_positions":[52,60,68],"positions":79,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.380281925201416,3.187776803970337,1.8507490158081055],"probabilities_raw":[0.2610172927379608,0.585274875164032,0.15370787680149078],"probabilities":[0.2885650098323822,0.5137538909912109,0.19768109917640686],"argmax_index":1,"argmax":"A","top2_margin":0.22518888115882874,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"A","confidence":0.5137538909912109,"probabilities":{"B":0.2885650098323822,"A":0.5137538909912109,"insufficient":0.19768109917640686}},"usage_input_tokens":79,"oracle_file":"records_ref.json"},{"row_id":"semif_97a35499bd8717f3e786/text/decision","fixture_id":"semif_97a35499bd8717f3e786","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Use the homeowner's latest permission decision; a withdrawal cancels the earlier grant. May the surveyor enter the attic?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,2681,28117,1509,17436,1530,793,9544,521,1757,22922,15011,936,2076,779,5089,523,3253,2919,6177,9647,523,18,14875,779,2681,28117,1090,10500,15001,6177,536,768,28687,32491,1706,779,7045,8825,523,3354,779,8153,777,7157,779,1530,793,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,21],"sequence_length":92,"marker_positions":[52,64,81],"positions":92,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.8441698551177979,2.0381903648376465,1.5088130235671997],"probabilities_raw":[0.34138965606689453,0.41448840498924255,0.24412190914154053],"probabilities":[0.34064170718193054,0.39128151535987854,0.26807668805122375],"argmax_index":1,"argmax":"insufficient","top2_margin":0.050639808177948,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"insufficient","confidence":0.39128151535987854,"probabilities":{"prohibited":0.34064170718193054,"insufficient":0.39128151535987854,"permitted":0.26807668805122375}},"usage_input_tokens":92,"oracle_file":"records_ref.json"},{"row_id":"semif_97d2ed2591fc7638806c/text/decision","fixture_id":"semif_97d2ed2591fc7638806c","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: this lamp qualifies for return under the stated policy.","criteria":{"insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,1789,5792,44117,60706,43675,9618,8414,25885,1203,13748,774,1883,5896,1919,1403,60706,523,941,29114,856,60706,521,1203,779,3360,810,6771,803,1883,5896,938,49033,523,18,9886,894,779,4673,535,1033,29114,4357,10812,875,1789,1490,779,9519,5792,523,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":86,"marker_positions":[54,65,75],"positions":86,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.4517431259155273,-2.3006997108459473,0.18750600516796112],"probabilities_raw":[0.765746533870697,0.017964694648981094,0.21628868579864502],"probabilities":[0.6784985661506653,0.04649622365832329,0.2750052809715271],"argmax_index":0,"argmax":"insufficient","top2_margin":0.4034932851791382,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.6784985661506653,"probabilities":{"insufficient":0.6784985661506653,"supported":0.04649622365832329,"contradicted":0.2750052809715271}},"usage_input_tokens":86,"oracle_file":"records_ref.json"},{"row_id":"semif_980e76145c925747220c/text/decision","fixture_id":"semif_980e76145c925747220c","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Release equipment exactly when the deposit is recorded paid; under this alternative protocol, the desk clerk overrides the ledger. May it be released?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,26862,53076,7944,779,7494,28553,906,10274,523,941,10423,4462,1856,7944,936,906,946,38008,523,9758,7944,938,2835,523,18,35516,7494,8544,1403,779,28553,856,8112,10274,536,1490,1033,8530,13061,521,779,26862,53076,1391,58340,779,4462,1856,523,3354,936,874,5789,540,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,21],"sequence_length":98,"marker_positions":[58,75,87],"positions":98,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.910322427749634,2.614384651184082,2.6769022941589355],"probabilities_raw":[0.3943755030632019,0.2933498024940491,0.3122747242450714],"probabilities":[0.37652453780174255,0.30477800965309143,0.3186974823474884],"argmax_index":0,"argmax":"insufficient","top2_margin":0.05782705545425415,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"insufficient","confidence":0.37652453780174255,"probabilities":{"insufficient":0.37652453780174255,"prohibited":0.30477800965309143,"permitted":0.3186974823474884}},"usage_input_tokens":98,"oracle_file":"records_ref.json"},{"row_id":"semif_9ac5cc82c4126c91174a/text/decision","fixture_id":"semif_9ac5cc82c4126c91174a","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Normally reject opened paint. Accept it for a manufacturing defect, except that added solvent always makes it ineligible. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,8414,22654,856,57077,521,1203,3984,41095,953,5231,856,1014,2538,523,18,41914,1587,22879,8414,5799,523,45681,936,875,768,12803,13789,521,6794,896,5231,41095,3469,4201,936,797,808,16979,523,3354,1033,1789,874,11086,540,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,21],"sequence_length":89,"marker_positions":[49,60,77],"positions":89,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.1729261875152588,2.151684284210205,0.8149329423904419],"probabilities_raw":[0.22934584319591522,0.6103235483169556,0.16033059358596802],"probabilities":[0.2641015648841858,0.5313907265663147,0.2045077234506607],"argmax_index":1,"argmax":"insufficient","top2_margin":0.2672891616821289,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.5313907265663147,"probabilities":{"permitted":0.2641015648841858,"insufficient":0.5313907265663147,"prohibited":0.2045077234506607}},"usage_input_tokens":89,"oracle_file":"records_ref.json"},{"row_id":"semif_a0128e551d095f3449f8/text/decision","fixture_id":"semif_a0128e551d095f3449f8","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: Mo has authority to approve exhibit labels.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,7576,6602,8376,2314,10562,811,51132,16993,18577,536,779,3084,44987,1033,8825,856,6612,523,18,9886,894,779,4673,535,8376,1178,10562,811,51132,16993,18577,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":68,"marker_positions":[36,46,57],"positions":68,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[6.080697059631348,0.6055939793586731,-2.5855391025543213],"probabilities_raw":[0.9956568479537964,0.004171600099653006,0.0001715581165626645],"probabilities":[0.9784088134765625,0.01958679035305977,0.0020043274853378534],"argmax_index":0,"argmax":"supported","top2_margin":0.9588220231235027,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"supported","confidence":0.9784088134765625,"probabilities":{"supported":0.9784088134765625,"insufficient":0.01958679035305977,"contradicted":0.0020043274853378534}},"usage_input_tokens":68,"oracle_file":"records_ref.json"},{"row_id":"semif_a02fd582d2881cc462b2/text/decision","fixture_id":"semif_a02fd582d2881cc462b2","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the sculpture garden?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,33648,1090,37181,6175,13546,779,30122,9041,810,4943,876,13748,774,779,1232,33388,920,2543,10074,523,18,542,37181,6175,22558,9544,8544,811,779,3508,936,13546,523,3354,779,33648,7157,779,30122,9041,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,21],"sequence_length":85,"marker_positions":[45,57,74],"positions":85,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.111523151397705,1.9371681213378906,3.128459930419922],"probabilities_raw":[0.2171688675880432,0.18242156505584717,0.6004095673561096],"probabilities":[0.25312647223472595,0.22348424792289734,0.5233892798423767],"argmax_index":2,"argmax":"permitted","top2_margin":0.27026280760765076,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"permitted","confidence":0.5233892798423767,"probabilities":{"prohibited":0.25312647223472595,"insufficient":0.22348424792289734,"permitted":0.5233892798423767}},"usage_input_tokens":85,"oracle_file":"records_ref.json"},{"row_id":"semif_a0a39ace7e5337d038a7/text/decision","fixture_id":"semif_a0a39ace7e5337d038a7","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the recovery email be changed?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,3863,12153,22558,9365,779,6332,2154,1203,4943,876,23369,2796,9365,779,11256,8355,523,18,18693,902,3863,2853,1550,1403,779,12153,3575,6229,9365,896,2853,523,3354,779,11256,8355,874,7093,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":83,"marker_positions":[43,55,66],"positions":83,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[4.502979278564453,2.016558885574341,1.5348275899887085],"probabilities_raw":[0.881363570690155,0.07333587110042572,0.045300524681806564],"probabilities":[0.7756208181381226,0.13130463659763336,0.09307458251714706],"argmax_index":0,"argmax":"prohibited","top2_margin":0.6443161815404892,"near_tie":false,"gold":"prohibited","official_answer":{"type":"choice","choice":"prohibited","confidence":0.7756208181381226,"probabilities":{"prohibited":0.7756208181381226,"permitted":0.13130463659763336,"insufficient":0.09307458251714706}},"usage_input_tokens":83,"oracle_file":"records_ref.json"},{"row_id":"semif_a22dab8ba073ff6cdb8d/text/decision","fixture_id":"semif_a22dab8ba073ff6cdb8d","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the submission qualifying under the stated exception and exclusion.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,49290,5019,11528,6261,875,20155,1230,1798,521,6794,896,10144,39208,3469,1134,15419,2563,523,46050,931,835,535,4411,5019,1230,1047,8896,38120,39273,779,19214,523,46050,931,879,535,4411,5019,1230,1047,8896,38120,1178,779,19214,523,9758,1230,1798,938,20155,523,18,15534,779,38120,46102,1490,779,9519,11287,810,32294,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,21],"sequence_length":94,"marker_positions":[67,78,86],"positions":94,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.378375768661499,0.7194362282752991,1.3831477165222168],"probabilities_raw":[0.6410338878631592,0.1220148503780365,0.23695123195648193],"probabilities":[0.5565075874328613,0.1701427549123764,0.27334970235824585],"argmax_index":0,"argmax":"insufficient","top2_margin":0.2831578850746155,"near_tie":false,"gold":"B","official_answer":{"type":"choice","choice":"insufficient","confidence":0.5565075874328613,"probabilities":{"insufficient":0.5565075874328613,"B":0.1701427549123764,"A":0.27334970235824585}},"usage_input_tokens":94,"oracle_file":"records_ref.json"},{"row_id":"semif_a3f18f3a63d45345942b/text/decision","fixture_id":"semif_a3f18f3a63d45345942b","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the replacement lenses have been fitted.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,2525,7867,13055,16400,31286,523,941,23120,44987,1186,1052,1014,4763,1381,25539,811,779,13608,1090,28623,523,18,9886,894,779,4673,535,779,16400,31286,1052,1381,25539,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":70,"marker_positions":[38,48,59],"positions":70,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-10.35418701171875,2.370882034301758,1.7727797031402588],"probabilities_raw":[1.919910801007063e-06,0.6452207565307617,0.3547772765159607],"probabilities":[6.825674063293263e-05,0.605174720287323,0.394756942987442],"argmax_index":1,"argmax":"insufficient","top2_margin":0.21041777729988098,"near_tie":false,"gold":"contradicted","official_answer":{"type":"choice","choice":"insufficient","confidence":0.605174720287323,"probabilities":{"supported":6.825674063293263e-05,"insufficient":0.605174720287323,"contradicted":0.394756942987442}},"usage_input_tokens":70,"oracle_file":"records_ref.json"},{"row_id":"semif_a9230edfc6ec9e524af0/text/decision","fixture_id":"semif_a9230edfc6ec9e524af0","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the complete incident package.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,542,5156,13509,13443,6835,2059,768,39037,810,768,33407,2312,523,46050,931,835,535,9758,10621,938,12930,523,46050,931,879,535,11005,779,39037,856,12930,536,779,33407,2312,856,26117,523,18,15534,779,5156,13509,13443,523,19,16,879,535,46050,931,879,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,835,535,46050,931,835,20,21],"sequence_length":75,"marker_positions":[48,56,67],"positions":75,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.8022832274436951,0.6754550933837891,0.012994378805160522],"probabilities_raw":[0.1308416724205017,0.5734819173812866,0.29567641019821167],"probabilities":[0.17655335366725922,0.5073632001876831,0.31608352065086365],"argmax_index":1,"argmax":"insufficient","top2_margin":0.19127967953681946,"near_tie":false,"gold":"A","official_answer":{"type":"choice","choice":"insufficient","confidence":0.5073632001876831,"probabilities":{"B":0.17655335366725922,"insufficient":0.5073632001876831,"A":0.31608352065086365}},"usage_input_tokens":75,"oracle_file":"records_ref.json"},{"row_id":"semif_acacd8b46acd25cb954b/text/decision","fixture_id":"semif_acacd8b46acd25cb954b","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Book the school only as a fallback when the town hall is unavailable or unsuitable; otherwise book the town hall. May the school be booked under this policy?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,4887,11917,856,11649,58754,521,2158,779,2254,856,3161,810,11500,523,18,21509,779,2254,1550,906,768,3981,3321,1403,779,4887,11917,856,58754,933,4994,3972,1108,536,9141,2643,779,4887,11917,523,3354,779,2254,874,2643,791,1490,1033,5792,540,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,21],"sequence_length":94,"marker_positions":[54,71,82],"positions":94,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.6015324592590332,4.51754903793335,1.0473811626434326],"probabilities_raw":[0.04989488422870636,0.9214375615119934,0.028667587786912918],"probabilities":[0.10307374596595764,0.8275465965270996,0.06937965005636215],"argmax_index":1,"argmax":"permitted","top2_margin":0.724472850561142,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"permitted","confidence":0.8275465965270996,"probabilities":{"insufficient":0.10307374596595764,"permitted":0.8275465965270996,"prohibited":0.06937965005636215}},"usage_input_tokens":94,"oracle_file":"records_ref.json"},{"row_id":"semif_b45c63fb92729af27f05/text/decision","fixture_id":"semif_b45c63fb92729af27f05","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: permission to scan the notebook is currently valid.","criteria":{"contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,3312,17980,779,12153,30403,33300,1450,52518,523,2357,22092,1689,22922,46159,896,52847,523,941,3084,3498,17688,1486,23496,7045,15001,521,810,1295,2919,15001,953,2997,523,18,9886,894,779,4673,535,15001,811,19382,779,52518,856,6654,6612,523,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,7951,535,941,4589,48308,779,4673,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,21],"sequence_length":83,"marker_positions":[51,62,72],"positions":83,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[3.0344765186309814,-6.221322059631348,1.7618658542633057],"probabilities_raw":[0.7811310291290283,7.464173540938646e-05,0.2187943458557129],"probabilities":[0.7121281027793884,0.0009574248106218874,0.2869144380092621],"argmax_index":0,"argmax":"contradicted","top2_margin":0.42521366477012634,"near_tie":false,"gold":"contradicted","official_answer":{"type":"choice","choice":"contradicted","confidence":0.7121281027793884,"probabilities":{"contradicted":0.7121281027793884,"supported":0.0009574248106218874,"insufficient":0.2869144380092621}},"usage_input_tokens":83,"oracle_file":"records_ref.json"},{"row_id":"semif_b49ecda75086d8c68d2a/text/decision","fixture_id":"semif_b49ecda75086d8c68d2a","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate reporting completed delivery.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,28356,893,6602,902,4332,523,46050,931,879,535,941,52414,1079,6034,809,814,969,523,18,52739,779,13719,15603,7850,11746,523,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,21],"sequence_length":61,"marker_positions":[34,42,50],"positions":61,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.6683297157287598,-2.664180040359497,-0.32061368227005005],"probabilities_raw":[0.8695845007896423,0.01142159104347229,0.11899389326572418],"probabilities":[0.7771173715591431,0.035188402980566025,0.1876942366361618],"argmax_index":0,"argmax":"B","top2_margin":0.5894231349229813,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"B","confidence":0.7771173715591431,"probabilities":{"B":0.7771173715591431,"A":0.035188402980566025,"insufficient":0.1876942366361618}},"usage_input_tokens":61,"oracle_file":"records_ref.json"},{"row_id":"semif_b531dd603bf080dab820/text/decision","fixture_id":"semif_b531dd603bf080dab820","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Approve final handover exactly when both the translated manual and glossary have been supplied. May final handover be approved?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,49104,21706,779,19089,16687,1203,22922,2172,1014,7018,779,37610,1126,523,18,8475,32144,2524,2090,1321,8544,1403,2059,779,19089,16687,810,37610,1126,1052,1381,21706,523,3354,2524,2090,1321,874,12043,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":84,"marker_positions":[44,56,67],"positions":84,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.07308841496706009,2.8655457496643066,3.741370916366577],"probabilities_raw":[0.015327489003539085,0.2895366847515106,0.695135772228241],"probabilities":[0.04096178337931633,0.33422577381134033,0.6248124837875366],"argmax_index":2,"argmax":"insufficient","top2_margin":0.2905867099761963,"near_tie":false,"gold":"prohibited","official_answer":{"type":"choice","choice":"insufficient","confidence":0.6248124837875366,"probabilities":{"prohibited":0.04096178337931633,"permitted":0.33422577381134033,"insufficient":0.6248124837875366}},"usage_input_tokens":84,"oracle_file":"records_ref.json"},{"row_id":"semif_b5e6788d95b8395d2888/text/decision","fixture_id":"semif_b5e6788d95b8395d2888","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the source files were delivered.","criteria":{"insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,11860,2090,1321,6835,2059,779,3956,8649,810,779,5196,8170,523,941,11746,3084,44987,779,3956,8649,11423,810,22922,3498,779,8170,953,1014,14465,523,18,9886,894,779,4673,535,779,3956,8649,1166,14465,523,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":78,"marker_positions":[46,57,68],"positions":78,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.7756969332695007,0.1709093302488327,-0.5731619596481323],"probabilities_raw":[0.5537936687469482,0.30247679352760315,0.1437295377254486],"probabilities":[0.49243271350860596,0.3196847140789032,0.18788257241249084],"argmax_index":0,"argmax":"insufficient","top2_margin":0.17274799942970276,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"insufficient","confidence":0.49243271350860596,"probabilities":{"insufficient":0.49243271350860596,"contradicted":0.3196847140789032,"supported":0.18788257241249084}},"usage_input_tokens":78,"oracle_file":"records_ref.json"},{"row_id":"semif_b7c4e62d07986286fce0/text/decision","fixture_id":"semif_b7c4e62d07986286fce0","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Sell copies only with the architect's sales permission. May copies be sold?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,2797,6765,1090,15001,7627,9647,521,1203,779,5582,936,6007,938,49033,523,18,560,965,16711,1550,916,779,6765,1090,11599,15001,523,3354,16711,874,4997,540,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,21],"sequence_length":75,"marker_positions":[35,52,63],"positions":75,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.9559473991394043,-0.06785887479782104,0.48010173439979553],"probabilities_raw":[0.8828402757644653,0.04292000085115433,0.07423967868089676],"probabilities":[0.7776677012443542,0.08968294411897659,0.13264939188957214],"argmax_index":0,"argmax":"insufficient","top2_margin":0.6450183093547821,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.7776677012443542,"probabilities":{"insufficient":0.7776677012443542,"permitted":0.08968294411897659,"prohibited":0.13264939188957214}},"usage_input_tokens":75,"oracle_file":"records_ref.json"},{"row_id":"semif_b8091be929b1e489fcc4/text/decision","fixture_id":"semif_b8091be929b1e489fcc4","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: a real egg was discovered.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,10720,768,25473,813,1914,521,768,3338,7061,1165,779,12037,803,768,33390,7267,523,941,3893,6634,4478,1033,856,37640,810,1295,2151,7267,953,7935,523,18,9886,894,779,4673,535,768,2151,7267,953,7935,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,21],"sequence_length":78,"marker_positions":[46,56,67],"positions":78,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-9.860912322998047,4.097040176391602,1.2136549949645996],"probabilities_raw":[8.212900866055861e-07,0.9470182657241821,0.05298099294304848],"probabilities":[4.146222636336461e-05,0.8868883848190308,0.11307018250226974],"argmax_index":1,"argmax":"contradicted","top2_margin":0.773818202316761,"near_tie":false,"gold":"contradicted","official_answer":{"type":"choice","choice":"contradicted","confidence":0.8868883848190308,"probabilities":{"supported":4.146222636336461e-05,"contradicted":0.8868883848190308,"insufficient":0.11307018250226974}},"usage_input_tokens":78,"oracle_file":"records_ref.json"},{"row_id":"semif_b863998d4be2721a5dfd/text/decision","fixture_id":"semif_b863998d4be2721a5dfd","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the routing rule assigns this repair to the outside workshop.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,45609,7330,9938,592,38183,811,779,797,41722,23120,1403,936,1011,2463,779,1336,536,1517,779,5329,23120,1550,1403,779,797,41722,23120,4584,523,9758,29142,10547,1186,1011,2463,1033,12940,523,18,9886,894,779,4673,535,779,45609,7330,9938,592,1033,12940,811,779,5329,23120,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":92,"marker_positions":[60,70,81],"positions":92,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.9867814779281616,0.012546658515930176,-0.5340939164161682],"probabilities_raw":[0.18907205760478973,0.5136059522628784,0.29732200503349304],"probabilities":[0.22605858743190765,0.4615784287452698,0.31236302852630615],"argmax_index":1,"argmax":"insufficient","top2_margin":0.14921540021896362,"near_tie":false,"gold":"contradicted","official_answer":{"type":"choice","choice":"insufficient","confidence":0.4615784287452698,"probabilities":{"supported":0.22605858743190765,"insufficient":0.4615784287452698,"contradicted":0.31236302852630615}},"usage_input_tokens":92,"oracle_file":"records_ref.json"},{"row_id":"semif_b8c5b9b285cbdddc9f8d/text/decision","fixture_id":"semif_b8c5b9b285cbdddc9f8d","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose A whenever A is able; choose B only when A is unable and B is able.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,856,779,16495,57490,1203,856,11649,12359,811,9936,1033,3869,523,46050,931,879,856,779,3981,3321,810,856,11649,3222,523,18,52739,835,19148,835,856,3222,536,7056,879,1550,1403,835,856,12359,810,879,856,3222,523,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,879,535,46050,931,879,20,21],"sequence_length":78,"marker_positions":[51,59,70],"positions":78,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.475014865398407,1.241917371749878,2.9685797691345215],"probabilities_raw":[0.0655418112874031,0.1411171853542328,0.7933409810066223],"probabilities":[0.11538352072238922,0.19955480098724365,0.6850616335868835],"argmax_index":2,"argmax":"B","top2_margin":0.4855068325996399,"near_tie":false,"gold":"B","official_answer":{"type":"choice","choice":"B","confidence":0.6850616335868835,"probabilities":{"A":0.11538352072238922,"insufficient":0.19955480098724365,"B":0.6850616335868835}},"usage_input_tokens":78,"oracle_file":"records_ref.json"},{"row_id":"semif_b8e2750307df9761e716/text/decision","fixture_id":"semif_b8e2750307df9761e716","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Approve final handover exactly when both the translated manual and glossary have been supplied. May final handover be approved?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,49104,21706,2059,779,19089,16687,810,779,37610,1126,523,18,8475,32144,2524,2090,1321,8544,1403,2059,779,19089,16687,810,37610,1126,1052,1381,21706,523,3354,2524,2090,1321,874,12043,540,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":81,"marker_positions":[41,52,64],"positions":81,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[6.782066822052002,-1.3062357902526855,3.482503652572632],"probabilities_raw":[0.9641282558441162,0.0002960940182674676,0.03557564318180084],"probabilities":[0.9109132885932922,0.0028198061045259237,0.08626683801412582],"argmax_index":0,"argmax":"permitted","top2_margin":0.8246464505791664,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"permitted","confidence":0.9109132885932922,"probabilities":{"permitted":0.9109132885932922,"prohibited":0.0028198061045259237,"insufficient":0.08626683801412582}},"usage_input_tokens":81,"oracle_file":"records_ref.json"},{"row_id":"semif_be2dd67bc733c1213bef/text/decision","fixture_id":"semif_be2dd67bc733c1213bef","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Approve the translation milestone when the manual has been supplied; the glossary is not required for this milestone. May this milestone be approved?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,49104,21706,779,19089,16687,1203,22922,2172,1014,7018,779,37610,1126,523,18,8475,32144,779,14144,58931,1403,779,16687,1178,1381,21706,536,779,37610,1126,856,1014,4225,875,1033,58931,523,3354,1033,58931,874,12043,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":87,"marker_positions":[47,59,70],"positions":87,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.8967389464378357,1.5261001586914062,3.427116632461548],"probabilities_raw":[0.06478925049304962,0.12157130241394043,0.8136394619941711],"probabilities":[0.11543203890323639,0.18095681071281433,0.7036111950874329],"argmax_index":2,"argmax":"insufficient","top2_margin":0.5226543843746185,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"insufficient","confidence":0.7036111950874329,"probabilities":{"prohibited":0.11543203890323639,"permitted":0.18095681071281433,"insufficient":0.7036111950874329}},"usage_input_tokens":87,"oracle_file":"records_ref.json"},{"row_id":"semif_be866e1dbec5d42d9423/text/decision","fixture_id":"semif_be866e1dbec5d42d9423","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: this lamp qualifies for return under the stated policy.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,1789,5792,44117,60706,43675,9618,8414,25885,521,1203,13748,774,1883,5896,1919,1403,60706,523,1470,8414,29114,1178,768,12803,45836,15868,521,916,1295,1883,5896,933,1312,32294,523,18,9886,894,779,4673,535,1033,29114,4357,10812,875,1789,1490,779,9519,5792,523,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":88,"marker_positions":[56,67,78],"positions":88,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.425513744354248,1.8368489742279053,0.7128006219863892],"probabilities_raw":[0.3334285318851471,0.5030873417854309,0.16348405182361603],"probabilities":[0.33983734250068665,0.455911785364151,0.20425090193748474],"argmax_index":1,"argmax":"insufficient","top2_margin":0.11607444286346436,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"insufficient","confidence":0.455911785364151,"probabilities":{"contradicted":0.33983734250068665,"insufficient":0.455911785364151,"supported":0.20425090193748474}},"usage_input_tokens":88,"oracle_file":"records_ref.json"},{"row_id":"semif_c100e84585149a04be49/text/decision","fixture_id":"semif_c100e84585149a04be49","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the package containing the timeline but missing the remedy report.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,542,5156,13509,13443,6835,2059,768,39037,810,768,33407,2312,523,46050,931,835,535,9758,10621,938,12930,523,46050,931,879,535,11005,779,39037,856,12930,536,779,33407,2312,856,26117,523,18,15534,779,13443,8960,779,39037,1203,10144,779,33407,2312,523,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,21],"sequence_length":81,"marker_positions":[54,62,70],"positions":81,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.07913510501384735,-0.2963172197341919,0.7918941378593445],"probabilities_raw":[0.23842720687389374,0.19188226759433746,0.5696905255317688],"probabilities":[0.2688654065132141,0.23022787272930145,0.5009067058563232],"argmax_index":2,"argmax":"insufficient","top2_margin":0.23204129934310913,"near_tie":false,"gold":"B","official_answer":{"type":"choice","choice":"insufficient","confidence":0.5009067058563232,"probabilities":{"B":0.2688654065132141,"A":0.23022787272930145,"insufficient":0.5009067058563232}},"usage_input_tokens":81,"oracle_file":"records_ref.json"},{"row_id":"semif_c4e9bd463b6fde5be743/text/decision","fixture_id":"semif_c4e9bd463b6fde5be743","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the complete incident package.","criteria":{"B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,542,5156,13443,6835,2059,10621,523,46050,931,835,810,46050,931,879,3619,1550,13443,20317,810,1295,28261,2203,523,18,15534,779,5156,13509,13443,523,19,16,879,535,46050,931,879,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,835,535,46050,931,835,20,21],"sequence_length":60,"marker_positions":[33,41,52],"positions":60,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.186413526535034,1.998923420906067,2.937082290649414],"probabilities_raw":[0.253327876329422,0.21001826226711273,0.5366538763046265],"probabilities":[0.27900218963623047,0.24402931332588196,0.47696855664253235],"argmax_index":2,"argmax":"A","top2_margin":0.19796636700630188,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"A","confidence":0.47696855664253235,"probabilities":{"B":0.27900218963623047,"insufficient":0.24402931332588196,"A":0.47696855664253235}},"usage_input_tokens":60,"oracle_file":"records_ref.json"},{"row_id":"semif_c6a411e7f2bbdf377173/text/decision","fixture_id":"semif_c6a411e7f2bbdf377173","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: Mo has authority to approve exhibit labels.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,7576,6516,903,1653,10562,811,51132,16993,18577,810,22922,23369,1339,3759,53859,523,903,1653,1757,5689,8376,811,51132,1377,523,941,7330,4478,902,18135,42183,53859,25274,1295,10562,536,8376,1178,1295,1312,8825,523,18,9886,894,779,4673,535,8376,1178,10562,811,51132,16993,18577,523,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":92,"marker_positions":[60,71,82],"positions":92,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.1703366041183472,1.2382922172546387,-0.3189012110233307],"probabilities_raw":[0.4355661869049072,0.4661942422389984,0.09823955595493317],"probabilities":[0.41755813360214233,0.438327819108963,0.14411407709121704],"argmax_index":1,"argmax":"insufficient","top2_margin":0.02076968550682068,"near_tie":false,"gold":"contradicted","official_answer":{"type":"choice","choice":"insufficient","confidence":0.438327819108963,"probabilities":{"contradicted":0.41755813360214233,"insufficient":0.438327819108963,"supported":0.14411407709121704}},"usage_input_tokens":92,"oracle_file":"records_ref.json"},{"row_id":"semif_c89b86e3f55123bfa506/text/decision","fixture_id":"semif_c89b86e3f55123bfa506","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the egg discovery was fictional.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,6634,11854,768,2151,5854,7267,7935,5329,768,4795,810,4943,876,3498,779,12037,953,1014,37640,523,18,9886,894,779,4673,535,779,7267,12037,953,37640,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,21],"sequence_length":68,"marker_positions":[36,46,57],"positions":68,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-8.718061447143555,3.4809868335723877,1.940075397491455],"probabilities_raw":[4.146997525822371e-06,0.8235937356948853,0.17640209197998047],"probabilities":[0.0001232229551533237,0.7503045797348022,0.24957218766212463],"argmax_index":1,"argmax":"contradicted","top2_margin":0.5007323920726776,"near_tie":false,"gold":"contradicted","official_answer":{"type":"choice","choice":"contradicted","confidence":0.7503045797348022,"probabilities":{"supported":0.0001232229551533237,"contradicted":0.7503045797348022,"insufficient":0.24957218766212463}},"usage_input_tokens":68,"oracle_file":"records_ref.json"},{"row_id":"semif_caf773b23934fa7362a1/text/decision","fixture_id":"semif_caf773b23934fa7362a1","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: permission to scan the notebook is currently valid.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,12153,30403,33300,779,52518,884,17980,523,941,5156,15001,3084,44987,896,1689,1178,1014,47927,936,810,936,6728,6612,523,18,9886,894,779,4673,535,15001,811,19382,779,52518,856,6654,6612,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,21],"sequence_length":75,"marker_positions":[43,53,64],"positions":75,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[3.9008922576904297,-2.43294620513916,0.15340781211853027],"probabilities_raw":[0.9752746224403381,0.0017313139978796244,0.0229940302670002],"probabilities":[0.9262592196464539,0.010040778666734695,0.06370002031326294],"argmax_index":0,"argmax":"supported","top2_margin":0.8625591993331909,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"supported","confidence":0.9262592196464539,"probabilities":{"supported":0.9262592196464539,"contradicted":0.010040778666734695,"insufficient":0.06370002031326294}},"usage_input_tokens":75,"oracle_file":"records_ref.json"},{"row_id":"semif_cdf7f428857e1b841cb3/text/decision","fixture_id":"semif_cdf7f428857e1b841cb3","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the candidate holding valid prop approval authority under the stated delegation rule.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,7330,22558,2314,53859,968,779,7576,1203,23369,2796,3759,53859,523,46050,931,835,535,4411,1550,13289,10562,4010,988,879,523,46050,931,879,535,941,7576,5828,15011,1123,3346,17784,10562,523,18,15534,779,13719,12467,6612,3346,17784,10562,1490,779,9519,53859,7330,523,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,879,535,46050,931,879,20,21],"sequence_length":84,"marker_positions":[57,65,76],"positions":84,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.717790365219116,1.3965704441070557,2.6717827320098877],"probabilities_raw":[0.45007655024528503,0.120084747672081,0.42983871698379517],"probabilities":[0.4243011772632599,0.16511568427085876,0.41058313846588135],"argmax_index":0,"argmax":"A","top2_margin":0.01371803879737854,"near_tie":true,"gold":"B","official_answer":{"type":"choice","choice":"A","confidence":0.4243011772632599,"probabilities":{"A":0.4243011772632599,"insufficient":0.16511568427085876,"B":0.41058313846588135}},"usage_input_tokens":84,"oracle_file":"records_ref.json"},{"row_id":"semif_cdfa9eed51fafcf09b72/text/decision","fixture_id":"semif_cdfa9eed51fafcf09b72","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"The stage manager may approve props but may not delegate that authority. Only authorized approvers may approve. May the assistant approve a prop?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,16396,5828,3575,6229,779,16701,811,51132,41955,523,18,1098,5937,11430,1400,51132,41955,1203,1400,1014,40500,896,10562,523,11005,30403,2396,2053,1400,51132,523,3354,779,16701,51132,768,3346,540,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,21],"sequence_length":82,"marker_positions":[42,59,71],"positions":82,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.991889238357544,0.2586843967437744,4.35775089263916],"probabilities_raw":[0.08453133702278137,0.014938133768737316,0.9005304574966431],"probabilities":[0.1490413397550583,0.043212514370679855,0.807746171951294],"argmax_index":2,"argmax":"permitted","top2_margin":0.6587048321962357,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"permitted","confidence":0.807746171951294,"probabilities":{"insufficient":0.1490413397550583,"prohibited":0.043212514370679855,"permitted":0.807746171951294}},"usage_input_tokens":82,"oracle_file":"records_ref.json"},{"row_id":"semif_d0e36c4cb10a29b3191c/text/decision","fixture_id":"semif_d0e36c4cb10a29b3191c","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the sculpture garden?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,33648,1178,768,37181,6175,1203,1352,8594,3508,938,1014,21706,523,18,542,37181,6175,22558,9544,8544,811,779,3508,936,13546,523,3354,779,33648,7157,779,30122,9041,540,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,21],"sequence_length":78,"marker_positions":[38,55,66],"positions":78,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[3.2459259033203125,3.2828853130340576,1.2678097486495972],"probabilities_raw":[0.45956292748451233,0.47686588764190674,0.06357114017009735],"probabilities":[0.44049862027168274,0.4522833526134491,0.10721801966428757],"argmax_index":1,"argmax":"permitted","top2_margin":0.011784732341766357,"near_tie":true,"gold":"insufficient","official_answer":{"type":"choice","choice":"permitted","confidence":0.4522833526134491,"probabilities":{"insufficient":0.44049862027168274,"permitted":0.4522833526134491,"prohibited":0.10721801966428757}},"usage_input_tokens":78,"oracle_file":"records_ref.json"},{"row_id":"semif_d167e51bb1f499099645/text/decision","fixture_id":"semif_d167e51bb1f499099645","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the candidate ready according to the controlling signed checklist.","criteria":{"A":"Candidate A","B":"Candidate B","insufficient":"Neither candidate supplies the requested evidence"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,10423,57119,12199,19980,523,46050,931,835,535,941,57119,3498,8177,810,22537,3498,1014,8177,523,46050,931,879,535,941,57119,3498,1014,8177,810,22537,3498,8177,523,18,15534,779,13719,8177,4262,811,779,19670,10423,57119,523,19,16,835,535,46050,931,835,20,19,16,879,535,46050,931,879,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,21],"sequence_length":76,"marker_positions":[49,57,65],"positions":76,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.7584027051925659,0.9608060121536255,1.1365561485290527],"probabilities_raw":[0.27144932746887207,0.33234673738479614,0.3962039053440094],"probabilities":[0.28854256868362427,0.3334279954433441,0.37802937626838684],"argmax_index":2,"argmax":"insufficient","top2_margin":0.044601380825042725,"near_tie":false,"gold":"A","official_answer":{"type":"choice","choice":"insufficient","confidence":0.37802937626838684,"probabilities":{"A":0.28854256868362427,"B":0.3334279954433441,"insufficient":0.37802937626838684}},"usage_input_tokens":76,"oracle_file":"records_ref.json"},{"row_id":"semif_d41072dc35d8cd14c056/text/decision","fixture_id":"semif_d41072dc35d8cd14c056","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Under the alternative rule, the badge requires both first-aid and mountain-navigation training. May this guide receive it?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,8170,7850,1509,9764,833,4611,1203,1014,11820,37414,9007,4611,523,18,15246,779,8530,7330,521,779,59534,6835,2059,1509,9764,833,810,11820,37414,9007,4611,523,3354,1033,8170,6984,936,540,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":82,"marker_positions":[42,53,65],"positions":82,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.1964302361011505,0.45343083143234253,1.8850579261779785],"probabilities_raw":[0.12978427112102509,0.16781702637672424,0.7023987770080566],"probabilities":[0.18042559921741486,0.21678446233272552,0.6027899384498596],"argmax_index":2,"argmax":"insufficient","top2_margin":0.3860054761171341,"near_tie":false,"gold":"prohibited","official_answer":{"type":"choice","choice":"insufficient","confidence":0.6027899384498596,"probabilities":{"permitted":0.18042559921741486,"prohibited":0.21678446233272552,"insufficient":0.6027899384498596}},"usage_input_tokens":82,"oracle_file":"records_ref.json"},{"row_id":"semif_d51b05bfb9d6d7f09331/text/decision","fixture_id":"semif_d51b05bfb9d6d7f09331","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"The stage manager may approve props but may not delegate that authority. Only authorized approvers may approve. May the assistant approve a prop?","criteria":{"prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,2797,16701,4117,884,779,3893,521,1203,1295,17784,25274,933,53859,3129,938,4617,523,18,1098,5937,11430,1400,51132,41955,1203,1400,1014,40500,896,10562,523,11005,30403,2396,2053,1400,51132,523,3354,779,16701,51132,768,3346,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,21],"sequence_length":88,"marker_positions":[48,60,77],"positions":88,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.244246482849121,2.568499803543091,2.9976813793182373],"probabilities_raw":[0.0949326828122139,0.3568877577781677,0.548179566860199],"probabilities":[0.14135213196277618,0.3640234172344208,0.49462443590164185],"argmax_index":2,"argmax":"permitted","top2_margin":0.13060101866722107,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"permitted","confidence":0.49462443590164185,"probabilities":{"prohibited":0.14135213196277618,"insufficient":0.3640234172344208,"permitted":0.49462443590164185}},"usage_input_tokens":88,"oracle_file":"records_ref.json"},{"row_id":"semif_d6de731cca52359b1c87/text/decision","fixture_id":"semif_d6de731cca52359b1c87","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate reporting an awarded grant.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,42271,6602,1352,2154,523,46050,931,879,535,941,5771,1178,768,7392,2080,523,18,52739,779,13719,15603,902,14708,8825,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,835,535,46050,931,835,20,19,16,879,535,46050,931,879,20,21],"sequence_length":60,"marker_positions":[33,44,52],"positions":60,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.32768940925598145,-2.296766757965088,-3.953437328338623],"probabilities_raw":[0.8574749231338501,0.1196911409497261,0.022833848372101784],"probabilities":[0.7575771808624268,0.18558989465236664,0.05683298781514168],"argmax_index":0,"argmax":"insufficient","top2_margin":0.5719872862100601,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.7575771808624268,"probabilities":{"insufficient":0.7575771808624268,"A":0.18558989465236664,"B":0.05683298781514168}},"usage_input_tokens":60,"oracle_file":"records_ref.json"},{"row_id":"semif_d7e84e4245a29817da1f/text/decision","fixture_id":"semif_d7e84e4245a29817da1f","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate asking for complete replacement.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,12153,21090,811,1039,578,786,779,7605,8449,2158,10512,1352,6212,523,46050,931,879,535,941,12153,21090,811,7827,779,8449,810,6812,768,5156,16400,523,18,52739,779,13719,14065,875,5156,16400,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,835,535,46050,931,835,20,19,16,879,535,46050,931,879,20,21],"sequence_length":75,"marker_positions":[48,59,67],"positions":75,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.3053580522537231,-1.3997042179107666,-3.440410614013672],"probabilities_raw":[0.4930465519428253,0.44865643978118896,0.058297015726566315],"probabilities":[0.4645945131778717,0.4343149662017822,0.10109055042266846],"argmax_index":0,"argmax":"insufficient","top2_margin":0.030279546976089478,"near_tie":false,"gold":"B","official_answer":{"type":"choice","choice":"insufficient","confidence":0.4645945131778717,"probabilities":{"insufficient":0.4645945131778717,"A":0.4343149662017822,"B":0.10109055042266846}},"usage_input_tokens":75,"oracle_file":"records_ref.json"},{"row_id":"semif_d880e65fef5e2993fdd0/text/decision","fixture_id":"semif_d880e65fef5e2993fdd0","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Accept defective paint even if opened or mixed with solvent; neither condition overrides the defect exception. May this return be accepted?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,8414,5799,22654,1178,768,12803,13789,810,1236,6034,5231,41095,523,9758,11445,938,11649,523,18,49290,57077,5799,1919,1288,8414,933,10372,916,41095,536,13834,4576,1391,58340,779,13789,11287,523,3354,1033,1789,874,11086,540,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,21],"sequence_length":88,"marker_positions":[48,59,76],"positions":88,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.5926122665405273,1.731564998626709,0.0559811033308506],"probabilities_raw":[0.4229806661605835,0.4860343039035797,0.09098505973815918],"probabilities":[0.4101724624633789,0.45297500491142273,0.13685256242752075],"argmax_index":1,"argmax":"insufficient","top2_margin":0.04280254244804382,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"insufficient","confidence":0.45297500491142273,"probabilities":{"permitted":0.4101724624633789,"insufficient":0.45297500491142273,"prohibited":0.13685256242752075}},"usage_input_tokens":88,"oracle_file":"records_ref.json"},{"row_id":"semif_def9fe59d262357a673a/text/decision","fixture_id":"semif_def9fe59d262357a673a","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate reporting an observed test result.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,18779,856,3460,523,46050,931,879,535,941,1927,770,856,797,768,64098,523,18,52739,779,13719,15603,902,6023,1927,1725,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,835,535,46050,931,835,20,19,16,879,535,46050,931,879,20,21],"sequence_length":61,"marker_positions":[34,45,53],"positions":61,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.8470367193222046,1.5626904964447021,-4.1368608474731445],"probabilities_raw":[0.08218152821063995,0.9147562980651855,0.0030621157493442297],"probabilities":[0.1495330035686493,0.83620685338974,0.0142601877450943],"argmax_index":1,"argmax":"A","top2_margin":0.6866738498210907,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"A","confidence":0.83620685338974,"probabilities":{"insufficient":0.1495330035686493,"A":0.83620685338974,"B":0.0142601877450943}},"usage_input_tokens":61,"oracle_file":"records_ref.json"},{"row_id":"semif_e0c140e9222d1bce5647/text/decision","fixture_id":"semif_e0c140e9222d1bce5647","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the guest endorses the listing's towel statement.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,21522,16461,521,997,1098,21008,3498,25583,1706,938,21706,523,859,11667,535,7448,25583,1706,1166,4617,2013,1727,5776,2927,18,9886,894,779,4673,535,779,21522,30261,774,779,21008,1090,50624,8294,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,21],"sequence_length":76,"marker_positions":[44,54,65],"positions":76,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[4.24207878112793,-1.3785492181777954,0.46038830280303955],"probabilities_raw":[0.9742728471755981,0.003529172856360674,0.022197969257831573],"probabilities":[0.9215284585952759,0.01662665419280529,0.06184489652514458],"argmax_index":0,"argmax":"supported","top2_margin":0.8596835620701313,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"supported","confidence":0.9215284585952759,"probabilities":{"supported":0.9215284585952759,"contradicted":0.01662665419280529,"insufficient":0.06184489652514458}},"usage_input_tokens":76,"oracle_file":"records_ref.json"},{"row_id":"semif_e1d610bd14e3d16c09b9/text/decision","fixture_id":"semif_e1d610bd14e3d16c09b9","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate opposing a paywall.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,10046,31494,768,18958,875,768,4025,17686,1550,811,17575,2723,936,523,46050,931,879,535,941,10046,30261,774,26677,768,4025,17686,523,18,52739,779,13719,36154,768,4025,17686,523,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,879,535,46050,931,879,20,21],"sequence_length":71,"marker_positions":[44,52,63],"positions":71,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.24006280303001404,0.42934364080429077,-0.9660718441009521],"probabilities_raw":[0.3987666666507721,0.48186177015304565,0.11937165260314941],"probabilities":[0.389519065618515,0.4459127187728882,0.16456826031208038],"argmax_index":1,"argmax":"insufficient","top2_margin":0.05639365315437317,"near_tie":false,"gold":"A","official_answer":{"type":"choice","choice":"insufficient","confidence":0.4459127187728882,"probabilities":{"A":0.389519065618515,"insufficient":0.4459127187728882,"B":0.16456826031208038}},"usage_input_tokens":71,"oracle_file":"records_ref.json"},{"row_id":"semif_e332aa927d0c2ed044aa/text/decision","fixture_id":"semif_e332aa927d0c2ed044aa","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: Lina has authority to approve exhibit labels.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,7576,6516,903,1653,10562,811,51132,16993,18577,810,22922,23369,1339,3759,53859,523,903,1653,1757,5689,8376,811,51132,1377,523,941,7330,4478,902,18135,42183,53859,25274,1295,10562,536,8376,1178,1295,1312,8825,523,18,9886,894,779,4673,535,903,1653,1178,10562,811,51132,16993,18577,523,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":93,"marker_positions":[61,72,83],"positions":93,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.6507751941680908,0.9516758918762207,1.8932214975357056],"probabilities_raw":[0.17196454107761383,0.23233702778816223,0.5956984758377075],"probabilities":[0.21418289840221405,0.26554253697395325,0.5202745199203491],"argmax_index":2,"argmax":"supported","top2_margin":0.2547319829463959,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"supported","confidence":0.5202745199203491,"probabilities":{"contradicted":0.21418289840221405,"insufficient":0.26554253697395325,"supported":0.5202745199203491}},"usage_input_tokens":93,"oracle_file":"records_ref.json"},{"row_id":"semif_e6155431ab7b0f8170d4/text/decision","fixture_id":"semif_e6155431ab7b0f8170d4","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate reporting completed delivery.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,859,1011,5514,768,52414,1079,27968,523,46050,931,879,535,941,52414,1079,953,14465,523,18,52739,779,13719,15603,7850,11746,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,21],"sequence_length":60,"marker_positions":[33,44,52],"positions":60,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.8405516147613525,0.2589983344078064,-0.3213144540786743],"probabilities_raw":[0.17594608664512634,0.5283334255218506,0.29572051763534546],"probabilities":[0.21540363132953644,0.47246497869491577,0.31213143467903137],"argmax_index":1,"argmax":"B","top2_margin":0.1603335440158844,"near_tie":false,"gold":"B","official_answer":{"type":"choice","choice":"B","confidence":0.47246497869491577,"probabilities":{"insufficient":0.21540363132953644,"B":0.47246497869491577,"A":0.31213143467903137}},"usage_input_tokens":60,"oracle_file":"records_ref.json"},{"row_id":"semif_ea7ec66a5e8eac886548/text/decision","fixture_id":"semif_ea7ec66a5e8eac886548","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"A sample may be made when sample production is approved. May the sample be made?","criteria":{"permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,22773,2396,2240,3525,768,6643,59534,1203,4943,876,916,4289,592,17784,875,3507,3893,523,18,542,6643,1400,874,2047,1403,6643,3893,856,12043,523,3354,779,6643,874,2047,540,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,21],"sequence_length":80,"marker_positions":[40,51,68],"positions":80,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.49817439913749695,1.6548174619674683,1.9368813037872314],"probabilities_raw":[0.04755781963467598,0.40950050950050354,0.542941689491272],"probabilities":[0.08811099082231522,0.410165399312973,0.5017235279083252],"argmax_index":2,"argmax":"prohibited","top2_margin":0.09155812859535217,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"prohibited","confidence":0.5017235279083252,"probabilities":{"permitted":0.08811099082231522,"insufficient":0.410165399312973,"prohibited":0.5017235279083252}},"usage_input_tokens":80,"oracle_file":"records_ref.json"},{"row_id":"semif_eac5121a4a1bf391a08c/text/decision","fixture_id":"semif_eac5121a4a1bf391a08c","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: terminal Maple is online.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,59063,15994,2059,55501,1203,4654,1512,1149,7845,5742,523,18,9886,894,779,4673,535,19459,60468,856,4175,523,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":58,"marker_positions":[26,37,48],"positions":58,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.0061943531036377,3.7291605472564697,-3.016935348510742],"probabilities_raw":[0.15133854746818542,0.8476650714874268,0.0009963964112102985],"probabilities":[0.2246410846710205,0.7691481113433838,0.006210819352418184],"argmax_index":1,"argmax":"insufficient","top2_margin":0.5445070266723633,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.7691481113433838,"probabilities":{"contradicted":0.2246410846710205,"insufficient":0.7691481113433838,"supported":0.006210819352418184}},"usage_input_tokens":58,"oracle_file":"records_ref.json"},{"row_id":"semif_eafc22c8c40df3932a8e/text/decision","fixture_id":"semif_eafc22c8c40df3932a8e","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: under the supplied protocol, the crate is currently in storage.","criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,13061,6602,779,30174,17150,521,1203,779,2835,30174,810,26862,2811,952,21148,875,779,2391,931,938,49033,523,18,9886,894,779,4673,535,1490,779,21706,13061,521,779,2391,931,856,6654,797,8784,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,21],"sequence_length":77,"marker_positions":[45,55,66],"positions":77,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.8015472888946533,0.15241731703281403,2.426419258117676],"probabilities_raw":[0.004840384237468243,0.09284752607345581,0.9023120999336243],"probabilities":[0.01956266537308693,0.16137823462486267,0.8190590739250183],"argmax_index":2,"argmax":"insufficient","top2_margin":0.6576808393001556,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.8190590739250183,"probabilities":{"supported":0.01956266537308693,"contradicted":0.16137823462486267,"insufficient":0.8190590739250183}},"usage_input_tokens":77,"oracle_file":"records_ref.json"},{"row_id":"semif_ebd74aa05324a1556288/text/decision","fixture_id":"semif_ebd74aa05324a1556288","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate asking to repair the existing window.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,12153,21090,811,1039,578,786,779,7605,8449,2158,10512,1352,6212,523,46050,931,879,535,941,12153,21090,811,7827,779,8449,810,6812,768,5156,16400,523,18,52739,779,13719,14065,811,12940,779,7605,8449,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,835,535,46050,931,835,20,19,16,879,535,46050,931,879,20,21],"sequence_length":77,"marker_positions":[50,61,69],"positions":77,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.8500902652740479,-1.6692605018615723,-4.712647914886475],"probabilities_raw":[0.44339337944984436,0.5312789082527161,0.02532772719860077],"probabilities":[0.4410553276538849,0.5018713474273682,0.05707336589694023],"argmax_index":1,"argmax":"A","top2_margin":0.060816019773483276,"near_tie":false,"gold":"A","official_answer":{"type":"choice","choice":"A","confidence":0.5018713474273682,"probabilities":{"insufficient":0.4410553276538849,"A":0.5018713474273682,"B":0.05707336589694023}},"usage_input_tokens":77,"oracle_file":"records_ref.json"},{"row_id":"semif_ebea4681e3dcc7d92ca2/text/decision","fixture_id":"semif_ebea4681e3dcc7d92ca2","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Under the alternative priority, choose B whenever B is able; choose A only when B is unable and A is able.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B","A":"Candidate A"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,856,779,16495,57490,810,856,11649,3222,811,9936,1033,3869,523,46050,931,879,856,779,3981,3321,57490,810,856,1236,3222,811,9936,936,523,18,15246,779,8530,17150,521,7056,879,19148,879,856,3222,536,7056,835,1550,1403,879,856,12359,810,835,856,3222,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,21],"sequence_length":87,"marker_positions":[60,71,79],"positions":87,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.0848593711853027,1.2152581214904785,2.261540174484253],"probabilities_raw":[0.3827943205833435,0.1604362428188324,0.4567693769931793],"probabilities":[0.37427595257759094,0.2011004239320755,0.42462357878685],"argmax_index":2,"argmax":"A","top2_margin":0.05034762620925903,"near_tie":false,"gold":"B","official_answer":{"type":"choice","choice":"A","confidence":0.42462357878685,"probabilities":{"insufficient":0.37427595257759094,"B":0.2011004239320755,"A":0.42462357878685}},"usage_input_tokens":87,"oracle_file":"records_ref.json"},{"row_id":"semif_ec94ef909f7918695ab3/text/decision","fixture_id":"semif_ec94ef909f7918695ab3","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: the corrected date is in July.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,40348,3575,5405,592,779,5562,811,4963,810,4943,876,22879,592,779,7045,5105,5562,523,18,9886,894,779,4673,535,779,35101,5562,856,797,5105,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":67,"marker_positions":[35,45,56],"positions":67,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-11.032297134399414,0.18279242515563965,-0.3950861096382141],"probabilities_raw":[8.628144314570818e-06,0.6405735611915588,0.35941776633262634],"probabilities":[0.0001995474158320576,0.6016390919685364,0.3981613516807556],"argmax_index":1,"argmax":"insufficient","top2_margin":0.20347774028778076,"near_tie":false,"gold":"contradicted","official_answer":{"type":"choice","choice":"insufficient","confidence":0.6016390919685364,"probabilities":{"supported":0.0001995474158320576,"insufficient":0.6016390919685364,"contradicted":0.3981613516807556}},"usage_input_tokens":67,"oracle_file":"records_ref.json"},{"row_id":"semif_edb708bd0fe5fd67d6ee/text/decision","fixture_id":"semif_edb708bd0fe5fd67d6ee","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"For this fictional protocol, the first recorded decision is binding and later withdrawals are ignored. May the surveyor enter the attic?","criteria":{"permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,2681,28117,1509,6767,768,8153,777,811,7157,779,1530,793,523,13629,779,5089,1689,22922,46159,896,15001,523,3253,2919,6177,9647,523,18,3111,1033,37640,13061,521,779,1509,8112,6177,856,13827,810,2919,17688,1486,938,21867,523,3354,779,8153,777,7157,779,1530,793,540,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":99,"marker_positions":[59,70,82],"positions":99,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.6859942078590393,0.7516666054725647,2.53814697265625],"probabilities_raw":[0.11846372485160828,0.12650465965270996,0.7550315856933594],"probabilities":[0.17232663929462433,0.1806035041809082,0.6470698714256287],"argmax_index":2,"argmax":"insufficient","top2_margin":0.46646636724472046,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"insufficient","confidence":0.6470698714256287,"probabilities":{"permitted":0.17232663929462433,"prohibited":0.1806035041809082,"insufficient":0.6470698714256287}},"usage_input_tokens":99,"oracle_file":"records_ref.json"},{"row_id":"semif_eeae283841ce75dd25bf/text/decision","fixture_id":"semif_eeae283841ce75dd25bf","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Change an account field only when the owner authorizes changing that field. May the display name be changed?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,3863,12153,22558,9365,779,6332,2154,1203,4943,876,23369,2796,9365,779,11256,8355,523,18,18693,902,3863,2853,1550,1403,779,12153,3575,6229,9365,896,2853,523,3354,779,6332,2154,874,7093,540,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,21],"sequence_length":83,"marker_positions":[43,60,71],"positions":83,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.9277441501617432,1.6321382522583008,3.0559921264648438],"probabilities_raw":[0.20685440301895142,0.1539163440465927,0.6392292380332947],"probabilities":[0.24700720608234406,0.199987530708313,0.5530052781105042],"argmax_index":2,"argmax":"prohibited","top2_margin":0.3059980720281601,"near_tie":false,"gold":"permitted","official_answer":{"type":"choice","choice":"prohibited","confidence":0.5530052781105042,"probabilities":{"insufficient":0.24700720608234406,"permitted":0.199987530708313,"prohibited":0.5530052781105042}},"usage_input_tokens":83,"oracle_file":"records_ref.json"},{"row_id":"semif_ef3e0f8e657bf59f8a29/text/decision","fixture_id":"semif_ef3e0f8e657bf59f8a29","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the candidate ready according to the controlling signed checklist.","criteria":{"A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence","B":"Candidate B"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,3111,1033,4889,521,779,2835,10423,57119,12199,52010,1403,936,19980,916,22537,523,46050,931,835,535,40831,3498,8177,521,1203,779,10423,57119,3498,1014,8177,523,46050,931,879,535,40831,3498,1014,8177,521,1203,779,10423,57119,3498,8177,523,18,15534,779,13719,8177,4262,811,779,19670,10423,57119,523,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,879,535,46050,931,879,20,21],"sequence_length":90,"marker_positions":[63,71,82],"positions":90,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.09687026590108871,1.2967870235443115,0.46955496072769165],"probabilities_raw":[0.1472424864768982,0.5933225154876709,0.2594349682331085],"probabilities":[0.19212587177753448,0.5199291706085205,0.2879450023174286],"argmax_index":1,"argmax":"insufficient","top2_margin":0.23198416829109192,"near_tie":false,"gold":"B","official_answer":{"type":"choice","choice":"insufficient","confidence":0.5199291706085205,"probabilities":{"A":0.19212587177753448,"insufficient":0.5199291706085205,"B":0.2879450023174286}},"usage_input_tokens":90,"oracle_file":"records_ref.json"},{"row_id":"semif_f2b4ec4930322fe45118/text/decision","fixture_id":"semif_f2b4ec4930322fe45118","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Sell copies only with the architect's sales permission. May copies be sold?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,6765,22558,10004,2782,768,2984,797,902,7477,4611,62248,810,4943,876,23369,2796,18109,16711,523,18,560,965,16711,1550,916,779,6765,1090,11599,15001,523,3354,16711,874,4997,540,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,21],"sequence_length":80,"marker_positions":[40,57,69],"positions":80,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.063524007797241,3.0282185077667236,0.5558172464370728],"probabilities_raw":[0.26005077362060547,0.6823695302009583,0.057579707354307175],"probabilities":[0.30006879568099976,0.597724199295044,0.1022069901227951],"argmax_index":1,"argmax":"prohibited","top2_margin":0.2976554036140442,"near_tie":false,"gold":"prohibited","official_answer":{"type":"choice","choice":"prohibited","confidence":0.597724199295044,"probabilities":{"insufficient":0.30006879568099976,"prohibited":0.597724199295044,"permitted":0.1022069901227951}},"usage_input_tokens":80,"oracle_file":"records_ref.json"},{"row_id":"semif_f3d13cf184f916677def/text/decision","fixture_id":"semif_f3d13cf184f916677def","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: under the supplied protocol, the crate is currently in storage.","criteria":{"contradicted":"The evidence establishes the opposite","insufficient":"The evidence does not establish either","supported":"The evidence establishes the claim"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,26862,3446,3498,779,2391,931,856,797,8784,523,941,51746,30174,3498,936,1178,3191,8784,523,9758,7944,938,2835,523,941,21706,13061,3498,779,51746,30174,12199,6249,9448,19148,1518,1547,5617,9171,523,18,9886,894,779,4673,535,1490,779,21706,13061,521,779,2391,931,856,6654,797,8784,523,19,16,18197,13735,535,941,4589,48308,779,11977,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,7951,535,941,4589,48308,779,4673,20,21],"sequence_length":95,"marker_positions":[63,74,85],"positions":95,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.1147890090942383,1.3649086952209473,3.313030481338501],"probabilities_raw":[0.08854779601097107,0.11371122300624847,0.7977409362792969],"probabilities":[0.14278262853622437,0.1707146316766739,0.6865026950836182],"argmax_index":2,"argmax":"supported","top2_margin":0.5157880634069443,"near_tie":false,"gold":"contradicted","official_answer":{"type":"choice","choice":"supported","confidence":0.6865026950836182,"probabilities":{"contradicted":0.14278262853622437,"insufficient":0.1707146316766739,"supported":0.6865026950836182}},"usage_input_tokens":95,"oracle_file":"records_ref.json"},{"row_id":"semif_f40beba9088c8db8bbd6/text/decision","fixture_id":"semif_f40beba9088c8db8bbd6","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: replacement lenses were ordered.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,2525,7867,13055,16400,31286,523,941,23120,44987,1186,1052,1014,4763,1381,25539,811,779,13608,1090,28623,523,18,9886,894,779,4673,535,16400,31286,1166,13055,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":68,"marker_positions":[36,46,57],"positions":68,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[9.068342208862305,0.18565919995307922,-5.66766357421875],"probabilities_raw":[0.9998608827590942,0.00013875206059310585,3.9826630882089376e-07],"probabilities":[0.998221218585968,0.0017519677057862282,2.6769057512865402e-05],"argmax_index":0,"argmax":"supported","top2_margin":0.9964692508801818,"near_tie":false,"gold":"supported","official_answer":{"type":"choice","choice":"supported","confidence":0.998221218585968,"probabilities":{"supported":0.998221218585968,"insufficient":0.0017519677057862282,"contradicted":2.6769057512865402e-05}},"usage_input_tokens":68,"oracle_file":"records_ref.json"},{"row_id":"semif_f41ae66b3c09630c7a42/text/decision","fixture_id":"semif_f41ae66b3c09630c7a42","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate asking to repair the existing window.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,27022,779,8449,810,5466,768,16400,523,46050,931,879,535,2382,578,786,779,7605,8449,810,3089,1352,6212,523,18,52739,779,13719,14065,811,12940,779,7605,8449,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,835,535,46050,931,835,20,19,16,879,535,46050,931,879,20,21],"sequence_length":68,"marker_positions":[41,52,60],"positions":68,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.4962294399738312,-0.13544343411922455,0.3153257966041565],"probabilities_raw":[0.4226154386997223,0.2247052937746048,0.3526792526245117],"probabilities":[0.3975161612033844,0.25315651297569275,0.34932732582092285],"argmax_index":0,"argmax":"insufficient","top2_margin":0.04818883538246155,"near_tie":false,"gold":"B","official_answer":{"type":"choice","choice":"insufficient","confidence":0.3975161612033844,"probabilities":{"insufficient":0.3975161612033844,"A":0.25315651297569275,"B":0.34932732582092285}},"usage_input_tokens":68,"oracle_file":"records_ref.json"},{"row_id":"semif_f46f392ef9e9e9df564b/text/decision","fixture_id":"semif_f46f392ef9e9e9df564b","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Assess the claim: permission to scan the notebook is currently valid.","criteria":{"supported":"The evidence establishes the claim","insufficient":"The evidence does not establish either","contradicted":"The evidence establishes the opposite"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,12153,30403,33300,884,17980,523,941,21706,3084,4654,1512,1113,2919,15001,3807,810,2177,1014,6122,2835,28195,523,18,9886,894,779,4673,535,15001,811,19382,779,52518,856,6654,6612,523,19,16,7951,535,941,4589,48308,779,4673,20,19,16,28441,535,941,4589,2177,1014,6122,4021,20,19,16,18197,13735,535,941,4589,48308,779,11977,20,21],"sequence_length":73,"marker_positions":[41,51,62],"positions":73,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.2643077373504639,2.459071397781372,-1.124413251876831],"probabilities_raw":[0.022959887981414795,0.9506327509880066,0.02640737034380436],"probabilities":[0.060983750969171524,0.8716233968734741,0.06739288568496704],"argmax_index":1,"argmax":"insufficient","top2_margin":0.8042305111885071,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.8716233968734741,"probabilities":{"supported":0.060983750969171524,"insufficient":0.8716233968734741,"contradicted":0.06739288568496704}},"usage_input_tokens":73,"oracle_file":"records_ref.json"},{"row_id":"semif_f6d68126b43f8c9ed183/text/decision","fixture_id":"semif_f6d68126b43f8c9ed183","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate reporting an observed test result.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,6634,8269,896,779,7850,33037,1927,7410,523,46050,931,879,535,941,6634,15994,33037,29067,7441,1203,3498,1295,1927,953,2685,523,18,52739,779,13719,15603,902,6023,1927,1725,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,835,535,46050,931,835,20,19,16,879,535,46050,931,879,20,21],"sequence_length":71,"marker_positions":[44,55,63],"positions":71,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.3054568767547607,-1.7994532585144043,-3.502875328063965],"probabilities_raw":[0.580965518951416,0.3544952869415283,0.06453920900821686],"probabilities":[0.5233494639396667,0.36773747205734253,0.10891313850879669],"argmax_index":0,"argmax":"insufficient","top2_margin":0.15561199188232422,"near_tie":false,"gold":"A","official_answer":{"type":"choice","choice":"insufficient","confidence":0.5233494639396667,"probabilities":{"insufficient":0.5233494639396667,"A":0.36773747205734253,"B":0.10891313850879669}},"usage_input_tokens":71,"oracle_file":"records_ref.json"},{"row_id":"semif_f756e1cb04e11766341e/text/decision","fixture_id":"semif_f756e1cb04e11766341e","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Under the alternative protocol, chat controls conflicts. Select the candidate ready according to chat.","criteria":{"insufficient":"Neither candidate supplies the requested evidence","A":"Candidate A","B":"Candidate B"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,3111,1033,4889,521,779,2835,10423,57119,12199,52010,1403,936,19980,916,22537,523,46050,931,835,535,40831,3498,8177,521,1203,779,10423,57119,3498,1014,8177,523,46050,931,879,535,40831,3498,1014,8177,521,1203,779,10423,57119,3498,8177,523,18,15246,779,8530,13061,521,22537,12199,19980,523,22269,779,13719,8177,4262,811,22537,523,19,16,28441,535,42749,13719,14350,779,21968,4589,20,19,16,835,535,46050,931,835,20,19,16,879,535,46050,931,879,20,21],"sequence_length":96,"marker_positions":[69,80,88],"positions":96,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.13149817287921906,-2.9378244876861572,-0.3618924617767334],"probabilities_raw":[0.5391861796379089,0.03258117288351059,0.42823266983032227],"probabilities":[0.5042976140975952,0.0679316371679306,0.4277706742286682],"argmax_index":0,"argmax":"insufficient","top2_margin":0.076526939868927,"near_tie":false,"gold":"A","official_answer":{"type":"choice","choice":"insufficient","confidence":0.5042976140975952,"probabilities":{"insufficient":0.5042976140975952,"A":0.0679316371679306,"B":0.4277706742286682}},"usage_input_tokens":96,"oracle_file":"records_ref.json"},{"row_id":"semif_fb73a85d386cf59b0c51/text/decision","fixture_id":"semif_fb73a85d386cf59b0c51","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Mass production is allowed only after mass-production approval. May it begin?","criteria":{"insufficient":"The supplied information does not settle whether the rule permits the action","permitted":"The stated rule permits the action","prohibited":"The stated rule prohibits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,22773,12800,768,6177,1203,1352,3893,12757,856,58754,523,18,31678,3893,856,6767,1550,1720,3507,25252,980,17784,523,3354,936,6305,540,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,21],"sequence_length":71,"marker_positions":[31,48,59],"positions":71,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.5704766511917114,-1.7801976203918457,2.1247355937957764],"probabilities_raw":[0.3602679967880249,0.0126312505453825,0.6271007061004639],"probabilities":[0.38803747296333313,0.035431068390607834,0.5765314698219299],"argmax_index":2,"argmax":"prohibited","top2_margin":0.1884939968585968,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"prohibited","confidence":0.5765314698219299,"probabilities":{"insufficient":0.38803747296333313,"permitted":0.035431068390607834,"prohibited":0.5765314698219299}},"usage_input_tokens":71,"oracle_file":"records_ref.json"},{"row_id":"semif_fda0ca94652c6d739ecf/text/decision","fixture_id":"semif_fda0ca94652c6d739ecf","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"A wristband permits entry exactly to the areas it covers. May the visitor enter the sculpture garden?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,37181,6175,4943,876,13748,774,779,30122,9041,810,13546,1550,779,1232,33388,920,2543,10074,523,18,542,37181,6175,22558,9544,8544,811,779,3508,936,13546,523,3354,779,33648,7157,779,30122,9041,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":84,"marker_positions":[44,56,67],"positions":84,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[2.2747561931610107,1.4293830394744873,2.278024196624756],"probabilities_raw":[0.41107088327407837,0.17651264369487762,0.4124164879322052],"probabilities":[0.3923072814941406,0.21446861326694489,0.3932241201400757],"argmax_index":2,"argmax":"insufficient","top2_margin":0.0009168386459350586,"near_tie":true,"gold":"prohibited","official_answer":{"type":"choice","choice":"insufficient","confidence":0.3932241201400757,"probabilities":{"prohibited":0.3923072814941406,"permitted":0.21446861326694489,"insufficient":0.3932241201400757}},"usage_input_tokens":84,"oracle_file":"records_ref.json"},{"row_id":"semif_fdb9fc214aea2710763a/text/decision","fixture_id":"semif_fdb9fc214aea2710763a","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Choose the candidate authorizing engraving.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,55945,931,835,535,941,13608,6602,768,2154,523,46050,931,879,535,941,11019,856,12534,523,18,52739,779,13719,3575,4651,41071,3665,523,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,21],"sequence_length":58,"marker_positions":[31,39,47],"positions":58,"window":64,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.5282842516899109,-4.471530914306641,1.901053786277771],"probabilities_raw":[0.20189766585826874,0.0013606270076707006,0.7967416644096375],"probabilities":[0.2706928849220276,0.0076097361743450165,0.7216973900794983],"argmax_index":2,"argmax":"insufficient","top2_margin":0.4510045051574707,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"insufficient","confidence":0.7216973900794983,"probabilities":{"B":0.2706928849220276,"A":0.0076097361743450165,"insufficient":0.7216973900794983}},"usage_input_tokens":58,"oracle_file":"records_ref.json"},{"row_id":"semif_fdd902cc6e65fad7cb4f/text/decision","fixture_id":"semif_fdd902cc6e65fad7cb4f","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"Select the candidate holding valid prop approval authority under the stated delegation rule.","criteria":{"B":"Candidate B","A":"Candidate A","insufficient":"Neither candidate supplies the requested evidence"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,7330,22558,2314,53859,968,779,7576,1203,23369,2796,1523,3759,53859,523,46050,931,835,535,859,5068,3346,17784,10562,5828,988,779,7576,523,46050,931,879,535,859,5068,936,1550,988,835,521,810,1052,1295,1312,8825,523,18,15534,779,13719,12467,6612,3346,17784,10562,1490,779,9519,53859,7330,523,19,16,879,535,46050,931,879,20,19,16,835,535,46050,931,835,20,19,16,28441,535,42749,13719,14350,779,21968,4589,20,21],"sequence_length":91,"marker_positions":[64,72,80],"positions":91,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.6088454723358154,2.894845724105835,1.3413251638412476],"probabilities_raw":[0.07742483913898468,0.7615134716033936,0.16106167435646057],"probabilities":[0.12809723615646362,0.6557409167289734,0.21616187691688538],"argmax_index":1,"argmax":"A","top2_margin":0.439579039812088,"near_tie":false,"gold":"A","official_answer":{"type":"choice","choice":"A","confidence":0.6557409167289734,"probabilities":{"B":0.12809723615646362,"A":0.6557409167289734,"insufficient":0.21616187691688538}},"usage_input_tokens":91,"oracle_file":"records_ref.json"},{"row_id":"semif_ffee861474a9665a093a/text/decision","fixture_id":"semif_ffee861474a9665a093a","mode":"text","question_id":"decision","question":{"type":"choice","instructions":"A resident's pass requires living in the valley and has no workplace restriction. May it be issued?","criteria":{"prohibited":"The stated rule prohibits the action","permitted":"The stated rule permits the action","insufficient":"The supplied information does not settle whether the rule permits the action"}},"type":"choice","qtype":0,"K":3,"media":[],"prefix_rows":0,"sequence_ids":[1,17,1098,3434,919,14350,768,2154,1203,1295,2681,933,1336,6249,523,18,542,12518,1090,2230,6835,4624,797,779,20610,810,1178,1295,21701,27434,523,3354,936,874,12800,540,19,16,33776,535,941,9519,7330,14480,1512,779,4272,20,19,16,20458,535,941,9519,7330,22558,779,4272,20,19,16,28441,535,941,21706,2203,2177,1014,27169,3984,779,7330,22558,779,4272,20,21],"sequence_length":77,"marker_positions":[37,49,60],"positions":77,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.34799861907958984,4.5191450119018555,2.7875254154205322],"probabilities_raw":[0.012943761423230171,0.8386222124099731,0.14843404293060303],"probabilities":[0.03788952901959419,0.745668888092041,0.2164415717124939],"argmax_index":1,"argmax":"permitted","top2_margin":0.5292273163795471,"near_tie":false,"gold":"insufficient","official_answer":{"type":"choice","choice":"permitted","confidence":0.745668888092041,"probabilities":{"prohibited":0.03788952901959419,"permitted":0.745668888092041,"insufficient":0.2164415717124939}},"usage_input_tokens":77,"oracle_file":"records_ref.json"},{"row_id":"tv4_000/text/answer","fixture_id":"tv4_000","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"The arms trade shifts from being a private to a government controlled enterprise.","b":"An increase in the defence trade.","c":"A decrease in the arms trade.","d":"A growth in the number and variety of weapons traded."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,53983,3609,1878,997,39420,6881,997,28828,803,779,2773,2172,1014,25764,906,768,13974,803,779,28857,7169,797,779,730,898,881,3935,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,941,10557,7169,26807,988,1994,768,3674,811,768,3134,10282,25624,523,20,19,16,794,535,1592,3761,797,779,28857,7169,523,20,19,16,781,535,835,10636,797,779,10557,7169,523,20,19,16,778,535,835,4320,797,779,2080,810,5681,803,14773,39451,523,20,21],"sequence_length":103,"marker_positions":[44,63,75,87],"positions":103,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-5.553777694702148,-7.747239112854004,-5.589899063110352,-5.4578471183776855],"probabilities_raw":[0.3147885501384735,0.035108376294374466,0.3036208748817444,0.3464823067188263],"probabilities":[0.3072996139526367,0.06413247436285019,0.29947182536125183,0.32909613847732544],"argmax_index":3,"argmax":"d","top2_margin":0.02179652452468872,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"d","confidence":0.32909613847732544,"probabilities":{"a":0.3072996139526367,"b":0.06413247436285019,"c":0.29947182536125183,"d":0.32909613847732544}},"usage_input_tokens":103,"oracle_file":"records_ref.json"},{"row_id":"tv4_001/text/answer","fixture_id":"tv4_001","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Increased amount of gastric acid.","b":"Gastric ulceration.","c":"Reduced amount of gastric acid.","d":"Pernicious anaemia."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,50988,4331,1878,997,39420,6881,997,3493,856,6365,2807,4791,14232,4027,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,42144,3685,803,35231,6104,523,20,19,16,794,535,15858,1227,41134,852,523,20,19,16,781,535,5148,44771,3685,803,35231,6104,523,20,19,16,778,535,63401,17712,30769,10254,523,20,21],"sequence_length":74,"marker_positions":[31,42,52,64],"positions":74,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.6107369661331177,-4.443951606750488,2.522935152053833,-6.117058753967285],"probabilities_raw":[0.12860864400863647,0.0008204402402043343,0.8704168796539307,0.00015396632079500705],"probabilities":[0.2018294632434845,0.0054557411931455135,0.7910635471343994,0.0016512047732248902],"argmax_index":2,"argmax":"c","top2_margin":0.5892340838909149,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"c","confidence":0.7910635471343994,"probabilities":{"a":0.2018294632434845,"b":0.0054557411931455135,"c":0.7910635471343994,"d":0.0016512047732248902}},"usage_input_tokens":74,"oracle_file":"records_ref.json"},{"row_id":"tv4_002/text/answer","fixture_id":"tv4_002","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"In the long run, all inputs are variable.","b":"In the short run, all inputs are variable.","c":"In the long run, supply is not able to adjust fully to changes in demand.","d":"In the short run, supply is able to adjust fully to changes in demand."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,4167,578,2499,1504,1878,997,39420,6881,997,28828,803,779,2773,856,5405,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,1030,779,1787,2685,521,1113,23032,938,9452,523,20,19,16,794,535,1030,779,3290,2685,521,1113,23032,938,9452,523,20,19,16,781,535,1030,779,1787,2685,521,7018,856,1014,3222,811,8529,7408,811,3807,797,6154,523,20,19,16,778,535,1030,779,3290,2685,521,7018,856,3222,811,8529,7408,811,3807,797,6154,523,20,21],"sequence_length":107,"marker_positions":[34,49,64,86],"positions":107,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-3.5198915004730225,-1.3380160331726074,-1.1915581226348877,-0.26958832144737244],"probabilities_raw":[0.021776100620627403,0.1929997056722641,0.22344081103801727,0.5617833733558655],"probabilities":[0.04711969569325447,0.2239195704460144,0.24861547350883484,0.4803452491760254],"argmax_index":3,"argmax":"d","top2_margin":0.23172977566719055,"near_tie":false,"gold":"a","official_answer":{"type":"choice","choice":"d","confidence":0.4803452491760254,"probabilities":{"a":0.04711969569325447,"b":0.2239195704460144,"c":0.24861547350883484,"d":0.4803452491760254}},"usage_input_tokens":107,"oracle_file":"records_ref.json"},{"row_id":"tv4_003/text/answer","fixture_id":"tv4_003","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Many pessimists die at a younger age","b":"They do not really know what is going on","c":"This may be a warning sign for dementia","d":"Our measures have low validity"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,28009,19179,1878,997,39420,6881,997,5711,4092,896,6993,7239,1914,1569,3476,803,62151,856,896,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,6405,62311,834,1817,1277,963,768,12280,3209,20,19,16,794,535,2216,1112,1014,3896,1842,1620,856,3529,884,20,19,16,781,535,1470,1400,874,768,15513,1723,875,22344,20,19,16,778,535,6504,7485,1052,2920,28195,20,21],"sequence_length":87,"marker_positions":[36,50,64,77],"positions":87,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.4365646839141846,-1.0311179161071777,-3.46833872795105,-1.6626391410827637],"probabilities_raw":[0.13154460489749908,0.5363531708717346,0.046879105269908905,0.2852230668067932],"probabilities":[0.1681860387325287,0.4589926302433014,0.08048214763402939,0.2923392355442047],"argmax_index":1,"argmax":"b","top2_margin":0.16665339469909668,"near_tie":false,"gold":"a","official_answer":{"type":"choice","choice":"b","confidence":0.4589926302433014,"probabilities":{"a":0.1681860387325287,"b":0.4589926302433014,"c":0.08048214763402939,"d":0.2923392355442047}},"usage_input_tokens":87,"oracle_file":"records_ref.json"},{"row_id":"tv4_004/text/answer","fixture_id":"tv4_004","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Were born before the centennial","b":"Study old age and older adults","c":"Are 100 or more years old","d":"Have not yet reached old age"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,28009,19179,1878,997,39420,6881,997,544,22525,20800,938,1519,1316,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,43391,5747,2076,779,1739,24452,20,19,16,794,535,12220,2731,3209,810,6993,7239,20,19,16,781,535,9195,730,2096,933,1209,1779,2731,20,19,16,778,535,14566,1014,4763,8107,2731,3209,20,21],"sequence_length":75,"marker_positions":[30,41,52,64],"positions":75,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.9950649738311768,-0.7058923244476318,4.306713104248047,0.4114217162132263],"probabilities_raw":[0.0017816874897107482,0.006467134226113558,0.9719834327697754,0.01976766437292099],"probabilities":[0.010075385682284832,0.025304943323135376,0.9084072113037109,0.05621250718832016],"argmax_index":2,"argmax":"c","top2_margin":0.8521947041153908,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"c","confidence":0.9084072113037109,"probabilities":{"a":0.010075385682284832,"b":0.025304943323135376,"c":0.9084072113037109,"d":0.05621250718832016}},"usage_input_tokens":75,"oracle_file":"records_ref.json"},{"row_id":"tv4_005/text/answer","fixture_id":"tv4_005","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Genetic reassortment amongst 8 genes","b":"High levels of recombination","c":"Rapid adsorption and fusion from without via a hydrophobic peptide","d":"High fidelity RNA replicase enzyme"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,5606,30049,1878,997,39420,6881,997,1098,22152,5939,1178,15161,28327,9251,55053,968,1144,803,779,2773,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,41305,29349,947,957,20467,730,533,7695,20,19,16,794,535,5001,3476,803,58190,20,19,16,781,535,40494,24612,14275,810,18357,988,2630,4492,768,5511,61129,37185,20,19,16,778,535,5001,63011,13415,33955,1146,19288,20,21],"sequence_length":87,"marker_positions":[37,50,59,76],"positions":87,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.5547366142272949,0.12310024350881577,-2.19450044631958,-2.8999176025390625],"probabilities_raw":[0.3067985475063324,0.6042742133140564,0.05952684208750725,0.029400456696748734],"probabilities":[0.3205031454563141,0.520139217376709,0.09933988004922867,0.06001778692007065],"argmax_index":1,"argmax":"b","top2_margin":0.1996360719203949,"near_tie":false,"gold":"b","official_answer":{"type":"choice","choice":"b","confidence":0.520139217376709,"probabilities":{"a":0.3205031454563141,"b":0.520139217376709,"c":0.09933988004922867,"d":0.06001778692007065}},"usage_input_tokens":87,"oracle_file":"records_ref.json"},{"row_id":"tv4_006/text/answer","fixture_id":"tv4_006","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Donepezil therapy","b":"Ferrous sulfate therapy","c":"Ginkgo biloba extract therapy","d":"Paroxetine therapy"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,30327,16964,9152,1878,997,39420,6881,997,542,730,5489,6194,10835,6394,4834,811,779,16329,1929,803,902,730,533,33344,3129,803,13467,20738,521,19236,521,810,8550,17828,523,7074,1033,2926,521,1689,1236,1178,1320,6448,59335,774,521,13467,57554,521,810,1178,1381,9730,834,27252,963,1336,875,34585,5019,523,7438,779,3650,730,527,6272,521,1689,1178,1320,2204,19941,803,54670,19877,810,3290,2122,803,8311,896,1052,44405,791,1450,988,5392,523,3365,24037,856,730,2346,40619,521,810,3186,4809,856,730,7984,524,2475,8061,904,580,523,20737,14435,4427,1295,32219,523,25650,5742,14435,4427,768,35459,15365,810,1737,34270,4531,523,2799,3498,896,1689,856,1295,5039,9368,797,4637,896,1689,1593,811,6401,523,2799,1178,60532,2542,852,2630,768,2984,523,3365,51879,9403,856,730,1632,836,51275,521,810,18941,7097,1215,769,9403,856,730,9300,21955,30147,523,20463,803,779,2773,856,779,1529,6994,4764,3801,797,3157,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,889,1174,14191,814,6738,20,19,16,794,535,12656,14641,51112,6738,20,19,16,781,535,918,1335,2086,13001,17292,8886,6738,20,19,16,778,535,2334,1974,813,976,6738,20,21],"sequence_length":243,"marker_positions":[202,212,221,233],"positions":243,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.3662583827972412,-2.0687501430511475,-1.8015267848968506,-0.1909313052892685],"probabilities_raw":[0.18581685423851013,0.09204428642988205,0.12024041265249252,0.6018983721733093],"probabilities":[0.21488778293132782,0.13009950518608093,0.1574624925851822,0.4975501596927643],"argmax_index":3,"argmax":"d","top2_margin":0.28266237676143646,"near_tie":false,"gold":"d","official_answer":{"type":"choice","choice":"d","confidence":0.4975501596927643,"probabilities":{"a":0.21488778293132782,"b":0.13009950518608093,"c":0.1574624925851822,"d":0.4975501596927643}},"usage_input_tokens":243,"oracle_file":"records_ref.json"},{"row_id":"tv4_007/text/answer","fixture_id":"tv4_007","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"mesoderm formation and occurs after neurulation.","b":"ectomesenchyme formation and occurs before neurulation.","c":"mesoderm formation and occurs before neurulation.","d":"ectomesenchyme formation and occurs after neurulation."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,780,776,5797,1878,997,39420,6881,997,61490,591,2115,856,779,1996,803,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,5287,57918,7401,810,7809,1720,11192,2115,523,20,19,16,794,535,54375,11150,20779,30667,7401,810,7809,2076,11192,2115,523,20,19,16,781,535,5287,57918,7401,810,7809,2076,11192,2115,523,20,19,16,778,535,54375,11150,20779,30667,7401,810,7809,1720,11192,2115,523,20,21],"sequence_length":92,"marker_positions":[32,46,62,76],"positions":92,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.0777621269226074,0.14282402396202087,-1.3859832286834717,-1.6446017026901245],"probabilities_raw":[0.17570854723453522,0.5955068469047546,0.12910236418247223,0.09968220442533493],"probabilities":[0.20572371780872345,0.4919832646846771,0.16506841778755188,0.13722461462020874],"argmax_index":1,"argmax":"b","top2_margin":0.2862595468759537,"near_tie":false,"gold":"b","official_answer":{"type":"choice","choice":"b","confidence":0.4919832646846771,"probabilities":{"a":0.20572371780872345,"b":0.4919832646846771,"c":0.16506841778755188,"d":0.13722461462020874}},"usage_input_tokens":92,"oracle_file":"records_ref.json"},{"row_id":"tv4_008/text/answer","fixture_id":"tv4_008","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"the strong adhesion property of water","b":"the strong cohesion of property of water","c":"the high heat capacity of water","d":"the strong cohesion property of cellulose in the xylem cell wall"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,16290,1878,997,39420,6881,997,28828,803,779,2773,6306,856,7253,875,12467,3658,779,1883,12709,797,768,12825,7322,803,1883,797,3053,2779,586,13761,803,768,3226,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,779,3636,49121,5401,803,1883,20,19,16,794,535,779,3636,44652,801,803,5401,803,1883,20,19,16,781,535,779,1569,5480,7614,803,1883,20,19,16,778,535,779,3636,44652,801,5401,803,59697,797,779,3053,2779,586,2009,7680,20,21],"sequence_length":106,"marker_positions":[52,63,76,87],"positions":106,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.249392509460449,0.9455831050872803,-7.214566707611084,1.1811388731002808],"probabilities_raw":[0.017758958041667938,0.4334886968135834,0.00012389960465952754,0.5486285090446472],"probabilities":[0.04459759220480919,0.4370166063308716,0.0012851412175223231,0.5171006321907043],"argmax_index":3,"argmax":"d","top2_margin":0.08008402585983276,"near_tie":false,"gold":"b","official_answer":{"type":"choice","choice":"d","confidence":0.5171006321907043,"probabilities":{"a":0.04459759220480919,"b":0.4370166063308716,"c":0.0012851412175223231,"d":0.5171006321907043}},"usage_input_tokens":106,"oracle_file":"records_ref.json"},{"row_id":"tv4_009/text/answer","fixture_id":"tv4_009","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"drive the French and Portuguese governments into bankruptcy.","b":"conquer large parts of Africa in the eighteenth century.","c":"establish several colonies along the Atlantic coastline of North America.","d":"destroy the Dutch commercial empire."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,1081,3129,1878,997,39420,6881,997,2443,3952,11335,811,779,2773,2203,13909,587,18588,550,48949,1014,768,3223,521,2084,17423,1665,521,896,2908,779,1509,12037,803,4367,843,7895,856,2166,3522,2759,8619,810,3676,1779,1109,1720,1101,2507,15808,3279,810,3226,1327,803,779,59362,2892,810,24408,1413,521,896,1023,803,5690,1738,3467,1052,779,22148,811,2021,5572,3789,800,797,1559,38501,810,48079,6454,906,938,3191,906,4763,946,53363,24597,803,1377,523,2439,1701,1701,1701,859,64096,2507,6905,896,779,1394,2396,3314,881,810,2166,856,896,1023,803,5690,1400,5611,810,1192,62975,1607,3015,1082,779,26695,570,843,888,1023,1188,13690,518,2059,916,779,59362,1008,810,779,24408,797,1192,803,4367,810,1312,7519,906,4763,1044,3762,18559,13909,587,18588,4483,27104,1288,1413,1166,797,1081,896,12856,811,10268,779,10669,803,1494,3374,1144,23089,811,874,797,1973,2184,1177,521,1023,1499,1014,1113,1033,2158,1052,1607,14908,3800,570,779,12762,800,803,1518,13967,1144,803,27569,810,2084,1514,1007,2344,30171,1081,521,906,968,779,2006,36159,896,1861,6283,4650,1529,15676,876,13909,511,569,587,1424,35247,53673,2417,3058,521,889,1449,33262,1798,36116,9874,779,31196,803,4367,810,779,15602,2100,583,14692,521,730,11909,527,13699,4486,2773,779,6157,803,7793,53673,2417,3058,521,5690,953,7962,3222,811,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,8323,779,5271,810,24408,12708,1419,52096,523,20,19,16,794,535,45389,2611,4863,803,6202,797,779,45034,3935,523,20,19,16,781,535,6122,2993,19087,3256,779,13874,53236,803,3859,4367,523,20,19,16,778,535,7730,779,14127,6547,22892,523,20,21],"sequence_length":339,"marker_positions":[283,297,312,328],"positions":339,"window":512,"window_r10":512,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-4.4359869956970215,-3.294753313064575,1.9535304307937622,-4.301299571990967],"probabilities_raw":[0.0016643258277326822,0.005210384260863066,0.9912209510803223,0.0019042867934331298],"probabilities":[0.009964779019355774,0.02251727692782879,0.9565467834472656,0.010971151292324066],"argmax_index":2,"argmax":"c","top2_margin":0.9340295065194368,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"c","confidence":0.9565467834472656,"probabilities":{"a":0.009964779019355774,"b":0.02251727692782879,"c":0.9565467834472656,"d":0.010971151292324066}},"usage_input_tokens":339,"oracle_file":"records_ref.json"},{"row_id":"tv4_010/text/answer","fixture_id":"tv4_010","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Whoopi Goldberg","b":"Jennifer Grey","c":"Kirstie Alley","d":"Helen Hunt"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,586,3762,965,12993,1878,997,39420,6881,997,28828,28763,6142,768,2129,598,5508,2160,34918,6557,797,779,12681,1452,31915,43084,3504,535,941,923,62401,803,19821,516,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,12747,920,582,6998,3750,20,19,16,794,535,34483,31191,20,19,16,781,535,984,1380,882,1491,2774,20,19,16,778,535,30360,20227,20,21],"sequence_length":84,"marker_positions":[50,60,67,77],"positions":84,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.631797432899475,-2.068960428237915,-1.538092017173767,-1.4344347715377808],"probabilities_raw":[0.2523794174194336,0.1630033254623413,0.2771722078323364,0.3074450194835663],"probabilities":[0.25301024317741394,0.18514660000801086,0.2705257534980774,0.2913174331188202],"argmax_index":3,"argmax":"d","top2_margin":0.020791679620742798,"near_tie":false,"gold":"d","official_answer":{"type":"choice","choice":"d","confidence":0.2913174331188202,"probabilities":{"a":0.25301024317741394,"b":0.18514660000801086,"c":0.2705257534980774,"d":0.2913174331188202}},"usage_input_tokens":84,"oracle_file":"records_ref.json"},{"row_id":"tv4_011/text/answer","fixture_id":"tv4_011","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"I only","b":"II only","c":"III only","d":"II and III only"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,2293,3810,5354,4575,1878,997,39420,6881,997,28828,803,779,2773,15450,1337,17924,13236,12825,4167,2783,20312,856,843,1072,518,3347,540,569,587,550,523,38679,875,17924,61359,3675,1209,1394,7441,1399,2025,875,12825,61359,13909,587,2667,523,31498,11236,4167,769,41913,938,946,10305,13033,13909,587,7986,523,31498,11236,4167,769,41913,3976,1052,768,3437,1782,3955,810,5321,2374,905,2460,21321,2927,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,859,1550,20,19,16,794,535,3504,1550,20,19,16,781,535,6572,1550,20,19,16,778,535,3504,810,6572,1550,20,21],"sequence_length":125,"marker_positions":[95,102,109,116],"positions":125,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-3.118961811065674,-3.313667058944702,-3.6159772872924805,-1.2328732013702393],"probabilities_raw":[0.11080395430326462,0.09120020270347595,0.06740687042474747,0.730588972568512],"probabilities":[0.15580348670482635,0.13557299971580505,0.10924122482538223,0.5993822813034058],"argmax_index":3,"argmax":"d","top2_margin":0.4435787945985794,"near_tie":false,"gold":"b","official_answer":{"type":"choice","choice":"d","confidence":0.5993822813034058,"probabilities":{"a":0.15580348670482635,"b":0.13557299971580505,"c":0.10924122482538223,"d":0.5993822813034058}},"usage_input_tokens":125,"oracle_file":"records_ref.json"},{"row_id":"tv4_012/text/answer","fixture_id":"tv4_012","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"disobey the law of demand","b":"are a joint product","c":"are a complementary good with ice cream","d":"are an inferior good"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,4167,578,2499,1504,1878,997,39420,6881,997,6626,31243,5068,768,9920,797,1450,8292,521,1689,4302,28204,1209,7302,16830,55324,810,13962,1944,592,4245,523,1890,31243,521,1944,592,4245,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,58969,2718,779,3036,803,6154,20,19,16,794,535,938,768,7411,2546,20,19,16,781,535,938,768,35281,2184,916,7302,16830,20,19,16,778,535,938,902,20451,2184,20,21],"sequence_length":97,"marker_positions":[56,67,76,88],"positions":97,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.5743370056152344,-2.8246326446533203,-0.30558764934539795,-1.4633177518844604],"probabilities_raw":[0.8245118856430054,0.010133247822523117,0.12582182884216309,0.03953303396701813],"probabilities":[0.70499187707901,0.030442405492067337,0.18406431376934052,0.08050142228603363],"argmax_index":0,"argmax":"a","top2_margin":0.5209275633096695,"near_tie":false,"gold":"d","official_answer":{"type":"choice","choice":"a","confidence":0.70499187707901,"probabilities":{"a":0.70499187707901,"b":0.030442405492067337,"c":0.18406431376934052,"d":0.08050142228603363}},"usage_input_tokens":97,"oracle_file":"records_ref.json"},{"row_id":"tv4_013/text/answer","fixture_id":"tv4_013","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Blinding prevents the server from using the CRT optimization, which is essential to the timing attack.","b":"Blinding changes the p and q primes that are used, so an adversary cannot learn the server’s true p and q values.","c":"Blinding randomizes the ciphertext being decrypted, thus obscuring the correlation between an adversary’s input and the timing differences.","d":"Blinding adds a random amount of time to the decryption due to the multiplication and division by the blinding random value r, which obscures the timing differences used in the attack."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,51584,6368,1878,997,39420,6881,997,1286,2642,900,2774,810,39279,581,1172,4066,884,3392,10598,6180,10725,521,3921,2177,1390,7075,3618,779,22515,4348,988,3822,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,2342,7075,22792,779,9397,988,1993,779,11811,561,25900,521,1144,856,5175,811,779,22515,4348,523,20,19,16,794,535,2342,7075,3807,779,784,810,9117,1002,1860,896,938,1593,521,1101,902,20639,1126,4584,2111,779,9397,1172,3347,784,810,9117,4396,20,19,16,781,535,2342,7075,7226,6229,779,781,15479,1885,1994,1908,36446,521,5430,22813,2844,779,14008,1685,902,20639,1126,1172,6136,810,779,22515,6825,20,19,16,778,535,2342,7075,16038,768,7226,3685,803,1394,811,779,1908,25704,2888,811,779,37708,810,9707,968,779,1390,7075,7226,2536,877,521,1144,20,21],"sequence_length":170,"marker_positions":[49,74,106,138],"positions":170,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.35539257526397705,0.6858941316604614,-0.02686423808336258,0.8678652048110962],"probabilities_raw":[0.11600898206233978,0.32863715291023254,0.1611274927854538,0.39422640204429626],"probabilities":[0.14782999455928802,0.31103119254112244,0.18693166971206665,0.35420718789100647],"argmax_index":3,"argmax":"d","top2_margin":0.04317599534988403,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"d","confidence":0.35420718789100647,"probabilities":{"a":0.14782999455928802,"b":0.31103119254112244,"c":0.18693166971206665,"d":0.35420718789100647}},"usage_input_tokens":170,"oracle_file":"records_ref.json"},{"row_id":"tv4_014/text/answer","fixture_id":"tv4_014","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"You buy a 1957 Chevy from a friend.","b":"The Department of Transportation repaves a road.","c":"Your friends make a music CD that doesn't sell any copies.","d":"A college buys computers."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,8496,21671,2499,1504,1878,997,39420,6881,997,28828,803,779,2773,4744,1178,1295,1862,884,27565,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,2297,7909,768,730,2081,532,5204,14785,988,768,6709,523,20,19,16,794,535,941,5455,803,34736,4228,3809,768,6671,523,20,19,16,781,535,6066,6190,1825,768,4072,6073,896,5535,1901,12374,1523,16711,523,20,19,16,778,535,835,8117,1675,1264,13505,523,20,21],"sequence_length":98,"marker_positions":[38,55,69,87],"positions":98,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.4280264377593994,-4.1390509605407715,-3.740838050842285,-2.7201414108276367],"probabilities_raw":[0.6943767070770264,0.04615429788827896,0.06873118132352829,0.190737783908844],"probabilities":[0.5769809484481812,0.08319791406393051,0.11057352274656296,0.22924762964248657],"argmax_index":0,"argmax":"a","top2_margin":0.3477333188056946,"near_tie":false,"gold":"a","official_answer":{"type":"choice","choice":"a","confidence":0.5769809484481812,"probabilities":{"a":0.5769809484481812,"b":0.08319791406393051,"c":0.11057352274656296,"d":0.22924762964248657}},"usage_input_tokens":98,"oracle_file":"records_ref.json"},{"row_id":"tv4_015/text/answer","fixture_id":"tv4_015","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"prevail, unless the police made a valid arrest.","b":"prevail, if he consented to the confinement because he was intoxicated.","c":"not prevail, because police cannot be held liable for false imprisonment.","d":"not prevail, because the student suffered no harm."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,30327,16964,3036,1878,997,39420,6881,997,542,8117,5006,953,12260,2681,988,768,6999,523,1405,1320,1381,11249,8693,963,779,6999,810,953,1943,57778,1048,523,15504,8167,12900,13525,779,5006,810,38995,1889,523,3145,950,17436,811,5642,1523,803,1149,4639,521,779,8167,3151,779,5006,797,779,1892,12799,803,768,8167,1681,523,941,8167,3191,779,5006,797,779,1892,12799,803,779,8167,1681,875,8174,1235,8831,521,810,1757,5789,1889,523,2182,779,5006,60153,768,4673,2723,779,8167,875,4831,46383,521,950,1188,1529,3849,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,64155,521,10967,779,8167,2047,768,6612,18626,523,20,19,16,794,535,64155,521,1288,950,1348,12034,811,779,53600,1929,950,953,57778,1048,523,20,19,16,781,535,1014,64155,521,1929,8167,4584,874,4908,50359,875,4831,46383,523,20,19,16,778,535,1014,64155,521,1929,779,5006,14759,1295,5808,523,20,21],"sequence_length":191,"marker_positions":[123,138,158,176],"positions":191,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.18301290273666382,-1.1472127437591553,0.13043098151683807,-0.8381863832473755],"probabilities_raw":[0.3885974586009979,0.10275200009346008,0.36869218945503235,0.13995833694934845],"probabilities":[0.3531140089035034,0.13653221726417542,0.3400966227054596,0.17025713622570038],"argmax_index":0,"argmax":"a","top2_margin":0.013017386198043823,"near_tie":true,"gold":"a","official_answer":{"type":"choice","choice":"a","confidence":0.3531140089035034,"probabilities":{"a":0.3531140089035034,"b":0.13653221726417542,"c":0.3400966227054596,"d":0.17025713622570038}},"usage_input_tokens":191,"oracle_file":"records_ref.json"},{"row_id":"tv4_016/text/answer","fixture_id":"tv4_016","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Punishment protects security.","b":"Punishment is a cost-effective method by which to prevent or reduce crime.","c":"Punishment protects personal safety.","d":"Punishment should fit the crime."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,54717,61895,1088,1878,997,39420,6881,997,28828,32630,4174,856,779,1529,45796,916,768,7243,2193,893,44352,803,22355,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,27619,14289,27952,6368,523,20,19,16,794,535,27619,14289,856,768,2780,40380,2511,968,1144,811,3618,933,4859,12383,523,20,19,16,781,535,27619,14289,27952,4297,6155,523,20,19,16,778,535,27619,14289,1753,6812,779,12383,523,20,21],"sequence_length":93,"marker_positions":[40,50,70,81],"positions":93,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-3.956939458847046,-2.1060941219329834,-4.5931596755981445,-2.959867238998413],"probabilities_raw":[0.09429672360420227,0.6002172827720642,0.04991023987531662,0.25557568669319153],"probabilities":[0.13468477129936218,0.5052562355995178,0.08549527823925018,0.274563729763031],"argmax_index":1,"argmax":"b","top2_margin":0.23069250583648682,"near_tie":false,"gold":"d","official_answer":{"type":"choice","choice":"b","confidence":0.5052562355995178,"probabilities":{"a":0.13468477129936218,"b":0.5052562355995178,"c":0.08549527823925018,"d":0.274563729763031}},"usage_input_tokens":93,"oracle_file":"records_ref.json"},{"row_id":"tv4_017/text/answer","fixture_id":"tv4_017","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Follow a different diet to the rest of the family","b":"Eat the same and take more exercise","c":"Reduce their total energy intake, increase their activity level and reduce their screen time","d":"Eat more fruit and vegetables"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,43046,1878,997,39420,6881,997,20581,1316,938,29286,1753,23606,587,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,29564,768,1847,5337,811,779,2280,803,779,2832,20,19,16,794,535,38084,779,2043,810,2337,1209,6886,20,19,16,781,535,49782,1149,3426,2915,13105,521,3761,1149,4099,2119,810,4859,1149,8660,1394,20,19,16,778,535,38084,1209,8123,810,11346,20,21],"sequence_length":87,"marker_positions":[30,45,57,77],"positions":87,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.8940390348434448,-2.765936851501465,6.657193660736084,-0.15114456415176392],"probabilities_raw":[0.00019304058514535427,8.072132186498493e-05,0.9986231327056885,0.0011030061868950725],"probabilities":[0.002199452603235841,0.0011798404157161713,0.9889820218086243,0.007638671435415745],"argmax_index":2,"argmax":"c","top2_margin":0.9813433503732085,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"c","confidence":0.9889820218086243,"probabilities":{"a":0.002199452603235841,"b":0.0011798404157161713,"c":0.9889820218086243,"d":0.007638671435415745}},"usage_input_tokens":87,"oracle_file":"records_ref.json"},{"row_id":"tv4_018/text/answer","fixture_id":"tv4_018","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"James-Lange theory of emotion","b":"Counterconditioning","c":"Catharsis hypothesis","d":"Transference"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,18849,1878,997,39420,6881,997,28828,803,779,2773,11335,811,779,7691,803,3785,902,10077,6872,811,4859,19553,54924,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,5993,5134,1668,5654,803,17916,20,19,16,794,535,47836,25556,800,20,19,16,781,535,7996,1580,783,16454,20,19,16,778,535,4885,4377,20,21],"sequence_length":77,"marker_positions":[42,53,61,70],"positions":77,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.7562735080718994,-1.111168622970581,1.7879376411437988,-3.8880667686462402],"probabilities_raw":[0.0690699890255928,0.04843512922525406,0.8794806599617004,0.003014178015291691],"probabilities":[0.12439634650945663,0.09654004871845245,0.7657828330993652,0.013280742801725864],"argmax_index":2,"argmax":"c","top2_margin":0.6413864865899086,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"c","confidence":0.7657828330993652,"probabilities":{"a":0.12439634650945663,"b":0.09654004871845245,"c":0.7657828330993652,"d":0.013280742801725864}},"usage_input_tokens":77,"oracle_file":"records_ref.json"},{"row_id":"tv4_019/text/answer","fixture_id":"tv4_019","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Wrong, Wrong","b":"Wrong, Not wrong","c":"Not wrong, Wrong","d":"Not wrong, Not wrong"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,586,8988,21469,1878,997,39420,6881,997,3111,1144,803,1518,1547,21469,2177,779,2120,3338,843,18275,5582,859,524,1477,524,3462,518,1112,3503,8870,61611,7871,521,4262,811,17027,11241,8079,797,779,2641,906,803,730,1718,525,540,3108,31933,730,526,2171,859,953,19049,811,8498,896,1727,3801,31396,1320,1014,17502,1626,1210,7927,523,3108,31933,730,527,2171,859,2588,730,20397,810,4468,896,1727,9652,953,14209,37168,963,1450,2681,2927,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,923,35736,521,923,35736,20,19,16,794,535,923,35736,521,4524,7871,20,19,16,781,535,4524,7871,521,923,35736,20,19,16,778,535,4524,7871,521,4524,7871,20,21],"sequence_length":145,"marker_positions":[105,115,125,135],"positions":145,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.6472692489624023,-2.074370861053467,-1.1838890314102173,-1.2575846910476685],"probabilities_raw":[0.4223143458366394,0.10135693848133087,0.24693623185157776,0.22939249873161316],"probabilities":[0.3718819320201874,0.1341748833656311,0.2534708082675934,0.24047236144542694],"argmax_index":0,"argmax":"a","top2_margin":0.118411123752594,"near_tie":false,"gold":"a","official_answer":{"type":"choice","choice":"a","confidence":0.3718819320201874,"probabilities":{"a":0.3718819320201874,"b":0.1341748833656311,"c":0.2534708082675934,"d":0.24047236144542694}},"usage_input_tokens":145,"oracle_file":"records_ref.json"},{"row_id":"tv4_020/text/answer","fixture_id":"tv4_020","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"stage of career development.","b":"personality.","c":"skills and willingness to assume responsibility.","d":"beliefs about the meaning and value of work."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,30327,16964,18849,1878,997,39420,6881,997,18588,9467,811,904,13011,810,2342,20052,1008,1090,3237,1430,11167,2801,521,43175,18588,870,3071,4171,36020,18588,18588,856,768,1947,803,779,19485,1172,535,18588,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,5937,803,6450,2917,523,20,19,16,794,535,16798,523,20,19,16,781,535,5051,810,41519,811,11282,10912,523,20,19,16,778,535,15209,1337,779,5564,810,2536,803,1336,523,20,21],"sequence_length":100,"marker_positions":[57,67,74,86],"positions":100,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.6914830803871155,0.4765990376472473,1.2798019647598267,-2.3174314498901367],"probabilities_raw":[0.08626310527324677,0.27740657329559326,0.6193602681159973,0.016970114782452583],"probabilities":[0.12978744506835938,0.2989580035209656,0.5306280255317688,0.040626559406518936],"argmax_index":2,"argmax":"c","top2_margin":0.23167002201080322,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"c","confidence":0.5306280255317688,"probabilities":{"a":0.12978744506835938,"b":0.2989580035209656,"c":0.5306280255317688,"d":0.040626559406518936}},"usage_input_tokens":100,"oracle_file":"records_ref.json"},{"row_id":"tv4_021/text/answer","fixture_id":"tv4_021","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"prolapsed stomas.","b":"high output stomas.","c":"herniation around the stoma.","d":"retraction of the stoma."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,50988,4331,1878,997,39420,6881,997,542,16426,2258,1011,874,1593,875,17912,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,14201,23551,878,5077,523,20,19,16,794,535,1569,6660,878,5077,523,20,19,16,781,535,51192,4149,2363,779,878,4315,523,20,19,16,778,535,1485,11478,803,779,878,4315,523,20,21],"sequence_length":75,"marker_positions":[31,41,51,63],"positions":75,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.173835039138794,-0.4603675603866577,-1.0826773643493652,-0.9287775754928589],"probabilities_raw":[0.18469972908496857,0.37698256969451904,0.20232780277729034,0.23598989844322205],"probabilities":[0.20311520993709564,0.33812984824180603,0.21678167581558228,0.24197322130203247],"argmax_index":1,"argmax":"b","top2_margin":0.09615662693977356,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"b","confidence":0.33812984824180603,"probabilities":{"a":0.20311520993709564,"b":0.33812984824180603,"c":0.21678167581558228,"d":0.24197322130203247}},"usage_input_tokens":75,"oracle_file":"records_ref.json"},{"row_id":"tv4_022/text/answer","fixture_id":"tv4_022","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"90","b":"120","c":"60","d":"320"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,17263,1878,997,39420,6881,997,542,14060,2312,9494,779,2773,2203,1337,2204,24190,803,62523,523,32989,856,1058,2198,13264,514,513,1209,11932,1399,37436,810,1178,1058,1698,12185,514,513,2442,7460,1399,3735,867,523,3735,867,856,1058,1698,12185,514,513,2442,11932,1399,32989,810,1178,1058,2625,12185,3088,600,526,2945,528,602,13264,514,513,1209,7460,1399,37436,523,37436,7025,1763,569,513,526,523,1056,513,1094,6232,803,7460,523,3747,856,779,2080,803,37843,1094,6232,803,7460,803,3735,867,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,730,2475,20,19,16,794,535,730,6505,20,19,16,781,535,730,2198,20,19,16,778,535,730,16511,20,21],"sequence_length":148,"marker_positions":[120,127,134,141],"positions":148,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.871951937675476,-2.1734390258789062,-2.9600579738616943,-2.754812240600586],"probabilities_raw":[0.4015790820121765,0.2970550060272217,0.13527347147464752,0.1660923957824707],"probabilities":[0.35737961530685425,0.28813666105270386,0.16427196562290192,0.19021180272102356],"argmax_index":0,"argmax":"a","top2_margin":0.06924295425415039,"near_tie":false,"gold":"a","official_answer":{"type":"choice","choice":"a","confidence":0.35737961530685425,"probabilities":{"a":0.35737961530685425,"b":0.28813666105270386,"c":0.16427196562290192,"d":0.19021180272102356}},"usage_input_tokens":148,"oracle_file":"records_ref.json"},{"row_id":"tv4_023/text/answer","fixture_id":"tv4_023","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"some optimal state of pollution.","b":"an overall maximally clean environment.","c":"some maximal state of pollution.","d":"the preservation of all living species."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,586,8988,38419,1878,997,39420,6881,997,1030,59507,886,1090,3160,521,779,1509,810,43997,3801,7069,768,5516,803,1494,5962,3775,856,768,4484,12487,896,1494,13010,856,1014,10349,3077,933,1883,1203,4286,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,1429,13985,2425,803,11244,523,20,19,16,794,535,902,6807,15372,1587,5158,3054,523,20,19,16,781,535,1429,29501,2425,803,11244,523,20,19,16,778,535,779,25369,803,1113,4624,3311,523,20,21],"sequence_length":104,"marker_positions":[58,69,81,92],"positions":104,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.9687066078186035,0.7563834190368652,-3.7341668605804443,-6.6945295333862305],"probabilities_raw":[0.023275204002857208,0.9653382897377014,0.01082576159387827,0.0005607756902463734],"probabilities":[0.06266120076179504,0.8966939449310303,0.03626834228634834,0.004376488737761974],"argmax_index":1,"argmax":"b","top2_margin":0.8340327441692352,"near_tie":false,"gold":"a","official_answer":{"type":"choice","choice":"b","confidence":0.8966939449310303,"probabilities":{"a":0.06266120076179504,"b":0.8966939449310303,"c":0.03626834228634834,"d":0.004376488737761974}},"usage_input_tokens":104,"oracle_file":"records_ref.json"},{"row_id":"tv4_024/text/answer","fixture_id":"tv4_024","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"To enable us to understand the workings of political subjecthoods left out of traditional discourses on security.","b":"To examine the base of power and its construction, in greater depth than traditional analysis.","c":"To challenge the idea that smart weapons and technology are the active subjects determining security today.","d":"All of these."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,53983,3609,1878,997,39420,6881,997,3493,856,779,2536,803,768,2375,4734,1785,1786,893,28714,797,6368,3609,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,2989,11787,1081,811,2846,779,1336,1327,803,4665,3910,5720,592,3191,1230,803,6257,2006,36159,884,6368,523,20,19,16,794,535,2989,14057,779,4077,803,2410,810,1352,6079,521,797,5574,11336,1399,6257,3606,523,20,19,16,781,535,2989,8759,779,4588,896,10338,14773,810,4109,938,779,5592,9715,18572,6368,4008,523,20,19,16,778,535,2595,803,1518,523,20,21],"sequence_length":119,"marker_positions":[39,66,88,110],"positions":119,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.9829537868499756,1.969934344291687,-0.5511482954025269,3.171776294708252],"probabilities_raw":[0.18692950904369354,0.18451154232025146,0.014829661697149277,0.6137293577194214],"probabilities":[0.22260868549346924,0.22054797410964966,0.03642337769269943,0.5204199552536011],"argmax_index":3,"argmax":"d","top2_margin":0.29781126976013184,"near_tie":false,"gold":"d","official_answer":{"type":"choice","choice":"d","confidence":0.5204199552536011,"probabilities":{"a":0.22260868549346924,"b":0.22054797410964966,"c":0.03642337769269943,"d":0.5204199552536011}},"usage_input_tokens":119,"oracle_file":"records_ref.json"},{"row_id":"tv4_025/text/answer","fixture_id":"tv4_025","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"The friend's heirs.","b":"The cousin.","c":"The woman.","d":"Divided equally between the friend's heirs and the woman."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,30327,16964,3036,1878,997,39420,6881,997,542,45301,20271,968,768,6394,797,730,1086,526,60843,768,13733,803,2496,875,768,15879,803,1235,23589,521,43152,803,1144,953,28984,521,43175,1095,1727,6709,875,2105,521,1203,1288,58154,856,1760,4997,884,779,13733,803,2496,521,1757,811,1727,30221,810,1210,950,14352,521,810,1288,875,1523,4092,779,2862,2712,2448,60843,811,1727,30221,856,1014,6612,521,1757,859,21225,779,2084,811,840,522,9381,779,13733,803,2496,810,2337,1892,1727,5401,523,43175,1030,730,1086,529,521,779,6709,5942,14643,931,2076,779,18302,950,1320,21440,1738,874,33533,523,12747,856,18887,811,779,40951,803,779,13838,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,941,6709,1090,950,14352,523,20,19,16,794,535,941,30221,523,20,19,16,781,535,941,6394,523,20,19,16,778,535,889,47348,15395,1685,779,6709,1090,950,14352,810,779,6394,523,20,21],"sequence_length":192,"marker_positions":[147,158,166,174],"positions":192,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.358635812997818,-0.6387908458709717,0.28748929500579834,0.5898016095161438],"probabilities_raw":[0.16011600196361542,0.12099432200193405,0.3055225908756256,0.4133670926094055],"probabilities":[0.18607696890830994,0.15232786536216736,0.2952173948287964,0.3663778007030487],"argmax_index":3,"argmax":"d","top2_margin":0.07116040587425232,"near_tie":false,"gold":"a","official_answer":{"type":"choice","choice":"d","confidence":0.3663778007030487,"probabilities":{"a":0.18607696890830994,"b":0.15232786536216736,"c":0.2952173948287964,"d":0.3663778007030487}},"usage_input_tokens":192,"oracle_file":"records_ref.json"},{"row_id":"tv4_026/text/answer","fixture_id":"tv4_026","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Both are concerned with the 'law in action.'","b":"Both are anti-formalist and sceptical","c":"Both adopt a liberal ideology.","d":"Both attempt to demystify the law."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,54717,61895,1088,1878,997,39420,6881,997,50902,32600,10723,843,7377,560,518,856,2492,4965,811,843,777,916,518,2473,59491,523,20463,803,779,2773,15450,856,55495,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,9758,938,11483,916,779,1452,26196,797,4272,12158,20,19,16,794,535,9758,938,5898,21127,786,893,810,775,1570,1021,20,19,16,781,535,9758,6378,768,19467,37344,523,20,19,16,778,535,9758,5308,811,1640,6033,2563,779,3036,523,20,21],"sequence_length":105,"marker_positions":[50,65,80,91],"positions":105,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-8.456859588623047,-7.910188674926758,-4.601129531860352,-2.9162330627441406],"probabilities_raw":[0.0032805362716317177,0.005667101591825485,0.15504834055900574,0.8360040187835693],"probabilities":[0.014177599921822548,0.0209506768733263,0.2227291464805603,0.7421425580978394],"argmax_index":3,"argmax":"d","top2_margin":0.519413411617279,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"d","confidence":0.7421425580978394,"probabilities":{"a":0.014177599921822548,"b":0.0209506768733263,"c":0.2227291464805603,"d":0.7421425580978394}},"usage_input_tokens":105,"oracle_file":"records_ref.json"},{"row_id":"tv4_027/text/answer","fixture_id":"tv4_027","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"result in lower domestic prices.","b":"sometimes raise and sometimes lower the amount of the product sold domestically.","c":"lower the amount of the product sold domestically.","d":"raise the amount of the product sold domestically."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,8496,21671,2499,1504,1878,997,39420,6881,997,561,788,20167,810,17683,802,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,1725,797,3730,10798,11128,523,20,19,16,794,535,5650,9920,810,5650,3730,779,3685,803,779,2546,4997,52797,1939,523,20,19,16,781,535,3730,779,3685,803,779,2546,4997,52797,1939,523,20,19,16,778,535,9920,779,3685,803,779,2546,4997,52797,1939,523,20,21],"sequence_length":94,"marker_positions":[34,45,64,79],"positions":94,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.4790819883346558,3.206943988800049,-0.3642081022262573,-1.1758590936660767],"probabilities_raw":[0.008785426616668701,0.9525288343429565,0.02678837813436985,0.011897370219230652],"probabilities":[0.03040599264204502,0.8644078373908997,0.06742636859416962,0.03775973990559578],"argmax_index":1,"argmax":"b","top2_margin":0.79698146879673,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"b","confidence":0.8644078373908997,"probabilities":{"a":0.03040599264204502,"b":0.8644078373908997,"c":0.06742636859416962,"d":0.03775973990559578}},"usage_input_tokens":94,"oracle_file":"records_ref.json"},{"row_id":"tv4_028/text/answer","fixture_id":"tv4_028","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"admissible as a declaration against interest.","b":"admissible as former testimony.","c":"inadmissible, because of collateral estoppel.","d":"inadmissible, because of double jeopardy."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,30327,16964,3036,1878,997,39420,6881,997,542,64188,856,779,9829,27355,803,768,5792,12800,968,902,12235,4129,2083,2844,779,2105,803,1210,7950,521,2166,40918,523,941,5792,6034,768,32787,7376,896,6527,32886,587,960,856,4025,1108,797,779,3308,896,3295,803,779,2083,3107,43175,27012,5828,521,810,19848,803,1113,1312,5817,521,988,40314,8462,1862,791,22333,1626,9774,17844,810,45127,3255,523,43175,1098,64188,1090,7950,1827,1450,3295,797,779,4598,1047,18660,803,1450,4799,523,941,3295,12393,988,30080,5457,968,779,21384,803,768,6527,1985,37249,1859,10231,42605,523,941,64188,953,20890,810,23548,12608,875,779,12494,803,1210,7950,523,4162,768,42212,8727,521,779,64188,953,5930,4477,803,779,7786,523,4162,779,12235,4129,17436,811,4025,779,64188,1090,12235,4673,521,779,64188,60509,1033,5614,4272,523,941,38203,8753,14340,779,5792,521,13621,774,779,11445,10674,779,3295,803,779,2083,3107,521,1563,871,896,779,3295,953,2638,779,5792,23984,875,6527,32886,587,960,16655,521,810,16365,20524,30525,523,2805,8727,521,779,12235,4129,1090,26917,48207,811,14993,41240,592,988,779,64188,1090,12494,8727,523,941,64188,1090,26917,7007,811,779,11258,803,1559,4589,523,941,8727,16277,1753,7330,779,899,1016,3501,4589,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,6416,41748,906,768,31137,2723,2862,523,20,19,16,794,535,6416,41748,906,5211,33763,523,20,19,16,781,535,797,51039,41748,521,1929,803,3749,15850,1170,31355,523,20,19,16,778,535,797,51039,41748,521,1929,803,6527,2250,36115,598,523,20,21],"sequence_length":319,"marker_positions":[263,276,287,303],"positions":319,"window":512,"window_r10":512,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.302972435951233,1.3318623304367065,-1.1114861965179443,-1.2292307615280151],"probabilities_raw":[0.058043066412210464,0.80917888879776,0.07029295712709427,0.06248502433300018],"probabilities":[0.10237134993076324,0.6723430156707764,0.11737720668315887,0.1079084724187851],"argmax_index":1,"argmax":"b","top2_margin":0.5549658089876175,"near_tie":false,"gold":"b","official_answer":{"type":"choice","choice":"b","confidence":0.6723430156707764,"probabilities":{"a":0.10237134993076324,"b":0.6723430156707764,"c":0.11737720668315887,"d":0.1079084724187851}},"usage_input_tokens":319,"oracle_file":"records_ref.json"},{"row_id":"tv4_029/text/answer","fixture_id":"tv4_029","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"I only","b":"II only","c":"III only","d":"I and III"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,16290,1878,997,39420,6881,997,37512,2574,586,3838,7809,797,859,5045,785,17868,4027,3504,523,49936,582,6572,523,50168,38822,592,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,859,1550,20,19,16,794,535,3504,1550,20,19,16,781,535,6572,1550,20,19,16,778,535,859,810,6572,20,21],"sequence_length":74,"marker_positions":[45,52,59,66],"positions":74,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.2669294476509094,-0.4548293352127075,-1.1632142066955566,1.690333366394043],"probabilities_raw":[0.10733412951231003,0.08894754946231842,0.04380122572183609,0.759917140007019],"probabilities":[0.155056893825531,0.1355808526277542,0.0817398950457573,0.6276223063468933],"argmax_index":3,"argmax":"d","top2_margin":0.4725654125213623,"near_tie":false,"gold":"d","official_answer":{"type":"choice","choice":"d","confidence":0.6276223063468933,"probabilities":{"a":0.155056893825531,"b":0.1355808526277542,"c":0.0817398950457573,"d":0.6276223063468933}},"usage_input_tokens":74,"oracle_file":"records_ref.json"},{"row_id":"tv4_030/text/answer","fixture_id":"tv4_030","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"a tendency to maximize happiness.","b":"a tendency to pursue justice.","c":"whatever mental action or quality gives a spectator a sentiment of approbation.","d":"whatever mental action or quality brings about the best consequences for all."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,52220,6028,598,1878,997,39420,6881,997,549,2995,21179,29375,906,17912,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,768,22028,811,33353,20845,523,20,19,16,794,535,768,22028,811,22710,11104,523,20,19,16,781,535,12738,6421,4272,933,4183,6602,768,5589,1888,768,29647,803,2396,575,852,523,20,19,16,778,535,12738,6421,4272,933,4183,14635,1337,779,2238,11364,875,1113,523,20,21],"sequence_length":91,"marker_positions":[30,41,52,73],"positions":91,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.01371368020772934,-3.478639602661133,-2.0193262100219727,0.3967164158821106],"probabilities_raw":[0.3740668296813965,0.011699061840772629,0.050341106951236725,0.5638930201530457],"probabilities":[0.375444233417511,0.03159448876976967,0.08960627764463425,0.5033549666404724],"argmax_index":3,"argmax":"d","top2_margin":0.12791073322296143,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"d","confidence":0.5033549666404724,"probabilities":{"a":0.375444233417511,"b":0.03159448876976967,"c":0.08960627764463425,"d":0.5033549666404724}},"usage_input_tokens":91,"oracle_file":"records_ref.json"},{"row_id":"tv4_031/text/answer","fixture_id":"tv4_031","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"intimacy vs. isolation","b":"identity vs. role confusion","c":"autonomy vs. shame and guilt","d":"industry vs. inferiority"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,18849,1878,997,39420,6881,997,9467,811,33620,2371,521,768,3444,8035,1090,21925,10373,14808,2363,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,26368,3384,8199,523,19628,20,19,16,794,535,9552,8199,523,3671,20694,20,19,16,781,535,36218,8199,523,30852,810,36025,20,19,16,778,535,5294,8199,523,20451,960,20,21],"sequence_length":79,"marker_positions":[38,48,58,69],"positions":79,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.1906007528305054,2.4247899055480957,-2.300215005874634,-2.701061248779297],"probabilities_raw":[0.025828873738646507,0.9599524736404419,0.008515420369803905,0.005703229922801256],"probabilities":[0.06655921787023544,0.8806865215301514,0.030127935111522675,0.022626303136348724],"argmax_index":1,"argmax":"b","top2_margin":0.8141273036599159,"near_tie":false,"gold":"a","official_answer":{"type":"choice","choice":"b","confidence":0.8806865215301514,"probabilities":{"a":0.06655921787023544,"b":0.8806865215301514,"c":0.030127935111522675,"d":0.022626303136348724}},"usage_input_tokens":79,"oracle_file":"records_ref.json"},{"row_id":"tv4_032/text/answer","fixture_id":"tv4_032","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"An absolute date is known by everyone, while a relative date ranges from 30 to 50 years.","b":"A relative date is derived from radiometric dating techniques, while an absolute date is only associated with written documents.","c":"A relative date refers to a year or a range of years, while an absolute date indicates a chronological order.","d":"An absolute date refers to a year or a range of years, while a relative date indicates a chronological order."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,3034,41336,1878,997,39420,6881,997,3493,856,779,5521,1685,902,13954,5562,810,768,7889,5562,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,1592,13954,5562,856,2538,968,7250,521,2158,768,7889,5562,19000,988,730,1664,811,730,1838,1779,523,20,19,16,794,535,835,7889,5562,856,10491,988,12255,14506,14820,6793,521,2158,902,13954,5562,856,1550,3973,916,4985,10621,523,20,19,16,781,535,835,7889,5562,11335,811,768,1423,933,768,3795,803,1779,521,2158,902,13954,5562,11136,768,17540,2802,2565,523,20,19,16,778,535,1592,13954,5562,11335,811,768,1423,933,768,3795,803,1779,521,2158,768,7889,5562,11136,768,17540,2802,2565,523,20,21],"sequence_length":145,"marker_positions":[36,62,89,117],"positions":145,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.510103464126587,-1.3151054382324219,0.091059111058712,0.8483920097351074],"probabilities_raw":[0.021492037922143936,0.07100004702806473,0.2896997630596161,0.6178081631660461],"probabilities":[0.04813937842845917,0.11303896456956863,0.30865028500556946,0.5301713943481445],"argmax_index":3,"argmax":"d","top2_margin":0.22152110934257507,"near_tie":false,"gold":"d","official_answer":{"type":"choice","choice":"d","confidence":0.5301713943481445,"probabilities":{"a":0.04813937842845917,"b":0.11303896456956863,"c":0.30865028500556946,"d":0.5301713943481445}},"usage_input_tokens":145,"oracle_file":"records_ref.json"},{"row_id":"tv4_033/text/answer","fixture_id":"tv4_033","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"mobilization for war in 1914 was greeted with great skepticism in Paris","b":"mobilization for war in 1914 was greeted with great skepticism all across Europe","c":"mobilization for war in 1914 was greeted with great enthusiasm all across Europe","d":"mobilization for war in 1914 was greeted with great enthusiasm in Paris"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,18807,780,3129,1878,997,39420,6881,997,2443,3952,11335,811,779,2773,2203,13909,587,18588,2299,1090,2743,510,1311,1090,18805,963,779,7952,9276,1090,6756,3823,511,768,27549,2448,54907,963,1123,906,950,9657,523,859,8107,779,44726,889,49994,797,1235,43053,523,7356,859,2062,779,6282,963,768,38455,523,7356,43175,1098,5707,6252,803,56312,2274,3742,1846,14231,521,4053,730,527,1607,2444,529,3787,18588,7356,1311,953,902,34760,811,768,3847,810,768,4311,5271,2217,523,7356,3408,510,7356,20385,22720,1203,48109,3085,1048,521,859,4549,1892,811,1607,3462,15600,1090,6756,570,810,21791,1419,779,6756,803,29281,52460,1088,1206,521,1494,10292,523,43175,3493,856,5242,8600,540,18588,950,5751,1123,523,43175,2299,1090,22062,521,19398,510,18588,43175,2299,1188,874,1113,2084,1757,568,3823,511,52460,1088,1206,25723,3787,569,587,559,62261,889,983,7569,2771,521,4162,730,1838,21045,521,781,523,730,1848,530,13699,6314,779,12543,521,1235,1400,10953,896,889,983,7569,2771,33871,896,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,19178,2274,875,2104,797,730,2444,529,953,2581,23073,916,2507,14801,19555,1586,797,5242,20,19,16,794,535,19178,2274,875,2104,797,730,2444,529,953,2581,23073,916,2507,14801,19555,1586,1113,3648,3200,20,19,16,781,535,19178,2274,875,2104,797,730,2444,529,953,2581,23073,916,2507,37875,1113,3648,3200,20,19,16,778,535,19178,2274,875,2104,797,730,2444,529,953,2581,23073,916,2507,37875,797,5242,20,21],"sequence_length":308,"marker_positions":[218,241,265,287],"positions":308,"window":512,"window_r10":512,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.10337462276220322,-2.55706524848938,-0.875851035118103,-0.2083251178264618],"probabilities_raw":[0.45919573307037354,0.03210579603910446,0.17247475683689117,0.3362237215042114],"probabilities":[0.408709853887558,0.06110251322388649,0.2030612677335739,0.3271264135837555],"argmax_index":0,"argmax":"a","top2_margin":0.08158344030380249,"near_tie":false,"gold":"d","official_answer":{"type":"choice","choice":"a","confidence":0.408709853887558,"probabilities":{"a":0.408709853887558,"b":0.06110251322388649,"c":0.2030612677335739,"d":0.3271264135837555}},"usage_input_tokens":308,"oracle_file":"records_ref.json"},{"row_id":"tv4_034/text/answer","fixture_id":"tv4_034","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Hunger","b":"Thirst","c":"Lack of coordination","d":"Hallucinations"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,18849,1878,997,39420,6881,997,542,2514,803,7524,797,768,5392,2033,938,811,874,48586,803,5392,875,2759,3378,523,4162,1149,2341,35210,36744,4915,521,7524,1400,6305,15603,1144,803,779,2773,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,63183,20,19,16,794,535,1097,1380,20,19,16,781,535,39393,803,24710,20,19,16,778,535,7329,1617,10021,20,21],"sequence_length":87,"marker_positions":[58,64,71,79],"positions":87,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.793691396713257,-2.0432310104370117,-1.2673485279083252,2.3498144149780273],"probabilities_raw":[0.005585509818047285,0.011829973198473454,0.02570071816444397,0.9568836688995361],"probabilities":[0.022172139957547188,0.037898775190114975,0.06596741825342178,0.8739617466926575],"argmax_index":3,"argmax":"d","top2_margin":0.8079943284392357,"near_tie":false,"gold":"d","official_answer":{"type":"choice","choice":"d","confidence":0.8739617466926575,"probabilities":{"a":0.022172139957547188,"b":0.037898775190114975,"c":0.06596741825342178,"d":0.8739617466926575}},"usage_input_tokens":87,"oracle_file":"records_ref.json"},{"row_id":"tv4_035/text/answer","fixture_id":"tv4_035","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"It would rise more quickly for negative disturbances than for positive ones of the same magnitude","b":"It would be symmetrical about zero","c":"It would rise less quickly for negative disturbances than for positive ones of the same magnitude","d":"It would be zero for all positive disturbances"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,578,2499,813,8731,1878,997,39420,6881,997,3383,1413,1166,768,43186,1862,797,4741,521,1620,1499,874,779,7128,803,779,6196,3616,13818,875,906,2801,896,34162,875,896,43186,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,1311,1499,7860,1209,6214,875,6384,43368,1399,875,4948,6531,803,779,2043,17165,20,19,16,794,535,1311,1499,874,13820,14977,1337,9617,20,19,16,781,535,1311,1499,7860,2442,6214,875,6384,43368,1399,875,4948,6531,803,779,2043,17165,20,19,16,778,535,1311,1499,874,9617,875,1113,4948,43368,20,21],"sequence_length":120,"marker_positions":[53,74,86,107],"positions":120,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.3277885913848877,-0.30665910243988037,-1.4691928625106812,-2.5176525115966797],"probabilities_raw":[0.7828211188316345,0.1526971459388733,0.04774731770157814,0.016734344884753227],"probabilities":[0.661874532699585,0.20592838525772095,0.08975527435541153,0.04244174808263779],"argmax_index":0,"argmax":"a","top2_margin":0.455946147441864,"near_tie":false,"gold":"a","official_answer":{"type":"choice","choice":"a","confidence":0.661874532699585,"probabilities":{"a":0.661874532699585,"b":0.20592838525772095,"c":0.08975527435541153,"d":0.04244174808263779}},"usage_input_tokens":120,"oracle_file":"records_ref.json"},{"row_id":"tv4_036/text/answer","fixture_id":"tv4_036","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"$4,000","b":"$8,000","c":"$8,400","d":"16800"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,30327,16964,21167,1878,997,39420,6881,997,553,26717,13807,730,1086,525,19190,803,24008,1969,37891,523,2624,9051,896,1166,15399,797,1423,730,526,963,1058,1155,523,1838,1094,5611,523,1030,1423,730,529,10765,772,5068,768,730,530,514,2222,522,40864,1108,7533,959,803,24008,1969,2624,9051,523,1030,1423,730,530,779,9051,13305,730,527,20432,522,526,523,1030,779,2835,1423,10765,772,4997,730,7765,19190,523,3747,856,10765,772,1090,6524,797,779,730,7765,19190,803,9051,4997,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,1058,529,521,1873,20,19,16,794,535,1058,533,521,1873,20,19,16,781,535,1058,533,521,6019,20,19,16,778,535,730,10173,1056,20,21],"sequence_length":150,"marker_positions":[115,124,133,142],"positions":150,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.7359061241149902,-1.6822254657745361,-1.4564356803894043,-5.317482948303223],"probabilities_raw":[0.13265100121498108,0.38046807050704956,0.47684529423713684,0.010035661049187183],"probabilities":[0.17315810918807983,0.36756089329719543,0.4318934977054596,0.027387524023652077],"argmax_index":2,"argmax":"c","top2_margin":0.06433260440826416,"near_tie":false,"gold":"a","official_answer":{"type":"choice","choice":"c","confidence":0.4318934977054596,"probabilities":{"a":0.17315810918807983,"b":0.36756089329719543,"c":0.4318934977054596,"d":0.027387524023652077}},"usage_input_tokens":150,"oracle_file":"records_ref.json"},{"row_id":"tv4_037/text/answer","fixture_id":"tv4_037","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Antibodies","b":"T cells","c":"Memory T cells","d":"Clonal expansion of pre-existing antibodies and/or T cells"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,5606,30049,1878,997,39420,6881,997,28828,803,779,2773,856,779,1529,3429,15488,803,779,29749,8046,1678,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,44437,6754,20,19,16,794,535,837,2998,20,19,16,781,535,28718,837,2998,20,19,16,778,535,1976,9529,11755,803,1181,55092,19453,810,8305,837,2998,20,21],"sequence_length":76,"marker_positions":[38,45,52,60],"positions":76,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.564216136932373,-0.7476570010185242,-0.27579110860824585,-0.22039031982421875],"probabilities_raw":[0.21848095953464508,0.18186379969120026,0.2915244400501251,0.3081308901309967],"probabilities":[0.22808265686035156,0.20007041096687317,0.28026649355888367,0.2915804088115692],"argmax_index":3,"argmax":"d","top2_margin":0.011313915252685547,"near_tie":true,"gold":"d","official_answer":{"type":"choice","choice":"d","confidence":0.2915804088115692,"probabilities":{"a":0.22808265686035156,"b":0.20007041096687317,"c":0.28026649355888367,"d":0.2915804088115692}},"usage_input_tokens":76,"oracle_file":"records_ref.json"},{"row_id":"tv4_038/text/answer","fixture_id":"tv4_038","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"ADF is a single equation approach to unit root testing while PP is a systems approach","b":"PP tests reverse the DF null and alternative hypotheses so that there is stationarity under the null hypothesis of the PP test","c":"The PP test incorporates an automatic correction for autocorrelated residuals in the test regression","d":"PP tests have good power in small samples whereas DF tests do not."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,578,2499,813,8731,1878,997,39420,6881,997,3493,856,779,2120,5521,1685,779,16865,2718,59024,843,7094,518,810,34258,3995,770,1711,843,15612,518,10524,811,6232,7152,6976,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,835,7094,856,768,3437,10805,4355,811,6232,7152,6976,2158,27967,856,768,3769,4355,20,19,16,794,535,27967,6630,15637,779,40129,3996,810,8530,43237,1101,896,1413,856,6195,25923,1490,779,3996,16454,803,779,27967,1927,20,19,16,781,535,941,27967,1927,42773,902,22156,23819,875,62228,777,6566,11201,13294,797,779,1927,20436,20,19,16,778,535,27967,6630,1052,2184,2410,797,2211,7515,11441,40129,6630,1112,1014,523,20,21],"sequence_length":143,"marker_positions":[52,74,102,124],"positions":143,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.7698924541473389,0.4720029830932617,-0.3455861806869507,-0.4429304003715515],"probabilities_raw":[0.13554823398590088,0.46929097175598145,0.20718948543071747,0.1879713535308838],"probabilities":[0.16541790962219238,0.401660680770874,0.22398363053798676,0.20893773436546326],"argmax_index":1,"argmax":"b","top2_margin":0.17767705023288727,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"b","confidence":0.401660680770874,"probabilities":{"a":0.16541790962219238,"b":0.401660680770874,"c":0.22398363053798676,"d":0.20893773436546326}},"usage_input_tokens":143,"oracle_file":"records_ref.json"},{"row_id":"tv4_039/text/answer","fixture_id":"tv4_039","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"marginal utility is zero","b":"marginal benefit equals marginal cost","c":"consumer surplus is zero","d":"distributive efficiency is achieved"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,4167,578,2499,1504,1878,997,39420,6881,997,1098,3426,18768,988,775,1008,1666,856,15372,1884,1403,1186,938,15399,3032,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,22482,18768,856,9617,20,19,16,794,535,22482,7691,30322,22482,2780,20,19,16,781,535,14060,41938,856,9617,20,19,16,778,535,16791,975,9806,856,10836,20,21],"sequence_length":81,"marker_positions":[43,52,62,71],"positions":81,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.7035473585128784,-1.886009931564331,-2.6342694759368896,-5.541640281677246],"probabilities_raw":[0.9602558612823486,0.026513198390603065,0.01254576351493597,0.0006852386868558824],"probabilities":[0.8867239356040955,0.06826366484165192,0.039999522268772125,0.005012943875044584],"argmax_index":0,"argmax":"a","top2_margin":0.8184602707624435,"near_tie":false,"gold":"a","official_answer":{"type":"choice","choice":"a","confidence":0.8867239356040955,"probabilities":{"a":0.8867239356040955,"b":0.06826366484165192,"c":0.039999522268772125,"d":0.005012943875044584}},"usage_input_tokens":81,"oracle_file":"records_ref.json"},{"row_id":"tv4_040/text/answer","fixture_id":"tv4_040","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"agree to see Mrs. Wang’s daughter in therapy since you’ve had some experience providing crisis intervention experiences.","b":"inform Mrs. Wang about your lack of experience and let her decide if she wants you to provide therapy to her daughter.","c":"inform Mrs. Wang that you cannot see her daughter because of your lack of experience.","d":"see Mrs. Wang’s daughter in therapy only until the crisis has ended or until you locate alternative services."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,30327,16964,18849,1878,997,39420,6881,997,4083,6984,768,1868,988,15202,523,13202,1316,856,1943,30133,1929,1450,730,1536,6194,10835,9545,31520,779,20481,803,1450,2238,6709,2204,3378,6027,810,856,1943,1709,853,57994,523,941,10761,27973,2094,51707,521,4746,2094,6791,521,810,1011,2094,5910,40698,523,2297,1052,6235,3758,3822,916,21570,810,797,7376,10373,11125,4135,523,2876,521,1413,856,1295,1235,3581,797,779,3681,1316,856,1209,9468,1399,1010,938,523,1783,902,19316,42724,521,1010,1188,17912,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,11667,811,2130,15202,523,13202,1172,9545,797,6738,2908,1010,6695,1320,1429,3758,7376,10373,11125,8938,523,20,19,16,794,535,1836,15202,523,13202,1337,1180,5388,803,3758,810,2944,1450,11245,1288,1689,11467,1010,811,3176,6738,811,1450,9545,523,20,19,16,781,535,1836,15202,523,13202,896,1010,4584,2130,1450,9545,1929,803,1180,5388,803,3758,523,20,19,16,778,535,2130,15202,523,13202,1172,9545,797,6738,1550,3032,779,10373,1178,11446,933,3032,1010,28062,8530,4135,523,20,21],"sequence_length":219,"marker_positions":[116,142,171,193],"positions":219,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.3792548179626465,0.8844364881515503,-0.7305554747581482,-2.2510766983032227],"probabilities_raw":[0.029866043478250504,0.7808738350868225,0.1553097814321518,0.03395038843154907],"probabilities":[0.06395912170410156,0.6582744121551514,0.2076745629310608,0.0700918585062027],"argmax_index":1,"argmax":"b","top2_margin":0.4505998492240906,"near_tie":false,"gold":"d","official_answer":{"type":"choice","choice":"b","confidence":0.6582744121551514,"probabilities":{"a":0.06395912170410156,"b":0.6582744121551514,"c":0.2076745629310608,"d":0.0700918585062027}},"usage_input_tokens":219,"oracle_file":"records_ref.json"},{"row_id":"tv4_041/text/answer","fixture_id":"tv4_041","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"the jury, after a preliminary determination by the judge.","b":"the jury, without a preliminary determination by the judge.","c":"the judge, with instruction to the jury that it may consider the dying declaration if it determines that the victim knew he was dying.","d":"the judge, without assistance from the jury."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,30327,16964,3036,1878,997,39420,6881,997,542,16779,953,11915,884,768,11513,12991,13668,811,1825,768,4153,7909,523,941,4153,55154,953,3469,5019,521,1101,779,16779,6806,811,7498,5329,797,779,34517,6707,7921,1929,950,4602,768,4605,9837,523,10238,53927,768,1681,33138,779,11513,12991,810,768,10231,7989,1230,523,941,16779,1928,3700,791,1391,810,10357,811,779,54322,523,2904,50322,1720,1994,10231,810,2076,950,5942,521,779,16779,940,21494,521,43175,1098,47715,2172,936,510,18588,941,47715,953,1459,13735,810,856,2166,884,8727,875,779,16779,1090,12494,523,18276,884,1518,11445,6955,521,779,6416,1271,3912,803,779,16779,1090,8294,1753,874,7877,968,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,779,28292,521,1720,768,27519,22430,968,779,16277,523,20,19,16,794,535,779,28292,521,2630,768,27519,22430,968,779,16277,523,20,19,16,781,535,779,16277,521,916,11497,811,779,28292,896,936,1400,2260,779,21712,31137,1288,936,25195,896,779,16779,8291,950,953,21712,523,20,19,16,778,535,779,16277,521,2630,12224,988,779,28292,523,20,21],"sequence_length":226,"marker_positions":[149,165,181,212],"positions":226,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.2598228454589844,-0.670441746711731,0.7866662740707397,-2.2362213134765625],"probabilities_raw":[0.21507954597473145,0.14264927804470062,0.6124681830406189,0.02980300597846508],"probabilities":[0.24382537603378296,0.18184082210063934,0.5149134993553162,0.059420328587293625],"argmax_index":2,"argmax":"c","top2_margin":0.2710881233215332,"near_tie":false,"gold":"d","official_answer":{"type":"choice","choice":"c","confidence":0.5149134993553162,"probabilities":{"a":0.24382537603378296,"b":0.18184082210063934,"c":0.5149134993553162,"d":0.059420328587293625}},"usage_input_tokens":226,"oracle_file":"records_ref.json"},{"row_id":"tv4_042/text/answer","fixture_id":"tv4_042","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Fallacy of accident","b":"Laudatory personality","c":"Begging the question","d":"Circular reasoning"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,3600,1021,3981,36158,1878,997,39420,6881,997,1098,3981,3384,803,36025,968,10039,856,768,3680,2432,803,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,12499,3384,803,10807,20,19,16,794,535,2097,949,3294,16798,20,19,16,781,535,10439,5222,779,3952,20,19,16,778,535,13104,1437,23223,20,21],"sequence_length":73,"marker_positions":[38,47,56,65],"positions":73,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.2054716348648071,-2.7291550636291504,-2.9371767044067383,-3.3536458015441895],"probabilities_raw":[0.9561097621917725,0.018694812431931496,0.01518373005092144,0.010011709295213223],"probabilities":[0.8691593408584595,0.052293624728918076,0.04507270082831383,0.03347426652908325],"argmax_index":0,"argmax":"a","top2_margin":0.8168657161295414,"near_tie":false,"gold":"a","official_answer":{"type":"choice","choice":"a","confidence":0.8691593408584595,"probabilities":{"a":0.8691593408584595,"b":0.052293624728918076,"c":0.04507270082831383,"d":0.03347426652908325}},"usage_input_tokens":73,"oracle_file":"records_ref.json"},{"row_id":"tv4_043/text/answer","fixture_id":"tv4_043","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"either minority carriers or majority carriers","b":"minority carriers","c":"majority carriers","d":"valance carriers"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,30735,14977,10028,1878,997,39420,6881,997,31081,15909,797,784,14681,2841,803,768,13989,10321,31067,938,2588,906,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,4021,20744,32637,933,7604,32637,20,19,16,794,535,20744,32637,20,19,16,781,535,7604,32637,20,19,16,778,535,1623,1143,32637,20,21],"sequence_length":72,"marker_positions":[39,50,57,64],"positions":72,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-1.8836767673492432,-2.334036350250244,-3.524329423904419,-3.608792781829834],"probabilities_raw":[0.497659832239151,0.31720784306526184,0.0964730903506279,0.08865928649902344],"probabilities":[0.4298710525035858,0.31161749362945557,0.13315387070178986,0.12535753846168518],"argmax_index":0,"argmax":"a","top2_margin":0.11825355887413025,"near_tie":false,"gold":"b","official_answer":{"type":"choice","choice":"a","confidence":0.4298710525035858,"probabilities":{"a":0.4298710525035858,"b":0.31161749362945557,"c":0.13315387070178986,"d":0.12535753846168518}},"usage_input_tokens":72,"oracle_file":"records_ref.json"},{"row_id":"tv4_044/text/answer","fixture_id":"tv4_044","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Truman Doctrine","b":"Fair Labor Standards Act of 1961","c":"National Industrial Recovery Act","d":"Pendleton Act of 1883"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,3134,810,13032,1878,997,39420,6881,997,1098,20069,3345,1678,803,9170,800,5614,3714,10380,953,10328,968,779,39629,1678,906,768,1725,803,779,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,53311,59812,20,19,16,794,535,14366,10335,23335,4120,803,730,1848,526,20,19,16,781,535,2947,21889,40468,4120,20,19,16,778,535,54130,13501,4120,803,730,3699,528,20,21],"sequence_length":89,"marker_positions":[48,55,68,77],"positions":89,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.3090124130249023,-2.2886719703674316,-0.1633472740650177,1.2341519594192505],"probabilities_raw":[0.022151127457618713,0.02260630577802658,0.1893417239189148,0.7659007906913757],"probabilities":[0.052051614969968796,0.05281345546245575,0.24104060232639313,0.654094398021698],"argmax_index":3,"argmax":"d","top2_margin":0.41305379569530487,"near_tie":false,"gold":"d","official_answer":{"type":"choice","choice":"d","confidence":0.654094398021698,"probabilities":{"a":0.052051614969968796,"b":0.05281345546245575,"c":0.24104060232639313,"d":0.654094398021698}},"usage_input_tokens":89,"oracle_file":"records_ref.json"},{"row_id":"tv4_045/text/answer","fixture_id":"tv4_045","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"A unit root process","b":"A stationary process","c":"A deterministic trend process","d":"A random walk with drift"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,578,2499,813,8731,1878,997,39420,6881,997,25551,779,2773,2801,875,1058,598,2884,513,23606,587,13699,513,598,2884,1051,13264,4634,2011,13264,5762,766,2011,832,2884,4927,587,13699,28828,1235,803,779,2773,1529,20298,11854,779,1996,875,1058,598,2884,513,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,835,6232,7152,1996,20,19,16,794,535,835,37970,1996,20,19,16,781,535,835,15249,3547,13974,1996,20,19,16,778,535,835,7226,5043,916,26436,20,21],"sequence_length":106,"marker_positions":[69,78,86,96],"positions":106,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.1159785985946655,-0.7625792026519775,-0.41189315915107727,-1.6540266275405884],"probabilities_raw":[0.6980953812599182,0.10667619109153748,0.15148460865020752,0.04374382644891739],"probabilities":[0.5762596130371094,0.15060095489025116,0.19347375631332397,0.07966570556163788],"argmax_index":0,"argmax":"a","top2_margin":0.3827858567237854,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"a","confidence":0.5762596130371094,"probabilities":{"a":0.5762596130371094,"b":0.15060095489025116,"c":0.19347375631332397,"d":0.07966570556163788}},"usage_input_tokens":106,"oracle_file":"records_ref.json"},{"row_id":"tv4_046/text/answer","fixture_id":"tv4_046","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Glycolysis","b":"Chemiosmosis","c":"Fermentation","d":"Calvin cycle"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,16290,1878,997,39420,6881,997,2443,1996,8806,811,779,9257,3893,803,1547,14471,1005,595,931,521,1547,26171,521,810,1547,55984,549,2927,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,47256,2293,2916,20,19,16,794,535,8670,2574,586,3838,20,19,16,781,535,12656,957,852,20,19,16,778,535,41778,8192,20,21],"sequence_length":78,"marker_positions":[46,54,63,71],"positions":78,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.21103397011756897,-4.946765422821045,-1.4108130931854248,-0.5645290017127991],"probabilities_raw":[0.4969521462917328,0.004361264873296022,0.14971217513084412,0.34897446632385254],"probabilities":[0.4473859369754791,0.015188070014119148,0.18987658619880676,0.3475493788719177],"argmax_index":0,"argmax":"a","top2_margin":0.0998365581035614,"near_tie":false,"gold":"a","official_answer":{"type":"choice","choice":"a","confidence":0.4473859369754791,"probabilities":{"a":0.4473859369754791,"b":0.015188070014119148,"c":0.18987658619880676,"d":0.3475493788719177}},"usage_input_tokens":78,"oracle_file":"records_ref.json"},{"row_id":"tv4_047/text/answer","fixture_id":"tv4_047","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Y chromosome","b":"dominant genes","c":"genotype","d":"phenotype"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,30327,16964,18849,1878,997,39420,6881,997,542,6394,5867,1512,4100,44888,811,1450,2153,5822,1689,13564,20972,4381,4100,8255,523,941,13789,20314,968,1033,6394,9784,768,15488,803,1450,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,1267,29871,20,19,16,794,535,16234,7695,20,19,16,781,535,49249,20,19,16,778,535,38939,20,21],"sequence_length":79,"marker_positions":[53,60,67,73],"positions":79,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-3.3293027877807617,2.4848766326904297,-5.358247756958008,1.0101518630981445],"probabilities_raw":[0.0024224012158811092,0.8115440607070923,0.000318483158480376,0.18571512401103973],"probabilities":[0.011484759859740734,0.7309231758117676,0.002695730421692133,0.25489628314971924],"argmax_index":1,"argmax":"b","top2_margin":0.47602689266204834,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"b","confidence":0.7309231758117676,"probabilities":{"a":0.011484759859740734,"b":0.7309231758117676,"c":0.002695730421692133,"d":0.25489628314971924}},"usage_input_tokens":79,"oracle_file":"records_ref.json"},{"row_id":"tv4_048/text/answer","fixture_id":"tv4_048","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Wrong, Wrong","b":"Wrong, Not wrong","c":"Not wrong, Wrong","d":"Not wrong, Not wrong"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,586,8988,21469,1878,997,39420,6881,997,3111,1144,803,1518,1547,21469,2177,779,2120,3338,843,18275,5582,859,524,1477,524,3462,518,1112,3503,8870,61611,7871,521,4262,811,17027,11241,8079,797,779,2641,906,803,730,1718,525,540,3108,31933,730,526,2171,859,3595,5665,1419,768,9953,1101,859,1738,1927,936,875,6471,523,3108,31933,730,527,2171,859,17453,779,30704,1680,1429,6248,29293,2927,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,923,35736,521,923,35736,20,19,16,794,535,923,35736,521,4524,7871,20,19,16,781,535,4524,7871,521,923,35736,20,19,16,778,535,4524,7871,521,4524,7871,20,21],"sequence_length":136,"marker_positions":[96,106,116,126],"positions":136,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[1.6919503211975098,-0.03276406228542328,-0.9112541675567627,-1.3428339958190918],"probabilities_raw":[0.7690266966819763,0.13705897331237793,0.0569356232881546,0.036978673189878464],"probabilities":[0.6402612328529358,0.18676437437534332,0.09971431642770767,0.07326005399227142],"argmax_index":0,"argmax":"a","top2_margin":0.45349685847759247,"near_tie":false,"gold":"d","official_answer":{"type":"choice","choice":"a","confidence":0.6402612328529358,"probabilities":{"a":0.6402612328529358,"b":0.18676437437534332,"c":0.09971431642770767,"d":0.07326005399227142}},"usage_input_tokens":136,"oracle_file":"records_ref.json"},{"row_id":"tv4_049/text/answer","fixture_id":"tv4_049","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"The participants should be selected at random.","b":"It is an example of a formal research tool for practitioners.","c":"The information can be generalized to describe a larger population.","d":"The findings generate qualitative data that can be used to develop surveys."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,5146,3717,1878,997,39420,6881,997,28828,8294,856,3347,1337,3560,2514,2098,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,941,7524,1753,874,7795,963,7226,523,20,19,16,794,535,1311,856,902,2623,803,768,9885,2098,6690,875,27877,523,20,19,16,781,535,941,2203,1011,874,33518,811,8487,768,5814,3443,523,20,19,16,778,535,941,7747,10361,29581,1818,896,1011,874,1593,811,1705,21622,523,20,21],"sequence_length":96,"marker_positions":[32,45,62,78],"positions":96,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-6.835418224334717,-0.9563892483711243,-3.4500579833984375,-0.31408247351646423],"probabilities_raw":[0.0009367892635054886,0.3348664939403534,0.027662096545100212,0.636534571647644],"probabilities":[0.005424226634204388,0.3615814447402954,0.0608958899974823,0.5720984935760498],"argmax_index":3,"argmax":"d","top2_margin":0.2105170488357544,"near_tie":false,"gold":"d","official_answer":{"type":"choice","choice":"d","confidence":0.5720984935760498,"probabilities":{"a":0.005424226634204388,"b":0.3615814447402954,"c":0.0608958899974823,"d":0.5720984935760498}},"usage_input_tokens":96,"oracle_file":"records_ref.json"},{"row_id":"tv4_050/text/answer","fixture_id":"tv4_050","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"7.4","b":"−7.4","c":"−19.32","d":"14.72"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,578,12453,17263,1878,997,39420,6881,997,64005,779,10805,15702,1417,523,5843,1051,15702,527,523,528,598,523,11205,1180,5642,2927,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,730,532,523,529,20,19,16,794,535,15702,532,523,529,20,19,16,781,535,15702,913,523,2925,20,19,16,778,535,730,1536,523,8333,20,21],"sequence_length":78,"marker_positions":[42,51,60,69],"positions":78,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-0.026801295578479767,-0.5347694158554077,-2.2902116775512695,-0.3073588013648987],"probabilities_raw":[0.4063265323638916,0.24449355900287628,0.04225601628422737,0.30692383646965027],"probabilities":[0.36864981055259705,0.25646352767944336,0.07318627089262009,0.3017003834247589],"argmax_index":0,"argmax":"a","top2_margin":0.06694942712783813,"near_tie":false,"gold":"a","official_answer":{"type":"choice","choice":"a","confidence":0.36864981055259705,"probabilities":{"a":0.36864981055259705,"b":0.25646352767944336,"c":0.07318627089262009,"d":0.3017003834247589}},"usage_input_tokens":78,"oracle_file":"records_ref.json"},{"row_id":"tv4_051/text/answer","fixture_id":"tv4_051","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"cut marks on animal bones that were made by butchering with pieces of stone.","b":"the anatomy of Australopithecus africanus hands and analogies with studies of living non-human primates that demonstrate tool use.","c":"simple stone cores, anvil stones, and flakes found at the Lomekwi 3 site in West Turkana, Kenya.","d":"all of the above."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,3034,41336,1878,997,39420,6881,997,1098,8757,797,6131,803,3528,769,2796,3525,810,1993,9903,6486,968,963,3653,730,528,523,528,3847,1779,6027,856,7951,968,17912,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,4169,16120,884,6134,13665,896,1166,2047,968,1203,824,2644,916,9556,803,9903,523,20,19,16,794,535,779,36448,803,4163,920,42437,11490,3545,33450,868,6108,810,18583,969,916,3609,803,4624,2222,54127,57203,896,13473,6690,1517,523,20,19,16,781,535,4048,9903,39052,521,902,20247,18438,521,810,1434,2385,1833,963,779,903,1071,584,14415,730,528,3475,797,3227,16485,2468,521,24973,20,19,16,778,535,1113,803,779,3537,523,20,21],"sequence_length":145,"marker_positions":[50,72,103,135],"positions":145,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-2.4747769832611084,1.4043216705322266,-0.4620096981525421,2.141174077987671],"probabilities_raw":[0.006331176962703466,0.30630606412887573,0.047382522374391556,0.6399801969528198],"probabilities":[0.020735442638397217,0.3312358558177948,0.08732527494430542,0.560703456401825],"argmax_index":3,"argmax":"d","top2_margin":0.22946760058403015,"near_tie":false,"gold":"d","official_answer":{"type":"choice","choice":"d","confidence":0.560703456401825,"probabilities":{"a":0.020735442638397217,"b":0.3312358558177948,"c":0.08732527494430542,"d":0.560703456401825}},"usage_input_tokens":145,"oracle_file":"records_ref.json"},{"row_id":"tv4_052/text/answer","fixture_id":"tv4_052","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Ag+(aq) + Br-(aq) → AgBr(s)","b":"NH4+(aq) + Ag+(aq) → Ag(NH4)3+2(aq)","c":"Br-(aq) + NO3-(aq) → NO3Br(aq)","d":"NH4Br(aq) + NO3-(aq) → NH4NO3(aq) + Br-(aq)"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,18166,1878,997,39420,6881,997,1098,9257,14951,793,10805,5559,1403,7864,803,33128,529,21369,810,4059,20099,528,938,10372,3658,856,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,4059,46777,63868,518,2011,2642,25960,63868,518,17493,4059,21369,5381,518,20,19,16,794,535,33128,529,46777,63868,518,2011,4059,46777,63868,518,17493,4059,22386,549,529,518,528,520,527,13792,590,518,20,19,16,781,535,2642,25960,63868,518,2011,15081,528,25960,63868,518,17493,15081,528,21369,13792,590,518,20,19,16,778,535,33128,529,21369,13792,590,518,2011,15081,528,25960,63868,518,17493,33128,529,20099,528,13792,590,518,2011,2642,25960,63868,518,20,21],"sequence_length":143,"marker_positions":[45,64,91,113],"positions":143,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.5273202657699585,-0.3178127408027649,-2.177464246749878,0.6166142225265503],"probabilities_raw":[0.3861321210861206,0.16584382951259613,0.02582634799182415,0.4221976399421692],"probabilities":[0.3626495897769928,0.19828923046588898,0.05252596735954285,0.38653525710105896],"argmax_index":3,"argmax":"d","top2_margin":0.023885667324066162,"near_tie":false,"gold":"a","official_answer":{"type":"choice","choice":"d","confidence":0.38653525710105896,"probabilities":{"a":0.3626495897769928,"b":0.19828923046588898,"c":0.05252596735954285,"d":0.38653525710105896}},"usage_input_tokens":143,"oracle_file":"records_ref.json"},{"row_id":"tv4_053/text/answer","fixture_id":"tv4_053","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"The obtaining of a loan from a bank was not worded as a \"condition\" to the buyer's duty to buy the business.","b":"Although obtaining a loan from a bank was a condition to the buyer's duty to buy the business, the condition should be excused because its non-occurrence would cause a forfeiture to the owner.","c":"Although obtaining a loan from a bank was a condition to the buyer's duty to buy the business, it should be stricken from the contract because it is an unconscionable term.","d":"The buyer breached an implied promise to make a reasonable effort to obtain a loan from a bank."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,30327,16964,3036,1878,997,39420,6881,997,2797,12153,16217,768,7021,18479,3436,523,1405,8091,950,7050,811,42766,810,9670,1210,3436,875,12614,523,835,48463,5944,779,21008,810,15596,779,18479,3436,523,941,48463,5689,779,12153,896,1689,953,1943,9368,797,18649,779,3436,521,1203,896,1689,4602,811,1673,768,16763,988,779,7493,811,17830,779,11931,523,941,9888,43360,768,6827,521,10423,968,2059,9888,521,1144,9519,896,779,48463,11860,811,11931,779,18479,3436,988,779,12153,875,1058,7896,521,1873,523,941,4985,10449,11874,768,19505,41622,779,12614,1499,1014,2337,1862,43175,885,2620,779,48463,856,7021,797,1450,23956,811,4387,768,16763,988,768,7493,811,17830,779,11931,803,779,3436,523,43175,50380,521,779,48463,2047,1295,4516,811,5308,811,4387,768,16763,988,768,7493,1101,896,779,12614,1738,874,2524,1884,523,4162,2993,4493,521,779,12153,25752,14503,2723,779,48463,875,43510,803,6827,523,20463,803,779,2773,1188,874,779,12153,1090,31712,8757,797,2258,803,1210,4272,2723,779,48463,875,43510,803,6827,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,941,27471,803,768,16763,988,768,7493,953,1014,2359,791,906,768,997,25556,511,811,779,48463,1090,15774,811,7909,779,3436,523,20,19,16,794,535,6463,27471,768,16763,988,768,7493,953,768,4576,811,779,48463,1090,15774,811,7909,779,3436,521,779,4576,1753,874,4016,6086,1929,20,19,16,781,535,6463,27471,768,16763,988,768,7493,953,768,4576,811,779,48463,1090,15774,811,7909,779,3436,521,936,1753,874,54604,3449,988,779,20,19,16,778,535,941,48463,2276,4660,902,32056,16114,811,1825,768,16336,4516,811,4387,768,16763,988,768,7493,523,20,21],"sequence_length":343,"marker_positions":[222,254,286,318],"positions":343,"window":512,"window_r10":512,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.5962756872177124,-0.33312368392944336,0.5012220740318298,-0.5879086852073669],"probabilities_raw":[0.38312551379203796,0.15125441551208496,0.3483853042125702,0.11723475158214569],"probabilities":[0.34742024540901184,0.17886479198932648,0.3246133625507355,0.14910155534744263],"argmax_index":0,"argmax":"a","top2_margin":0.022806882858276367,"near_tie":false,"gold":"d","official_answer":{"type":"choice","choice":"a","confidence":0.34742024540901184,"probabilities":{"a":0.34742024540901184,"b":0.17886479198932648,"c":0.3246133625507355,"d":0.14910155534744263}},"usage_input_tokens":343,"oracle_file":"records_ref.json"},{"row_id":"tv4_054/text/answer","fixture_id":"tv4_054","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"2.166 x 10^-34 J s","b":"3.691 x 10^-34 J s","c":"4.185 x 10^-34 J s","d":"5.493 x 10^-34 J s"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,2293,3810,18166,1878,997,39420,6881,997,59061,931,779,12277,26698,21232,803,730,4613,35117,523,1607,550,1051,730,532,41562,736,527,570,511,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,730,527,523,10782,3053,730,1155,34193,2933,962,775,20,19,16,794,535,730,528,523,25785,3053,730,1155,34193,2933,962,775,20,19,16,781,535,730,529,523,4420,3053,730,1155,34193,2933,962,775,20,19,16,778,535,730,530,523,33052,3053,730,1155,34193,2933,962,775,20,21],"sequence_length":109,"marker_positions":[45,61,77,93],"positions":109,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-3.5054848194122314,-2.425651788711548,-0.6409085988998413,-3.510509490966797],"probabilities_raw":[0.044482409954071045,0.13096457719802856,0.7802935242652893,0.04425947368144989],"probabilities":[0.08404666930437088,0.18176917731761932,0.6504386067390442,0.08374553918838501],"argmax_index":2,"argmax":"c","top2_margin":0.46866942942142487,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"c","confidence":0.6504386067390442,"probabilities":{"a":0.08404666930437088,"b":0.18176917731761932,"c":0.6504386067390442,"d":0.08374553918838501}},"usage_input_tokens":109,"oracle_file":"records_ref.json"},{"row_id":"tv4_055/text/answer","fixture_id":"tv4_055","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"$710,000","b":"$750,000","c":"$830,000","d":"1000000"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,30327,16964,21167,1878,997,39420,6881,997,43211,853,3862,523,5582,768,26856,30174,1678,523,941,2773,938,30174,23301,875,779,5805,803,5250,535,730,526,524,526,47044,30174,730,1086,1056,8202,963,1058,1532,730,526,524,898,873,43418,730,5091,1056,8202,963,1058,1386,730,526,524,1968,873,43418,730,6019,1056,8202,963,1058,1417,730,526,524,2999,35791,963,1058,898,1094,6232,730,4291,1056,8202,47751,5582,779,903,8526,556,2511,811,6187,779,2536,803,1352,30174,523,3747,3685,1753,47751,2312,906,2780,803,11391,4997,884,1352,8292,8294,875,779,5805,803,5250,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,1058,22557,521,1873,20,19,16,794,535,1058,14671,521,1873,20,19,16,781,535,1058,23975,521,1873,20,19,16,778,535,730,2096,1873,525,20,21],"sequence_length":169,"marker_positions":[133,142,151,160],"positions":169,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-3.6334569454193115,-0.35530930757522583,-3.424314260482788,-5.927770137786865],"probabilities_raw":[0.034649983048439026,0.9191460013389587,0.04271029308438301,0.003493780270218849],"probabilities":[0.07840510457754135,0.8153303861618042,0.09103896468877792,0.015225550159811974],"argmax_index":1,"argmax":"b","top2_margin":0.7242914214730263,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"b","confidence":0.8153303861618042,"probabilities":{"a":0.07840510457754135,"b":0.8153303861618042,"c":0.09103896468877792,"d":0.015225550159811974}},"usage_input_tokens":169,"oracle_file":"records_ref.json"},{"row_id":"tv4_056/text/answer","fixture_id":"tv4_056","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Tariff reductions on exports","b":"Raw materials imported duty-free","c":"Exemption from customs duties","d":"Strict pollution regulations"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,31116,1878,997,39420,6881,997,28828,803,779,2773,856,14353,768,5992,803,902,13793,9147,9985,843,10153,567,518,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,14710,3890,35495,884,30118,20,19,16,794,535,27965,5168,28443,15774,11867,20,19,16,781,535,2315,34987,988,30101,20698,20,19,16,778,535,1087,2930,11244,14272,20,21],"sequence_length":81,"marker_positions":[42,52,62,72],"positions":81,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-7.277989864349365,-2.838202953338623,-4.520140171051025,-0.9320415258407593],"probabilities_raw":[0.0014887587167322636,0.12618252635002136,0.02347162738442421,0.8488571047782898],"probabilities":[0.007995810359716415,0.1906471699476242,0.057337407022714615,0.744019627571106],"argmax_index":3,"argmax":"d","top2_margin":0.5533724576234818,"near_tie":false,"gold":"d","official_answer":{"type":"choice","choice":"d","confidence":0.744019627571106,"probabilities":{"a":0.007995810359716415,"b":0.1906471699476242,"c":0.057337407022714615,"d":0.744019627571106}},"usage_input_tokens":81,"oracle_file":"records_ref.json"},{"row_id":"tv4_057/text/answer","fixture_id":"tv4_057","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Volunteering","b":"Travel","c":"Redoing the home","d":"Household chores"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,28009,19179,1878,997,39420,6881,997,28828,803,779,2773,856,14353,1235,803,779,1209,9146,5668,4637,968,2025,1316,938,19620,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,42154,2644,20,19,16,794,535,26596,20,19,16,781,535,5148,54599,779,2681,20,19,16,778,535,5754,4289,979,2362,20,21],"sequence_length":73,"marker_positions":[42,49,55,64],"positions":73,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[-8.583589553833008,-5.527565956115723,-6.5222697257995605,-5.621074676513672],"probabilities_raw":[0.020224157720804214,0.4296201169490814,0.15888769924640656,0.39126795530319214],"probabilities":[0.044379282742738724,0.39378514885902405,0.19349488615989685,0.36834073066711426],"argmax_index":1,"argmax":"b","top2_margin":0.02544441819190979,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"b","confidence":0.39378514885902405,"probabilities":{"a":0.044379282742738724,"b":0.39378514885902405,"c":0.19349488615989685,"d":0.36834073066711426}},"usage_input_tokens":73,"oracle_file":"records_ref.json"},{"row_id":"tv4_058/text/answer","fixture_id":"tv4_058","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"Wrong, Wrong","b":"Wrong, Not wrong","c":"Not wrong, Wrong","d":"Not wrong, Not wrong"}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,586,8988,21469,1878,997,39420,6881,997,3111,1144,803,1518,1547,21469,2177,779,2120,3338,843,18275,5582,859,524,1477,524,3462,518,1112,3503,8870,61611,7871,521,4262,811,17027,11241,8079,797,779,2641,906,803,730,1718,525,540,3108,31933,730,526,2171,859,8940,1727,6709,1673,1230,803,779,3622,1720,950,5807,1210,58110,523,3108,31933,730,527,2171,859,4925,1886,45201,2601,11468,797,779,3681,12420,521,2158,7250,953,963,779,12420,3662,2927,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,923,35736,521,923,35736,20,19,16,794,535,923,35736,521,4524,7871,20,19,16,781,535,4524,7871,521,923,35736,20,19,16,778,535,4524,7871,521,4524,7871,20,21],"sequence_length":147,"marker_positions":[107,117,127,137],"positions":147,"window":256,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[0.6942271590232849,-2.231766939163208,-0.7739002704620361,-1.7253468036651611],"probabilities_raw":[0.7283706665039062,0.039048947393894196,0.16778486967086792,0.06479549407958984],"probabilities":[0.605466902256012,0.07487740367650986,0.21214313805103302,0.10751249641180038],"argmax_index":0,"argmax":"a","top2_margin":0.39332376420497894,"near_tie":false,"gold":"c","official_answer":{"type":"choice","choice":"a","confidence":0.605466902256012,"probabilities":{"a":0.605466902256012,"b":0.07487740367650986,"c":0.21214313805103302,"d":0.10751249641180038}},"usage_input_tokens":147,"oracle_file":"records_ref.json"},{"row_id":"tv4_059/text/answer","fixture_id":"tv4_059","mode":"text","question_id":"answer","question":{"type":"choice","instructions":"Which option correctly answers the question?","criteria":{"a":"People are more likely to recall information that is congruent with their prior schemas.","b":"People will not recall information unless it is stored in working memory.","c":"People have no ability to recall information that has been extinguished.","d":"People will recall nonsense syllables more rapidly than they will sets of numbers."}},"type":"choice","qtype":0,"K":4,"media":[],"prefix_rows":0,"sequence_ids":[1,17,38685,42505,6881,997,17756,2254,18849,1878,997,39420,6881,997,28828,803,779,2773,856,2492,3347,803,6448,18328,8523,602,18,28828,7832,16066,13041,779,3952,540,19,16,768,535,7817,938,1209,3849,811,18328,2203,896,856,880,27354,806,916,1149,5344,5812,7321,523,20,19,16,794,535,7817,1188,1014,18328,2203,10967,936,856,11611,797,3822,6448,523,20,19,16,781,535,7817,1052,1295,4978,811,18328,2203,896,1178,1381,1741,40927,523,20,19,16,778,535,7817,1188,18328,34045,2378,28450,3037,1209,11675,1399,1186,1188,8753,803,5743,523,20,21],"sequence_length":115,"marker_positions":[35,58,76,94],"positions":115,"window":128,"window_r10":256,"temperature_key":"choice:3-5","temperature":1.3998981714248657,"raw_logits":[3.9577229022979736,-3.6439967155456543,-2.666415214538574,-0.6292939186096191],"probabilities_raw":[0.9881318807601929,0.0004936622572131455,0.0013121648225933313,0.01006233412772417],"probabilities":[0.951524019241333,0.004169808700680733,0.008382895030081272,0.03592329099774361],"argmax_index":0,"argmax":"a","top2_margin":0.9156007282435894,"near_tie":false,"gold":"a","official_answer":{"type":"choice","choice":"a","confidence":0.951524019241333,"probabilities":{"a":0.951524019241333,"b":0.004169808700680733,"c":0.008382895030081272,"d":0.03592329099774361}},"usage_input_tokens":115,"oracle_file":"records_ref.json"}],"summary":{"records":237,"request_modes":255,"rows":360,"rows_by_mode":{"text":306,"image":9,"audio":45},"rows_by_window":{"64":85,"128":167,"256":69,"512":25,"1024":0,"2048":10,"4096":4},"rows_by_window_r10":{"256":321,"512":25,"1024":0,"2048":10,"4096":4},"near_ties":["own_t04/text/blockers","semif_42384af9ae56beb37ff5/text/decision","semif_695dd61195ca025c4f4c/text/decision","semif_712f6ffb763eef7c9a12/text/decision","semif_cdf7f428857e1b841cb3/text/decision","semif_d0e36c4cb10a29b3191c/text/decision","semif_fda0ca94652c6d739ecf/text/decision","tv4_015/text/answer","tv4_037/text/answer"],"by_source":{"card":{"records":3,"questions":5},"transfer_v4_dev_head":{"records":60,"questions":60},"semif_authored144":{"records":144,"questions":144},"own_text":{"records":6,"questions":23},"own_json":{"records":3,"questions":9},"own_long":{"records":2,"questions":7},"own_long_extended":{"records":1,"questions":4},"own_image":{"records":3,"questions":9},"own_audio":{"records":15,"questions":45}}},"built_from":{"fixtures/records.json":{"bytes":721270,"sha256":"18b4ddff73a8044b63f949c69245210e1af8a26c48c5fdb785fd91a7807ead5b"},"results/encode_rows.v3.json":{"bytes":465588,"sha256":"874dfbab1b3aea6c9a5e44de52f82b8e74bda8a06599e21ac781d3bd8e23eb0b"},"ref/records_ref_audio.json":{"bytes":113222,"sha256":"c483f8c511cfcdb05a8211e495a7476b11f9318899a1125c387c9e722a6d8fbc"},"ref/records_ref_images.json":{"bytes":131795,"sha256":"681d2d10eef35ecb8980a3caf93d922a77a17e88ee58795468e78b915ac91b08"},"ref/records_ref.json":{"bytes":1068659,"sha256":"e7dba6f44d0452a6aea3e401746d417503056d6f468ff72b56c5511883436c5f"},"fixtures/LICENSE-SemIf":{"bytes":1066,"sha256":"f765f2140f8507a8f0d81ec0fd2c4bd72fe6a066841ef27883ff876a76bf61be"},"fixtures/LICENSE-MMLU-MIT.txt":{"bytes":1070,"sha256":"5d841b9eee8bf0d721359b79fbae956a8b426d3139bfefee32149c7d4f2a1a9c"}}}